Multiface: A Dataset for Neural Face Rendering
Authors:
Cheng-hsin Wuu,
Ningyuan Zheng,
Scott Ardisson,
Rohan Bali,
Danielle Belko,
Eric Brockmeyer,
Lucas Evans,
Timothy Godisart,
Hyowon Ha,
Xuhua Huang,
Alexander Hypes,
Taylor Koska,
Steven Krenn,
Stephen Lombardi,
Xiaomin Luo,
Kevyn McPhail,
Laura Millerschoen,
Michal Perdoch,
Mark Pitts,
Alexander Richard,
Jason Saragih,
Junko Saragih,
Takaaki Shiratori,
Tomas Simon,
Matt Stewart
, et al. (6 additional authors not shown)
Abstract:
Photorealistic avatars of human faces have come a long way in recent years, yet research along this area is limited by a lack of publicly available, high-quality datasets covering both, dense multi-view camera captures, and rich facial expressions of the captured subjects. In this work, we present Multiface, a new multi-view, high-resolution human face dataset collected from 13 identities at Reali…
▽ More
Photorealistic avatars of human faces have come a long way in recent years, yet research along this area is limited by a lack of publicly available, high-quality datasets covering both, dense multi-view camera captures, and rich facial expressions of the captured subjects. In this work, we present Multiface, a new multi-view, high-resolution human face dataset collected from 13 identities at Reality Labs Research for neural face rendering. We introduce Mugsy, a large scale multi-camera apparatus to capture high-resolution synchronized videos of a facial performance. The goal of Multiface is to close the gap in accessibility to high quality data in the academic community and to enable research in VR telepresence. Along with the release of the dataset, we conduct ablation studies on the influence of different model architectures toward the model's interpolation capacity of novel viewpoint and expressions. With a conditional VAE model serving as our baseline, we found that adding spatial bias, texture warp field, and residual connections improves performance on novel view synthesis. Our code and data is available at: https://github.com/facebookresearch/multiface
△ Less
Submitted 26 June, 2023; v1 submitted 22 July, 2022;
originally announced July 2022.
HAA500: Human-Centric Atomic Action Dataset with Curated Videos
Authors:
Jihoon Chung,
Cheng-hsin Wuu,
Hsuan-ru Yang,
Yu-Wing Tai,
Chi-Keung Tang
Abstract:
We contribute HAA500, a manually annotated human-centric atomic action dataset for action recognition on 500 classes with over 591K labeled frames. To minimize ambiguities in action classification, HAA500 consists of highly diversified classes of fine-grained atomic actions, where only consistent actions fall under the same label, e.g., "Baseball Pitching" vs "Free Throw in Basketball". Thus HAA50…
▽ More
We contribute HAA500, a manually annotated human-centric atomic action dataset for action recognition on 500 classes with over 591K labeled frames. To minimize ambiguities in action classification, HAA500 consists of highly diversified classes of fine-grained atomic actions, where only consistent actions fall under the same label, e.g., "Baseball Pitching" vs "Free Throw in Basketball". Thus HAA500 is different from existing atomic action datasets, where coarse-grained atomic actions were labeled with coarse action-verbs such as "Throw". HAA500 has been carefully curated to capture the precise movement of human figures with little class-irrelevant motions or spatio-temporal label noises. The advantages of HAA500 are fourfold: 1) human-centric actions with a high average of 69.7% detectable joints for the relevant human poses; 2) high scalability since adding a new class can be done under 20-60 minutes; 3) curated videos capturing essential elements of an atomic action without irrelevant frames; 4) fine-grained atomic action classes. Our extensive experiments including cross-data validation using datasets collected in the wild demonstrate the clear benefits of human-centric and atomic characteristics of HAA500, which enable training even a baseline deep learning model to improve prediction by attending to atomic human poses. We detail the HAA500 dataset statistics and collection methodology and compare quantitatively with existing action recognition datasets.
△ Less
Submitted 16 August, 2021; v1 submitted 11 September, 2020;
originally announced September 2020.
