Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. MetaCap: Meta-learning Priors from Multi-view Imagery for Sparse-View Human Performance Capture and Rendering
 
conference paper

MetaCap: Meta-learning Priors from Multi-view Imagery for Sparse-View Human Performance Capture and Rendering

Sun, Guoxing
•
Dabral, Rishabh
•
Fua, Pascal  
Show more
Leonardis, Aleš
•
Ricci, Elisa
Show more
2025
Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
18th European Conference on Computer Vision

Faithful human performance capture and free-view rendering from sparse RGB observations is a long-standing problem in Vision and Graphics. The main challenges are the lack of observations and the inherent ambiguities of the setting, e.g. occlusions and depth ambiguity. As a result, radiance fields, which have shown great promise in capturing high-frequency appearance and geometry details in dense setups, perform poorly when naïvely supervising them on sparse camera views, as the field simply overfits to the sparse-view inputs. To address this, we propose MetaCap, a method for efficient and high-quality geometry recovery and novel view synthesis given very sparse or even a single view of the human. Our key idea is to meta-learn the radiance field weights solely from potentially sparse multi-view videos, which can serve as a prior when fine-tuning them on sparse imagery depicting the human. This prior provides a good network weight initialization, thereby effectively addressing ambiguities in sparse-view capture. Due to the articulated structure of the human body and motion-induced surface deformations, learning such a prior is non-trivial. Therefore, we propose to meta-learn the field weights in a pose-canonicalized space, which reduces the spatial feature range and makes feature learning more effective. Consequently, one can fine-tune our field parameters to quickly generalize to unseen poses, novel illumination conditions as well as novel and sparse (even monocular) camera views. For evaluating our method under different scenarios, we collect a new dataset, WildDynaCap, which contains subjects captured in, both, a dense camera dome and in-the-wild sparse camera rigs, and demonstrate superior results compared to recent state-of-the-art methods on, both, public and WildDynaCap dataset.

  • Details
  • Metrics
Type
conference paper
DOI
10.1007/978-3-031-72952-2_20
Scopus ID

2-s2.0-85206351985

Author(s)
Sun, Guoxing

Max Planck Institute for Informatics

Dabral, Rishabh

Max Planck Institute for Informatics

Fua, Pascal  

École Polytechnique Fédérale de Lausanne

Theobalt, Christian

Max Planck Institute for Informatics

Habermann, Marc

Max Planck Institute for Informatics

Editors
Leonardis, Aleš
•
Ricci, Elisa
•
Roth, Stefan
•
Russakovsky, Olga
•
Sattler, Torsten
•
Varol, Gül
Date Issued

2025

Publisher

Springer Science and Business Media Deutschland GmbH

Published in
Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
Series title/Series vol.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 15104 LNCS

ISSN (of the series)

1611-3349

0302-9743

Start page

341

End page

361

Subjects

Human Performance Capture

•

Meta Learning

Editorial or Peer reviewed

REVIEWED

Written at

EPFL

EPFL units
CVLAB  
Event nameEvent acronymEvent placeEvent date
18th European Conference on Computer Vision

Milan, Italy

2024-09-29 - 2024-10-04

FunderFunding(s)Grant NumberGrant URL

ERC

770784

Available on Infoscience
January 26, 2025
Use this identifier to reference this record
https://infoscience.epfl.ch/handle/20.500.14299/245092
Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés