Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces

Mesh-Tension Driven Expression-Based Wrinkles for Synthetic Faces

Raman, C.A. (TU Delft Pattern Recognition and Bioinformatics)
Hewitt, Charlie (Microsoft)
Wood, Erroll (Microsoft)
Baltrusaitis, Tadas (Microsoft)

O’Conner, Lisa (editor)

2023

Recent advances in synthesizing realistic faces have shown that synthetic training data can replace real data for various face-related computer vision tasks. A question arises: how important is realism? Is the pursuit of photorealism excessive? In this work, we show otherwise. We boost the realism of our synthetic faces by introducing dynamic skin wrinkles in response to facial expressions, and observe significant performance improvements in downstream computer vision tasks. Previous approaches for producing such wrinkles either required prohibitive artist effort to scale across identities and expressions, or were not capable of reconstructing high-frequency skin details with sufficient fidelity. Our key contribution is an approach that produces realistic wrinkles across a large and diverse population of digital humans. Concretely, we formalize the concept of mesh-tension and use it to aggregate possible wrinkles from high-quality expression scans into albedo and displacement texture maps. At synthesis, we use these maps to produce wrinkles even for expressions not represented in the source scans. Additionally, to provide a more nuanced indicator of model performance under deformations resulting from com-pressed expressions, we introduce the 300W-winks evaluation subset and the Pexels dataset of closed eyes and winks.

http://resolver.tudelft.nl/uuid:ebb69ad0-a938-45df-bd60-ea9cb35f2f05

https://doi.org/10.1109/WACV56688.2023.00351

IEEE, Piscataway

2023-08-06

978-1-6654-9347-5

Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

WACV, 2023-01-02 → 2023-01-07, Waikoloa, United States

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

conference paper

© 2023 C.A. Raman, Charlie Hewitt, Erroll Wood, Tadas Baltrusaitis