LU
L. Uzolas
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
4 records found
1
Being able to deform and manipulate 3D representations brings static objects to life in movies, video games, and other digital media. Recent advances in neural 3D representations have had a great impact on the field. However, animating and deforming them remains challenging, particularly due to the scarcity of training data in the 3D domain. This dissertation addresses these challenges along two directions: animating neural 3D representations and leveraging neural features from pre-trained 2D foundation models for 3D downstream tasks.
Chapter 2 introduces a method for reconstructing an animatable 3D neural radiance field from multi-view video. While previous methods reconstruct dynamic neural radiance fields via ray bending, we forward-warp an explicit sparse point cloud using a kinematic model, thereby avoiding projection ambiguities. We first initialize a sparse neural feature point cloud and an overparameterized kinematic skeleton using the medial axis transformation. The skeleton drives the point cloud deformation via linear blend skinning, after which the result is rendered via volumetric rendering, jointly optimizing the feature point cloud and deformation parameters. After training, the kinematic skeleton can be pruned, enabling easy manual animation of the reconstructed representation.
Chapter 3 explores how to animate 3D meshes from text prompts in a zero-shot setting, without training any deformation models. To this end, we embed 3D meshes into the latent feature space of pre-trained video diffusion models by re-projecting semantic neural features onto the mesh surface. We then optimize the mesh deformation to match a generated latent video, which is more robust than fitting to RGB outputs. In contrast to related work, our method produces plausible animation sequences in minutes, demonstrating the power of these features for 3D downstream tasks.
Chapter 4 further investigates how to improve semantic neural features from 2D foundation models for 3D applications that rely on establishing correspondences between meshes. We address the issue that these features struggle to differentiate between instances of the same semantic class, such as ``left hand’’ versus ``right hand’’. We learn an embedding space that disambiguates such instances using the geodesic distance on the mesh surface, while preserving the original semantic information. The method is purely self-supervised, requires only a small number of unpaired training meshes, and we show that the resulting surface-aware features significantly improve performance on several 3D tasks.
Chapter 5 investigates a rotation-equivariant formulation of Neural Jacobian Fields, a method for learning detail-preserving mesh deformations in the gradient domain.
In the previous chapter, we showed that semantic features can condition a Neural Jacobian Field to learn generalized deformations across shape categories. However, Neural Jacobian Fields are inherently tied to the orientation of training shapes in ambient space, requiring inference shapes to be aligned to the training data. We address this limitation by introducing a rotation-equivariant formulation that produces consistent results independent of input orientation, while generalizing across diverse shapes. ...
Chapter 2 introduces a method for reconstructing an animatable 3D neural radiance field from multi-view video. While previous methods reconstruct dynamic neural radiance fields via ray bending, we forward-warp an explicit sparse point cloud using a kinematic model, thereby avoiding projection ambiguities. We first initialize a sparse neural feature point cloud and an overparameterized kinematic skeleton using the medial axis transformation. The skeleton drives the point cloud deformation via linear blend skinning, after which the result is rendered via volumetric rendering, jointly optimizing the feature point cloud and deformation parameters. After training, the kinematic skeleton can be pruned, enabling easy manual animation of the reconstructed representation.
Chapter 3 explores how to animate 3D meshes from text prompts in a zero-shot setting, without training any deformation models. To this end, we embed 3D meshes into the latent feature space of pre-trained video diffusion models by re-projecting semantic neural features onto the mesh surface. We then optimize the mesh deformation to match a generated latent video, which is more robust than fitting to RGB outputs. In contrast to related work, our method produces plausible animation sequences in minutes, demonstrating the power of these features for 3D downstream tasks.
Chapter 4 further investigates how to improve semantic neural features from 2D foundation models for 3D applications that rely on establishing correspondences between meshes. We address the issue that these features struggle to differentiate between instances of the same semantic class, such as ``left hand’’ versus ``right hand’’. We learn an embedding space that disambiguates such instances using the geodesic distance on the mesh surface, while preserving the original semantic information. The method is purely self-supervised, requires only a small number of unpaired training meshes, and we show that the resulting surface-aware features significantly improve performance on several 3D tasks.
Chapter 5 investigates a rotation-equivariant formulation of Neural Jacobian Fields, a method for learning detail-preserving mesh deformations in the gradient domain.
In the previous chapter, we showed that semantic features can condition a Neural Jacobian Field to learn generalized deformations across shape categories. However, Neural Jacobian Fields are inherently tied to the orientation of training shapes in ambient space, requiring inference shapes to be aligned to the training data. We address this limitation by introducing a rotation-equivariant formulation that produces consistent results independent of input orientation, while generalizing across diverse shapes. ...
Being able to deform and manipulate 3D representations brings static objects to life in movies, video games, and other digital media. Recent advances in neural 3D representations have had a great impact on the field. However, animating and deforming them remains challenging, particularly due to the scarcity of training data in the 3D domain. This dissertation addresses these challenges along two directions: animating neural 3D representations and leveraging neural features from pre-trained 2D foundation models for 3D downstream tasks.
Chapter 2 introduces a method for reconstructing an animatable 3D neural radiance field from multi-view video. While previous methods reconstruct dynamic neural radiance fields via ray bending, we forward-warp an explicit sparse point cloud using a kinematic model, thereby avoiding projection ambiguities. We first initialize a sparse neural feature point cloud and an overparameterized kinematic skeleton using the medial axis transformation. The skeleton drives the point cloud deformation via linear blend skinning, after which the result is rendered via volumetric rendering, jointly optimizing the feature point cloud and deformation parameters. After training, the kinematic skeleton can be pruned, enabling easy manual animation of the reconstructed representation.
Chapter 3 explores how to animate 3D meshes from text prompts in a zero-shot setting, without training any deformation models. To this end, we embed 3D meshes into the latent feature space of pre-trained video diffusion models by re-projecting semantic neural features onto the mesh surface. We then optimize the mesh deformation to match a generated latent video, which is more robust than fitting to RGB outputs. In contrast to related work, our method produces plausible animation sequences in minutes, demonstrating the power of these features for 3D downstream tasks.
Chapter 4 further investigates how to improve semantic neural features from 2D foundation models for 3D applications that rely on establishing correspondences between meshes. We address the issue that these features struggle to differentiate between instances of the same semantic class, such as ``left hand’’ versus ``right hand’’. We learn an embedding space that disambiguates such instances using the geodesic distance on the mesh surface, while preserving the original semantic information. The method is purely self-supervised, requires only a small number of unpaired training meshes, and we show that the resulting surface-aware features significantly improve performance on several 3D tasks.
Chapter 5 investigates a rotation-equivariant formulation of Neural Jacobian Fields, a method for learning detail-preserving mesh deformations in the gradient domain.
In the previous chapter, we showed that semantic features can condition a Neural Jacobian Field to learn generalized deformations across shape categories. However, Neural Jacobian Fields are inherently tied to the orientation of training shapes in ambient space, requiring inference shapes to be aligned to the training data. We address this limitation by introducing a rotation-equivariant formulation that produces consistent results independent of input orientation, while generalizing across diverse shapes.
Chapter 2 introduces a method for reconstructing an animatable 3D neural radiance field from multi-view video. While previous methods reconstruct dynamic neural radiance fields via ray bending, we forward-warp an explicit sparse point cloud using a kinematic model, thereby avoiding projection ambiguities. We first initialize a sparse neural feature point cloud and an overparameterized kinematic skeleton using the medial axis transformation. The skeleton drives the point cloud deformation via linear blend skinning, after which the result is rendered via volumetric rendering, jointly optimizing the feature point cloud and deformation parameters. After training, the kinematic skeleton can be pruned, enabling easy manual animation of the reconstructed representation.
Chapter 3 explores how to animate 3D meshes from text prompts in a zero-shot setting, without training any deformation models. To this end, we embed 3D meshes into the latent feature space of pre-trained video diffusion models by re-projecting semantic neural features onto the mesh surface. We then optimize the mesh deformation to match a generated latent video, which is more robust than fitting to RGB outputs. In contrast to related work, our method produces plausible animation sequences in minutes, demonstrating the power of these features for 3D downstream tasks.
Chapter 4 further investigates how to improve semantic neural features from 2D foundation models for 3D applications that rely on establishing correspondences between meshes. We address the issue that these features struggle to differentiate between instances of the same semantic class, such as ``left hand’’ versus ``right hand’’. We learn an embedding space that disambiguates such instances using the geodesic distance on the mesh surface, while preserving the original semantic information. The method is purely self-supervised, requires only a small number of unpaired training meshes, and we show that the resulting surface-aware features significantly improve performance on several 3D tasks.
Chapter 5 investigates a rotation-equivariant formulation of Neural Jacobian Fields, a method for learning detail-preserving mesh deformations in the gradient domain.
In the previous chapter, we showed that semantic features can condition a Neural Jacobian Field to learn generalized deformations across shape categories. However, Neural Jacobian Fields are inherently tied to the orientation of training shapes in ambient space, requiring inference shapes to be aligned to the training data. We address this limitation by introducing a rotation-equivariant formulation that produces consistent results independent of input orientation, while generalizing across diverse shapes.
Many 3D tasks such as pose alignment, animation, motion transfer, and 3D reconstruction rely on establishing correspondences between 3D shapes. This challenge has recently been approached by pairwise matching of semantic features from pre-trained vision models. However, despite their power, these features struggle to differentiate instances of the same semantic class such as "left hand"versus "right hand"which leads to substantial mapping errors. To solve this, we learn a surface-aware embedding space that is robust to these ambiguities while facilitating shared mapping for an entire family of 3D shapes. Importantly, our approach is self-supervised and requires only a small number of unpaired training meshes to infer features for new possibly imperfect 3D shapes at test time. We achieve this by introducing a contrastive loss that preserves the semantic content of the features distilled from foundational models while disambiguating features located far apart on the shape's surface. We observe superior performance in correspondence matching benchmarks and enable downstream applications including 2D-to-3D and 3D-to-3D texture transfer, in-part segmentation, pose alignment, and motion transfer in low-data regimes. Unlike previous pairwise approaches, our solution constructs a joint embedding space, where both seen and unseen 3D shapes are implicitly aligned without further optimization. The code is available at https://graphics.tudelft.nl/SurfaceAware3DFeatures.
...
Many 3D tasks such as pose alignment, animation, motion transfer, and 3D reconstruction rely on establishing correspondences between 3D shapes. This challenge has recently been approached by pairwise matching of semantic features from pre-trained vision models. However, despite their power, these features struggle to differentiate instances of the same semantic class such as "left hand"versus "right hand"which leads to substantial mapping errors. To solve this, we learn a surface-aware embedding space that is robust to these ambiguities while facilitating shared mapping for an entire family of 3D shapes. Importantly, our approach is self-supervised and requires only a small number of unpaired training meshes to infer features for new possibly imperfect 3D shapes at test time. We achieve this by introducing a contrastive loss that preserves the semantic content of the features distilled from foundational models while disambiguating features located far apart on the shape's surface. We observe superior performance in correspondence matching benchmarks and enable downstream applications including 2D-to-3D and 3D-to-3D texture transfer, in-part segmentation, pose alignment, and motion transfer in low-data regimes. Unlike previous pairwise approaches, our solution constructs a joint embedding space, where both seen and unseen 3D shapes are implicitly aligned without further optimization. The code is available at https://graphics.tudelft.nl/SurfaceAware3DFeatures.
MotionDreamer
Exploring Semantic Video Diffusion Features for Zero-Shot 3D Mesh Animation
Animation techniques bring digital 3D worlds and characters to life. However, manual animation is tedious and automated techniques are often specialized to narrow shape classes. In our work, we propose a technique for automatic re-animation of various 3D shapes based on a motion prior extracted from a video diffusion model. Unlike existing $4 D$ generation methods, we focus solely on the motion, and we leverage an explicit mesh-based representation compatible with existing computer-graphics pipelines. Furthermore, our utilization of diffusion features enhances accuracy of our motion fitting. We analyze efficacy of these features for animation fitting and we experimentally validate our approach for two different diffusion models and four animation models. Finally, we demonstrate that our time-efficient zero-shot method achieves a superior performance re-animating a diverse set of 3D shapes when compared to existing techniques in a user study.
...
Animation techniques bring digital 3D worlds and characters to life. However, manual animation is tedious and automated techniques are often specialized to narrow shape classes. In our work, we propose a technique for automatic re-animation of various 3D shapes based on a motion prior extracted from a video diffusion model. Unlike existing $4 D$ generation methods, we focus solely on the motion, and we leverage an explicit mesh-based representation compatible with existing computer-graphics pipelines. Furthermore, our utilization of diffusion features enhances accuracy of our motion fitting. We analyze efficacy of these features for animation fitting and we experimentally validate our approach for two different diffusion models and four animation models. Finally, we demonstrate that our time-efficient zero-shot method achieves a superior performance re-animating a diverse set of 3D shapes when compared to existing techniques in a user study.
Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains. In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates. Code will be made available at https://github.com/lukasuz/Articulated-Point-NeRF.
...
Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains. In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates. Code will be made available at https://github.com/lukasuz/Articulated-Point-NeRF.