Feedforward 3D Shape Generation: Aligning Semantic Latents with Geometric Spaces
X. Han (TU Delft - Electrical Engineering, Mathematics and Computer Science)
P. Kellnhofer – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
X. Zhang – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Current text-to-3D generation frameworks are bottlenecked by extreme computational demands, relying heavily on expensive test-time optimization or massive datasets to learn multi-modal alignments. This research investigates exploiting the highly structured semantic latent space of pre-trained Large Language Models (LLMs) to directly predict continuous 3D shapes, bypassing the need to build intricate alignment networks from scratch. To bridge the topological mismatch between language and 3D space, we propose a decoupled, two-stage architecture utilizing continuous Signed Distance Functions (SDFs). First, an auto-decoder optimization framework explicitly aligns the semantic text space with the geometric shape space to ensure high structural fidelity. Second, a mapping network learns the non-linear projection between the frozen LLM text latents and the optimized geometric latents. Extensive evaluations demonstrate that this decoupled paradigm prevents the geometric collapse observed in fully joint, end-to-end training, and confirms that pre-trained LLM embeddings serve as superior geometric priors compared to random initializations. Ultimately, this methodology establishes a stable, optimization-free inference pipeline, enabling feedforward synthesis of 3D geometries directly from raw textual prompts.