Material-informed Gaussian Splatting for 3D World Reconstruction in Digital Twin

Abstract

3D reconstruction for Digital Twin has become crucial, providing controlled and scalable environments for developing and validating advanced sensor algorithms before real-world deployment. Traditional reconstruction approaches have several limitations: they often overlook physical rendering properties of materials, lack detailed textures, and rely on lidar sensors that require complex calibration and perform poorly with capturing retro-reflective and transparent surfaces. We propose a modular, camera-centric, and material-informed 3D Gaussian Splatting (3DGS) pipeline for reconstructing large 3D scenes. Our approach extracts semantic material masks using segmentation models, converts Gaussian representations to explicit mesh surfaces, and automatically projects 2D material labels onto 3D geometry. This combines photorealistic reconstruction with physics-based material assignment for accurate sensor simulation and rendering in modern graphics engines and simulators. Evaluation on an internal Siemens autonomous driving dataset demonstrates that our material-informed approach achieves sensor simulation fidelity comparable to lidar-based ground truth while providing high visual fidelity, as validated through image similarity metrics and lidar reflectivity analysis.