This paper presents a method that shows how a lidar-based 3D static environment can be constructed from a driving scenario and is used to aid in the creation of a digital twin from that scenario. Built with limited computational resources in mind, the resulting 3D static background reconstruction method is lightweight and nonetheless up to par with similar works, proving itself to be a viable alternative to similar methods. It uses ground truth labels and open-source, out-of-the-box working building blocks to create the pipeline that filters the lidar frames, performs registration between those frames, and meshes the resulting point cloud into a 3D mesh of the static background. We performed experiments in the Siemens Prescan ADAS Simulator of a reenactment of the traffic scenario in combination with our 3D static background and measured the inference domain gap using the Bidirectional Chamfer Distance and a real-data trained object detector, comparing both the synthetic and original data. Using Prescan as the simulator, AD models van be evaluated closed-loop. This work opens up the possibilities of applying sensor domain shifts to existing data or creating reality-based traffic scenarios from existing traffic scenarios and their corresponding 3D static background of edge cases that do not, or too little, exist in real-world data.

Training models for autonomous vehicles (AVs) necessitates substantial volumes of high-quality data due to the strong correlation between dataset size and model performance. However, acquiring such datasets is labor-intensive and expensive, requiring significant resources for collection and labeling. To optimize the utility of available data, augmenting the dataset or generating synthetic data presents a cost-effective and efficient solution. Traditional methods that operate within the RGB domain frequently overlook crucial information, such as object frequency, scene composition, and agent trajectories. To address these limitations, a pipeline employing controllable diffusion models and vehicle simulation software is proposed. This approach involves loading collected data into a physics-based simulator, which allows for augmentation beyond the pixel space into the structural space. The augmented simulated data is subsequently transformed back into the photorealistic domain using generative artificial intelligence. This process generates high-fidelity synthetic data, thereby enabling models to train effectively on an expanded and varied dataset, enhancing robustness through the increased variation. The proposed method is evaluated in both image and video domains to assess its effectiveness.

Autonomous driving is attracting growing attention because of the potential advantages it poses on safety, leisure, energy efficiency, reduced emissions, and traffic reduction. Current research is focusing on areas related to artificial intelligence to solve complex planning and decision-making tasks, object detection, or simultaneous localization and mapping.

However, training and testing these methods in the real world is rather unsafe and can be harmful to other road users. For that reason, the focus is set on developing such methods first in a simulated and controlled environment and then transferring the control strategy to the target domain. The issue with this approach is that the performance drops when the domain changes, a problem known as sim2real. In addition to this, certain methods cannot be deployed to real-time applications as the computational complexity of the techniques is too large to produce outcomes in a constrained time frame.

To date, research on reducing the sim2real for autonomous driving gap has focused on domain adaptation or domain randomization techniques, however, none has combined them with a high-fidelity vehicle dynamics model. This work aims to validate a safe transfer learning approach that reduces the reality gap in a zero-shot manner by combining the advantages of using simulated and real-world data, a digital twin of the vehicle, and a traffic scenario simulator.

To test this framework reinforcement learning agents are trained to track different paths. Further evaluation includes the influence of using a high-fidelity model and real-world data. Furthermore, the effect of the reward function on the control strategy of the learning agents is also examined. The training process of this transfer learning framework starts with virtually generated scenarios, which are simpler than their real-life counterparts and noise-free. The motion of the learning agent is simulated with the digital twin and in every episode, its parameters are randomized. Moreover, noise is added to the control action. These randomizations are useful for the controller to operate under uncertainty and avoid overfitting to model inaccuracies. Once the performance saturates, real-world logged data is included so that the learning agent adapts to the target distribution, i.e. on noise levels and driving style. After the performance stops improving, the results are tested in Model-in-the-Loop and Vehicle-in-the-Loop with the SimRod, an all-electric drive-by-wire vehicle.

The results of this study indicate that using this zero-shot transfer learning framework yields much better tracking accuracy (about 30% on average) in the physical world with respect to controllers trained only with lower-fidelity models or synthetically generated data. This study stresses that a combination of the three sim2real methods and synthetically generated and real-world data are necessary to reduce the reality gap. Additional results include the influence of the reward function design and using different reinforcement learning algorithms (proximal policy optimization and soft actor-critic).