Autonomous driving is a rapidly growing sector that is attracting increasing attention from industry and academia. The rise of deep learning techniques has made it possible for autonomous vehicles to perceive the environment around them, including detecting objects of interest around the vehicle. The technology used to detect objects is quite dependent on sensors such as cameras, radars, and LiDARs, which are commonly used in autonomous vehicles for 3D object detection. Current state-of-the-art 3D object detection methods often use a fusion of LiDAR and camera features. As such, the perception abilities of autonomous vehicles are susceptible to sensor corruptions, whether the corruptions are caused by internal sensor faults or extreme environmental conditions. A particularly hazardous corruption occurs when a LiDAR does not record points in certain regions, which can be caused, among other reasons, by dirt accumulation on the LiDAR, a wet ground, and dark-colored objects, which can absorb LiDAR beams or reflect them away from the LiDAR. Corruptions to the LiDAR can be very dangerous as they contribute significantly to the ability of multi-modal 3D object detectors, and the robustness of multi-modal 3D object detection methods against such corruptions is understudied. In this thesis, I propose Ada-UniBEV, which detects missing laser beam returns for the LiDAR based on its geometric pattern of shooting beams, and determines a severity of corruption based on the number of missing beams. Camera features corresponding to the corrupted region of the LiDAR are weighed as more important, with the exact weight being determined as a function of the severity of corruption. The weights also vary spatially around the autonomous vehicle depending on the properties of each spatial location. Ada-UniBEV improves the 3D object detection performance over state-of-the-art UniBEV in moderate to high-severity LiDAR corruption scenarios involving missing points while maintaining the same performance on clean data.

Detecting nearby vehicles involves utilizing data from various sensors installed on a car as it moves. Common sensors for identifying nearby vehicles include LiDAR, cameras, and RADAR. However, all of these sensors suffer from the same issue -- they cannot detect an approaching vehicle that is not yet visible. Hence, this thesis explores the potential of using a microphone array -- an array of sensors capable of detecting vehicles that are out of sight. Exploring prior research on detecting obstructed vehicles using sound reveals an existing model capable of detecting nearby vehicles approaching from behind blind corners. However, as the local geometry around the ego vehicle affects the perceived sound patterns, this model was only designed to work within a specific set of T-junctions. Therefore, the thesis aims to take a step further and develop a detection model capable of detecting vehicles behind blind corners in environments not included in the training set of the deployed model. This is challenging for multiple reasons. First, literature review revealed a lack of suitable datasets comprising sounds from approaching vehicles behind blind corners within various road junctions. In addition, microphones, like other sensors, come with limitations. Sound inherently provides less spatial information compared to commonly used sensors in autonomous driving, such as LiDAR or cameras. Considering sound propagation variations in different road junction geometries, building a model adaptable across diverse junction types presents a challenge. To overcome the data scarcity and sound's inherent spatial limitations, the study investigates the potential of employing simulated acoustic responses within artificial road environments as training data for real-world vehicle detection. Simultaneously, to complement the sounds inherent advantage of detecting objects that are out of sight, the thesis proposes to use a Bird's Eye View (BEV) encoding of the top-down map from the driving vehicle's perspective. Having an encoding of the top-down map of the current driving environment would allow a detection model to expect sound signatures commonly observed within a given setting. Overall, the assessment of acoustic simulations could not outline a singular configuration of simulation properties allowing realistic sound propagation for any kind of considered junctions when hearing an approaching vehicle. However, it was observed that the utilization of specific simulation parameters can result in realistic sound propagation within the given junction. Subsequently, evaluating a novel BEV encoding within the newly proposed acoustic detection pipeline demonstrated either equivalent or superior performance compared to a model relying solely on sound. Overall, this research underscores the potential of incorporating BEV encoding in non-line-of-sight acoustic detection and suggests the promise of acoustic simulations within the field. This study contributes to advancing the integration of sound as an additional data modality in vehicle detection.

This thesis introduces a Learned-Based Multi-View Semantic Stereo method, addressing the limitations of traditional and learned-based Multi-View Stereo (MVS) techniques in reconstructing reflective and low-textured regions, particularly prevalent in 3D models of buildings. Traditional methods lack completeness, while learned-based methods struggle with accuracy. Focusing on enhancing 3D models of buildings, this research integrates semantic information into the existing deep learning architecture for depth prediction, specifically CasMVSNet, to guide the reconstruction process. Three key strategies are employed: first, the incorporation of semantic maps into the network through a multi-modal approach; second, the introduction of a multi-modal refinement module at the end of the CasMVSNet model to improve the initial output depth maps; and third, the introduction of two new loss terms designed to enforce varying degrees of smoothness on specific semantic categories. Experimental results, conducted on the DTU dataset, demonstrate a significant enhancement in accuracy at the point cloud level while maintaining the completeness of the reconstructed models. Validation and generalization on the ETH3D dataset show consistent patterns. This research showcases the potential of integrating semantic guidance in 3D reconstruction of buildings, advancing the field of computer vision.