Autonomous driving relies heavily on cameras and LiDAR for 3D perception, yet these vision-based sensors face limitations under poor illumination, adverse weather, or occlusion. Inspired by human hearing, we explore whether microphone arrays can enhance vehicle perception. We propose SonicVision, the first bird’s-eye-view (BEV) acoustic detection framework that jointly localizes and classifies traffic participants using sound alone. Our method employs a horizontally arranged 32-channel microphone array and transforms raw waveforms into short-time Fourier transform (STFT) features augmented with positional embeddings. A ResNet-based architecture is trained with novel Gaussian label representations to predict class-conditioned direction–distance distributions. To support this study, we collect three datasets (simulation, test track, and real road) with synchronized audio and LiDAR, where LiDAR detections serve as pseudo-labels. Experiments show that SonicVision significantly outperforms beamforming-based baselines, achieving accurate localization and classification performance. In some cases, our approach is able to identify objects that are missed by LiDAR, suggesting its potential as both an independent sensor and a complementary modality. These results provide the first evidence that low-cost microphone arrays can meaningfully contribute to 3D perception for autonomous vehicles.

Abstract—Crowd-sourced imagery is increasingly important for urban mapping and visual localization. However, its reliability is limited by GPS inaccuracies and heterogeneous capture condi- tions, including device variability, viewpoint differences, illumi- nation changes, and temporal shifts. In these settings, achieving metric-scale pose estimation remains a central challenge. Deep Learning-based pose estimation models address this problem by learning to estimate the 6-DoF pose using geometric cues between image views and metric supervision during training on large datasets. This encourages spatial consistency and sup- ports generalization across diverse conditions. Recent learning- based architectures, often based on vision transformer encoders, approach the task through unified multi-task frameworks that jointly predict metric depthmaps and 2D–2D correspondences, with relative pose estimated downstream. This thesis evaluates whether such frameworks predict accurate metric depthmaps under domain shifts. Experiments show that, even with scale correction through data-driven fine-tuning with metric supervi- sion, depth predictions from multi-task relative pose estimation models fail to generalize reliably to out-of-domain environments. In contrast, monocular models, trained on significantly larger and more varied datasets, demonstrate strong zero-shot reliability for metric depth prediction. A hybrid pipeline is proposed that combines the geometric consistency of relative pose models with the stable metric cues of monocular models, enabling robust pose estimation in crowd-sourced outdoor environments.

Autonomous driving is a rapidly growing sector that is attracting increasing attention from industry and academia. The rise of deep learning techniques has made it possible for autonomous vehicles to perceive the environment around them, including detecting objects of interest around the vehicle. The technology used to detect objects is quite dependent on sensors such as cameras, radars, and LiDARs, which are commonly used in autonomous vehicles for 3D object detection. Current state-of-the-art 3D object detection methods often use a fusion of LiDAR and camera features. As such, the perception abilities of autonomous vehicles are susceptible to sensor corruptions, whether the corruptions are caused by internal sensor faults or extreme environmental conditions. A particularly hazardous corruption occurs when a LiDAR does not record points in certain regions, which can be caused, among other reasons, by dirt accumulation on the LiDAR, a wet ground, and dark-colored objects, which can absorb LiDAR beams or reflect them away from the LiDAR. Corruptions to the LiDAR can be very dangerous as they contribute significantly to the ability of multi-modal 3D object detectors, and the robustness of multi-modal 3D object detection methods against such corruptions is understudied. In this thesis, I propose Ada-UniBEV, which detects missing laser beam returns for the LiDAR based on its geometric pattern of shooting beams, and determines a severity of corruption based on the number of missing beams. Camera features corresponding to the corrupted region of the LiDAR are weighed as more important, with the exact weight being determined as a function of the severity of corruption. The weights also vary spatially around the autonomous vehicle depending on the properties of each spatial location. Ada-UniBEV improves the 3D object detection performance over state-of-the-art UniBEV in moderate to high-severity LiDAR corruption scenarios involving missing points while maintaining the same performance on clean data.

Visual Place Recognition (VPR) remains a challenging problem, particularly under difficult conditions such as night-time or winter weather, which are often underrepresented in existing training datasets. Although transformer-based models have recently advanced the state-of-the-art, their high computational demands can hinder deployment in real-world robotic systems. This thesis proposes a new data augmentation strategy for VPR using image-to-image Vision Foundation Model InstructPix2Pix to generate realistic visual variations such as night and snow scenes from the original training data. These synthetic augmentations are added to the original training dataset to extend dataset diversity without requiring additional data collection. To further improve performance, the method is combined with more advanced augmentations using the Kornia library, which already improves robustness over traditional augmentation techniques. Experiments on multiple benchmark datasets show that lightweight, ResNet-based models trained with our VFM augmentations achieve significantly improved performance under challenging visual conditions. Additional ablations demonstrate the importance of careful prompt design and hyperparameter tuning. Overall, this work shows that VFMs can serve as practical tools for targeted dataset augmentation, improving the robustness of existing VPR methods in difficult scenarios.

Visual counting is an important task in computer vision with broad applications in areas such as crowd monitoring, agriculture, and environmental analysis. While deep learning has significantly advanced this field by enabling models to learn robust feature representations, deep learning approaches suffer from sensitivity to data imbalances, which occur in the distribution of object counts across counting datasets as a result of annotation effort. Most state-of-the-art counting models, categorized into clustering-, detection-, regression-, and density estimation-based methods, are built upon Convolutional Neural Networks (CNNs) and Transformers, both of which are known to be susceptible to imbalances in the training data. This study introduces a hybrid model that incorporates a programmatically guaranteed counting mechanism using the RASP language and the Tracr compiler, enabling the construction of Transformer-based models that can reliably execute predefined tasks, such as counting. By combining this exact counting mechanism with a trainable embedding module, we present a model that is capable of learning to count various tokens, even under significant data imbalance. We validate our approach on a synthetic, imbalanced dataset and compare its performance, training time, and data efficiency against standard CNN- and Transformer-based models. Results suggest that our method achieves strong generalization across the full spectrum of object counts while requiring less training data, highlighting the potential for this architecture to be further investigated and adapted to be used for robust and efficient visual counting.

Perception is a fundamental component of autonomous and self-driving vehicles, with reliable object detection and understanding of the environment being critical for safe operation. While lidar and camera based systems are widely used, radar remains a promising option due to its robustness in poor weather conditions and ability to directly measure the radial velocity of objects via the Doppler effect. However, radar’s sparse data and resulting limitations have constrained its potential. This thesis investigates the use of dual automotive radar setups to mitigate these limitations, and use the specific advantages of such a setup to improve full velocity vector estimation methods. A novel algorithm is proposed to achieve more accurate velocity estimation in non-ideal, real world conditions. The work further explores how improved velocity estimation can be used to improve classification performance using graph neural networks. Here it was found that including velocity information via a ground truth method did increase classification performance significantly, though the same result could not be obtained via the previously mentioned velocity estimation method. To support evaluation, a simplified simulation environment and ground truth velocity data for the RadarScenes dataset are developed. This research aims to close the performance gap between radar and other more data-dense sensors, offering a robust and more cost effective alternative, especially in conditions where optical systems under perform.

Accurate and up-to-date road maps are vital for Automated Vehicles (AVs) to navigate urban environments safely and predict the behavior of surrounding agents. However, generating such maps typically requires manual annotation or depends on expensive sensor-equipped vehicles, limiting scalability. Aerial imagery offers a more scalable alternative, but existing methods either fail to capture both geometric and topological details or do not generalize well to unseen urban areas.
This thesis addresses these challenges by introducing SAM-Maps, a novel framework that automatically generates road maps from aerial imagery without requiring additional model training. Leveraging foundation models, SAM-Maps extracts both the drivable area geometry and road connectivity of urban environments.
Experiments on the View-of-Delft Prediction dataset demonstrate that SAM-Maps achieves a recall of 43.4% in a fully automatic mode, improving to 75.6% with minimal manual steps. Additionally, trajectory prediction experiments using the state-of-the-art Wayformer model show a 37.9% improvement on the minADE6 metric when incorporating SAM-Maps, compared to scenarios without map input. To the best of our knowledge, this is the first method to extract both drivable areas and road connections for European urban environments from aerial imagery, providing a scalable solution for road map generation.

Plastic pollution in rivers is a growing environmental issue with widespread impacts. Monitoring the movement of plastic waste across different river systems is challenging due to environmental variability and the limited availability of labeled data. This thesis investigates camera-based methods for detecting floating macroplastics in rivers and explores ways to adapt detection systems to new locations with minimal data. Collecting data from the Limmat River in Zurich and the uMhlangane River in Durban, South Africa, the study assesses the impact of domain shifts on detection performance and proposes a semi-automated annotation pipeline to improve labeling. Furthermore, it tests techniques like few-shot learning and pseudolabeling to address the performance dip. The results show that model performance decreases significantly when applied to new locations but that even with minimal data, camera-based monitoring can provide useful insights for understanding waste movement and informing plastic waste management strategies.

Temporal segmentation of medical procedures holds the potential to improve patient safety, provide decision support to clinicians, and serve as the basis for context-aware robotic assistance systems. However, clinical adoption remains hindered by two key challenges: the scarcity of annotated data and limited generalizability across diverse surgical settings. This thesis therefore explores 2D skeleton-based temporal segmentation as a privacy-preserving and data-efficient alternative to conventional RGB-based methods. Using the CAG-skeleton dataset, which consists of pose sequences extracted from external cardiac angiography (CAG) recordings, the study investigates various model architectures and limited supervision strategies for identifying 14 procedural phases.

A two-stage framework, combining a skeleton-based feature extractor with a temporal model, was adopted. A review and comparison of proven models revealed combinations of PR-GCN or MS-G3D feature extractors with LSTM or TCN temporal models to hold the most promise in the low-data medical domain. After training all combinations on low-data subsets of the CAG-skeleton dataset, it was found that all models outperformed a non-learning baseline model, which always predicts the mean procedure. Between the learning models, clip-wise segmentation accuracy differences held no statistical significance, but LSTM-based models showed a statistically significant superior understanding of sequential order. Considering both sequential metrics and computational efficiency, the PR-GCN + LSTM combination was selected for extensive evaluation, achieving a clip-wise segmentation accuracy of 83.95\% when trained on 146 CAG procedures.

To further address the data scarcity challenge, two limited supervision approaches were explored. Transfer learning using the Kinetics-skeleton dataset showed no statistically significant performance gains, suggesting that the knowledge learned from Kinetics-skeleton does not effectively transfer to the surgical domain, and/or that the information transferred is relatively easy for the model to learn from scratch during training. In contrast, pseudo-labeling via class-balanced self-training showed great potential for reducing annotation requirements as it provided consistent improvements to the models' clip-wise segmentation accuracy in the low data regime.

Overall, this study introduces skeleton-based representation as a modality holding large potential for medical temporal segmentation and highlights pseudo-labeling as an effective strategy for reducing annotation requirements.

Visual place recognition (VPR) is a form of visual localization. Current approaches are designed to handle common VPR challenges, such as appearance and viewpoint variations. With the introduction of DINOv2, vision foundation models have been used as feature extractors to improve performance for VPR techniques, as they show great generalizing capabilities for image representations. By fine-tuning these large models on VPR-specific datasets, performance increases even more. A problem with these big VPR datasets is the bias towards urban environments. To solve this problem, we propose to use a simple pipeline to fine-tune existing techniques on the reference databases of test datasets. Our experiments show that performance improves by reference database fine-tuning for multiple techniques on different datasets. To handle appearance and viewpoint variations as well, image augmentations can be used during training. With this complete pipeline, techniques improve performance. The experiments show improvement even if a large query-reference domain gap exists for that dataset given that a part of the test queries are know during fine-tuning.

The training process of machine learning models for self-driving applications suffers from bottlenecks during loading and processing of LiDAR point clouds with large storage complexity.
Many studies aim to remedy this problem from an implementation perspective by developing efficient data loading and processing pipelines.
This study, on the other hand, explores an alternative approach by augmenting data representations to achieve lower storage complexity known as point cloud compression.

A broad analysis is presented on novel point cloud compression codecs using tensor decomposition methods.
Several point cloud representations and tensor decomposition methods are considered over a range of hyperparameter choices and compression values.
In order to assess the performance of the presented codecs: the compression rate, quality of the reconstruction, and time complexity is compared to the octree-based baseline model: TMC13.
Compared to the baseline model, the performance of the presented tensor decomposition-based codecs falls short. One of the presented codecs does notably outperform the others. This codec uses synthetic tensorization followed by sorting using z-location and decomposition using the TT-SVD algorithm.
Sorting by z-value isolates the ground plane, which is a dominant low-rank feature, which can effectively be decomposed using the TT-SVD algorithm yielding adequate results.

Several limitations of the presented tensor decomposition-based codecs are: the omission of bitwise compression on the factor matrices, and the trade-off between bitwise precision and truncation due to tensor decomposition.
Future work could improve in these areas along with considering the use of different heuristics and optimizing the tensor network topology.

The ever-increasing complexity of Artificial Intelligence (AI) models has led to environmental challenges due to high computation and energy demands. This thesis explores the application of tensor decomposition methods—CP, Tucker, and TT—to improve the energy efficiency of large Convolutional Neural Networks (CNNs) during inference by reducing energy consumption. The energy consumption of several convolution layers was measured using a watt meter across various CNN configurations and different hardware architectures (Central Processing Unit(CPU) and Graphics Processing Unit (GPU)). In addition, several regression models were fitted to estimate energy savings, incorporating memory usage. It was found that TT decomposition consistently provided the most significant energy savings across various compression ratios, influenced by CNN hyperparameters such as input/output channels, feature sizes, and kernel sizes, whereas CP decomposition was the least effective in reducing energy. The GPU implementations generally resulted in additional energy consumption, and the GPU regression models suggested a need for more complex relationships. The thesis also revealed that the efficiency of tensor decompositions might be highly dependent on the implementation details of software libraries, such as TensorlyTorch, which can significantly impact the computation and memory complexities. These findings underscore the importance of both hardware specific considerations and careful software implementation in achieving energy efficient CNNs, providing a foundation for further research in energy-constrained environments.

The shift to sustainable energy sources has increased demand for Energy Transition Metals such as nickel, copper, cobalt, and manganese. To satisfy this need while reducing the negative social and environmental effects of conventional mining, Deep-sea Nodule Collection (DSNC) appears to be a feasible option. Despite its potential, DSNC encounters difficulties with path planning since it depends on imperfect global paths, which result in inefficiencies and safety issues. This thesis looks into the possibility of using a local path planning algorithm, specifically an Interpolating Curve Planner (ICP).

To assess the ICP's performance under various model hyperparameters and environmental scenarios, a simulation tool is created. The study examines how well the ICP performs in terms of area coverage, collection rate, and reliability along different hyperparameters and presurvey measurement resolutions. It integrates novel production rate objectives, previous track following, and real-time sensor measurements within an environment generated from bathymetric data.

The examined local path planner shows a step towards optimised DSNC. Further research is needed to refine objectives and the structure of the local path planning algorithm to maximize its potential in addressing DSNC challenges effectively.

Automated driving has immense potential for improving road safety. Over the past decades, extensive research has been conducted in this field. Although the technological capability for highly automated driving exists today, its widespread application is not yet present. One major limiting factor of current automated driving solutions is that vehicle localization heavily relies on high-definition maps (HD maps), which are highly expensive to construct and maintain. This dissertation focuses on developing a more scalable solution for vehicle localization. It explores a novel technique that estimates the ego vehicle’s pose (location and orientation) by matching ground-level images captured by the vehicle’s onboard camera to publicly available geo-referenced aerial imagery...

This work addresses visual localization of intelligent vehicles as an alternative to traditional GPS- of HD map-based localization options. Specifically, the problem of Cross-View Pose Estimation (CVPE) is explored, which involves estimating the vehicle pose within an encompassing aerial patch, given a ground image from the on-board camera feed. The aerial patch containing the ground truth pose can be obtained through a rough localization prior, such as GPS. We find that existing CVPE methods start with a location prior that is too coarse given both the GPS performance and the required localization error. Therefore, we define a fine-grained localization setting and propose three approaches, targeting performance, interpretability, and data efficiency. Furthermore, the approaches have a unique capacity to predict a 6-DoF camera pose. Two approaches involve matching point-level local features in 3D space using a novel point cross-attention, while the last one aims to tailor an existing dense feature matching method to the fine-grained setting. Despite quantitative performance of the local feature matching approaches being inferior to the state-of-the-art, we establish a new state-of-the-art on the fine-grained setting with the improved dense-feature baseline. Nevertheless, we show the key limitations of the local feature matching, namely the influence of the “unmatchable” queries. Furthermore, using a 6-DoF projective transformation we discover severe issues with the ground truth quality on the KITTI dataset, commonly used in CVPE literature, potentially accounting to the large degree to the substandard performance of most available CVPE methods. Finally, our local feature matching methods demonstrate the capability of predicting pitch and roll angles of the camera, estimating which has not yet been attempted in CVPE.

Tracker-level fusion (TLF) is recognized as an effective approach to comprehensively improve visual object tracking performance by combining the capabilities of multiple baseline trackers. Although there is considerable interest in TLF, there are still challenges related to insufficient understanding, high cost, and unstable performance that make studying TLF difficult. In this thesis, I begin with an explicit summary of the overall pipeline of TLF which has significant guidance for TLF study. Additionally, I conduct a deep analysis of the positive and negative effects of baseline trackers to fully understand their influence on TLF. For visual object tracking, I propose three TLF frameworks based on three Distributed Kalman filters which are optimized for different scenarios and enable the fusion of different baseline trackers to enhance tracking performance. My TLF frameworks fuse the tracking results of different baseline trackers based on the principle of minimal trace to produce fusion results.Additionally, they exhibit superior and stable performance with general baseline tracker requirements, while also being simple, online, and real-time.Furthermore, the proposed frameworks can benefit from state-of-the-art baseline trackers over time, which will further improve their tracking performance. The proposed analysis and frameworks are studied extensively on three challenging benchmarks: generic tracking OTB2015, short-term tracking GOT-10k, and long-term tracking LaSOT. At over 240 FPS, the state-of-the-art success AUC score of 72.7% is achieved on OTB2015.

Inadvertent bird nest destruction by autonomous mowing machines poses significant threats to the breeding success of meadow birds. Drone-based detection methods represent the current state-of-the-art for bird nest localization to attain mower circumvention. However, they only identify 80% of bird nests with average localization error of 3.344 meters and are restricted to specific application times. This paper introduces alternative, fully automated nest localization systems integrated with autonomous mowers. Two strategies are proposed, 1) Directly detecting bird nests using thermal data, or 2) Indirectly, by tracking birds and extrapolating their trajectories back to their nests using RGBD data. These methods were validated with warmed chicken eggs hidden in grasslands and with drones simulating bird flight. YOLOv8 models were modified for both approaches. The thermal localization method is able to detect all bird nests with an average confidence of 73.4%. It allows for real-time localization and yields one unnecessary nest circumvention for every ten bird nests saved due to false positives. This method is shown to be effective in all breeding season temperatures, both day and night. Conversely, the trajectory extrapolation method detects birds with an average confidence of 82.2% and has localization error of 0.794 meters. Birds taking flight prematurely or from locations other than nests impact the number of bird nests saved and the number of unnecessary circumventions. It is demonstrated that this method fails to detect birds during nighttime. In conclusion, an automated thermal-based localization system integrated with autonomous mowers outperforms both RGBD- and current state-of-the-art drone-based methods. This study highlights therefore the potential of thermal-based solutions for bird nest protection in grasslands.

Robots are increasingly deployed in various locations to automate tasks, including in barns. However, in barns cows can obstruct the sensors such as LiDAR or camera, leading to a lack of environmental information. As a result, the robot’s localization system only relies on odometry at those moments, introducing additional uncertainty to the robot’s pose. When the visibility of the environment is restored, the robot may mistakenly believe it is in a location that does not correspond to its actual position.
The first contribution of this master thesis is a novel method for improving the self-localization in barns by implementing a line detection algorithm which is called Line Adaptive Monte Carlo Localization (LAMCL). The novelty is that only line segments are used to detect a localization error instead of corners between different line segments. It also retains the robot’s current pose to filter out fault-detected localization errors. In addition, the new approach is applied in a dynamic environment. The proposed method combines the Split-and-Merge line detection algorithm with AMCL. The detected line segments are compared with the walls in the environment to identify localization errors. When an error is detected, the robot’s pose is adjusted by placing a section of the particles at the location of the error. In this way, the robot can find its true location again.
The second contribution is a new dataset. This new dataset, called DataCow, consists of four recorded routes in a barn with GT on a handful of spots to evaluate the self-localization. DataCow includes the pseudo-2D LiDAR scans and the odometry of a robot driving through a barn. This dataset is used to evaluate the new self-localization method LAMCL. Through the experiments, it has been discovered that this new method improves the system’s recovery ability, but the accuracy and precision are compromised.

We explored the possibility of improving cross-view matching performance with self-supervised learning techniques and perform interpretations in terms of the embedding space of image features. The effect of pre-training by contrastive learning is verified quantitatively by experiments, and also exhibited by visualization of the feature space.

The automotive industry currently has been working on developing various levels of autonomy to assist in different Advanced Driver Assistance Systems (ADAS) with the ultimate aim of moving closer to the realization of an autonomous vehicle. For such ADAS, the industry has been using multiple sensors like Cameras, Radar, LiDAR, etc. LiDAR has been at the forefront of the research as it provides extremely rich and precise information about the surrounding. In research Motorized Optomechanical LiDAR has been used for almost a decade now. However, it can not be deployed in mass-produced vehicles because of how expensive the sensor is, thus, limiting it to academia without practical use. Therefore, the industry has been working on a different category of LiDAR namely Microelectromechanical Systems (MEMS) LiDAR. Which uses solid-state technology instead of servo motors. Thus, they are compact and much cheaper in cost, making them feasible to deploy in mass-produced vehicles.

There has been a considerable amount of research already done on Motorized Optomechanical LiDAR, therefore, one must take advantage of it to learn about MEMS LiDAR. The domain of Transfer Learning provides an opportunity to learn from the accumulated knowledge of Motorized Optomechanical LiDAR (Source Domain) and use it to reduce the learning time on MEMS LiDAR (Target Domain). Thus, reducing the effort, cost, and time to research MEMS LiDAR technology.

This thesis explores the Domain Mapping strategy of transfer learning for point clouds. A scan pattern-based domain mapping approach has been described in this thesis to reduce the domain gap between the source and target domain. Moreover, a point cloud densification pipeline that utilizes a depth completion network has been described to further reduce the gap between the two domains. Since semantic segmentation is one of the most actively researched tasks in the industry as it provides the researcher with in-depth information about the environment which could be used in activities like lane detection, parking assist, etc, therefore, this thesis utilizes it as the proxy for evaluating the performance of the domain mapping strategy. It compares a few domain mapping strategies like cropping the Field of View, an adaptation based on scanning pattern, and densification & adaptation to simulate the density of points in the source domain similar to the target domain. This thesis verifies that domain mapping strategy is a suitable solution to learn from Motorized Optomechanical LiDAR and use it to enhance performance on MEMS LiDAR and therefore reduce the data required for research of MEMS.