Raw radar perception is emerging as a promising approach for all-weather autonomous driving. While current deep learning approaches have made significant efforts using established vision-based architectures, they often rely on localized spatial convolutions that do not fully exploit the underlying wave-based physics of radar signals, such as phase coherence and global frequency structures. To harness this potential, we propose WaveRadNet, an architecture that introduces two physics-aligned novelties: (1) a complex-valued MIMO pre-encoder to preserve native phase information, and (2) a hierarchical Fourier-domain encoder based on Global Filter Network to enable un-truncated global feature mixing.

Integrating these complementary modules improves the baseline model (FFTRadNet), achieving state-of-the-art performance on the RADIal dataset with an overall 95% F1 score and 81% mIoU. On the Hard split, WaveRadNet delivers a 90% F1 score, an absolute improvement of more than 11% over the baseline. When configured with a reduced number of active antenna pairs, our sub-sampled network still outperforms the full-array baseline. Finally, WaveRadNet improves perception edge cases, particularly for long-range and off-boresight targets. ... Read more

We present AutoBox, a 4D auto-labeling pipeline that systematically composes vision foundation models to generate dense 3D annotations for autonomous driving. To address the limitations of sparse point clustering and cross-camera fragmentation, AutoBox operates in a unified multi-modal space. The pipeline first maintains object identity across the surround-view by stitching overlapping panoramic views. It then discovers instances via open-vocabulary video segmentation and geometrically reconstructs them by densifying sparse LiDAR data into metric-scale point clouds. Finally, prototype-based visual feature alignment estimates orientation, resolving the ambiguity of geometric box fitting. Unlike methods that require scene flow, high-definition (HD) maps, or iterative training, AutoBox matches each sub-problem to a specialized, pre-trained foundation model. On nuScenes, AutoBox achieves an average precision of 42.0 and a nuScenes Detection Score of 38.0, demonstrating competitive detection performance and state-of-the-art orientation accuracy among auto-labeling methods. ... Read more

This thesis addresses the problem of vision-based tracking of Vulnerable Road Users (VRUs) such as pedestrians and riders in the context of Automated Driving (AD). The work primarily builds on camera-based perception, while also considering LiDAR as a complementary sensor. Robust tracking of persons is a critical perception task, as it enables downstream modules for motion prediction and safe planning. The motivation stems from the disproportionate involvement of VRUs in traffic fatalities, as globally pedestrians and riders account for more than half of all road deaths, particularly in urban environments. By providing temporally consistent trajectories and identity preservation across occlusion and re-entries, tracking contributes to safer and more anticipatory automated driving systems.

The thesis first covers an extended review of the state-of-the-art in multi-object tracking. A modular framework is outlined to describe traditional tracking pipelines. The review then focuses on deep learning-based methods. Early methods typically extended classical pipelines with deep features, while later works moved toward end-to-end formulations and transformer-based architectures. This survey provides the methodological foundation and highlights the transition toward data-driven approaches.

As these approaches rely on large amounts of data for both training and evaluation, fully manual annotation becomes increasingly challenging and costly, as the required effort grows rapidly with dataset size. To address this challenge, the thesis develops a semi-supervised trajectory generation pipeline. The approach leverages camera detections, LiDAR-based estimation of 3D locations from 2D boxes, and vehicle inertial sensing for ego-motion compensation. Based on this data, the approach generates temporally dense pseudo ground-truth (i.e. 2D bounding boxes, 3D person locations) from sparse, manual annotations at keyframes. Two complementary strategies are employed to generate tracks within each batch, whereby each batch is defined to begin and end at a keyframe. The first is a graph-based optimization for global association that leverages a spatio-temporal graph. The second is a tracklet-based approach that combines tracklet generation with subsequent tracklet merging. For both strategies, the outputs obtained within each batch are subsequently aggregated into complete trajectories, guided by the keyframe annotations. An experimental evaluation of these approaches, including comparisons to baselines, ablation studies, and ceiling analyses, demonstrates that the tracklet-based pipeline achieves the most robust overall performance. This approach significantly reduces annotation effort while producing temporally consistent trajectories suitable for training deep tracking models.

The EuroCity Persons (ECP) 2.0 dataset is introduced as a large-scale image dataset for person detection, tracking, and prediction in traffic. The dataset was collected onboard a vehicle driving through 29 cities in 11 European countries. It contains more than 220K unique person trajectories, in more than 2.0M images and comes with a size of 11 TB. ECP2.0 is about one order of magnitude larger than previous state-of-the art person datasets in automotive context. It offers remarkable diversity in terms of geographical coverage, time of day, weather and seasons. The pipeline proposed in this thesis is used to generate the dense annotations, providing temporally consistent pseudo ground-truth on top of the recorded image data. Experiments analyze the resulting annotation quality and object-dependent effects. The benefits of using the pseudo ground-truth for training and testing of state-of-the-art tracking methods are further demonstrated, showing improved tracking accuracy and robustness compared to training on sparsely annotated or purely detector-based supervision, while also highlighting remaining challenges in handling long-term occlusions and identity switches. The ECP2.0 dataset is made freely available for non-commercial research use.

Finally, this thesis improves person appearance embeddings as a means to strengthen identity association in multi-object tracking. To this end, a novel method for creating appearance embeddings to identify individual persons using an object Re- Identification (ReID) framework is introduced. CLFormer (Camera LiDAR Transformer) is presented, a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. The 3D Cuboid-Inclusive Point Embedding (3D-CIPE) is introduced, which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, various strategies for the early and late fusion of multi-modal input data are explored and analyzed. To evaluate the proposed CLFormer, the nuScenes dataset [14] is reinterpreted for ReID purposes and used for the experiments. CLFormer demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).

In conclusion, this thesis advances the state of-the-art in VRU tracking for automated driving by providing an extended methodological review, novel semi-supervised dataset creation techniques, a large-scale benchmark dataset, and a complementary ReID method that provides stronger identity cues for tracking. Together, these contributions offer new methodological insights, practical tools, and benchmarks that can support future research and development in the field of automated driving perception and tracking. Future work may focus on further improving the semi-supervised annotation method, particularly by reducing or even moving beyond the reliance on manual keyframe annotations, toward full unsupervised or self-supervised methods. In general, the long-term vision for multi-object tracking in automated driving points toward systems that unify perception, prediction, and decision-making in end-to-end frameworks, enabling robust and reliable operation in diverse and challenging real-world environments.
... Read more

Radar (Radio Detection and Ranging) sensors are cost-efficient and robust under adverse weather conditions, making them an attractive component in modern automated driving perception systems, but they provide significantly sparser information about the environment than camera or LiDAR sensors. Thus, to fully exploit radars in perception solutions, it is crucial to exploit not only local but also global contextual information of the scene. However, existing 4D radar models often struggle to fully exploit both types of information, resulting in suboptimal performance. This paper proposes DRIFT, a dual-representation model that effectively captures and fuses both local and global contexts through a dual-path architecture. The model incorporates a point path to aggregate fine-grained local features and a pillar path to encode coarse-grained global features. These two parallel paths are inter-fused via novel feature-sharing layers at multiple stages, enabling full utilization of both representations. DRIFT is evaluated on the widely used View-of-Delft (VoD) dataset and an internal dataset, demonstrating its state-of-the-art performance across multiple tasks, including object detection and free-road segmentation. Notably, DRIFT achieves a mean average precision (mAP) of 52.6% (compared to 45.4% from the CenterPoint baseline) on the VoD dataset, surpassing existing methods. ... Read more

Accurate and up-to-date road maps are vital for Automated Vehicles (AVs) to navigate urban environments safely and predict the behavior of surrounding agents. However, generating such maps typically requires manual annotation or depends on expensive sensor-equipped vehicles, limiting scalability. Aerial imagery offers a more scalable alternative, but existing methods either fail to capture both geometric and topological details or do not generalize well to unseen urban areas.
This thesis addresses these challenges by introducing SAM-Maps, a novel framework that automatically generates road maps from aerial imagery without requiring additional model training. Leveraging foundation models, SAM-Maps extracts both the drivable area geometry and road connectivity of urban environments.
Experiments on the View-of-Delft Prediction dataset demonstrate that SAM-Maps achieves a recall of 43.4% in a fully automatic mode, improving to 75.6% with minimal manual steps. Additionally, trajectory prediction experiments using the state-of-the-art Wayformer model show a 37.9% improvement on the minADE6 metric when incorporating SAM-Maps, compared to scenarios without map input. To the best of our knowledge, this is the first method to extract both drivable areas and road connections for European urban environments from aerial imagery, providing a scalable solution for road map generation. ... Read more

We present MoSIS (Motion-Supervised Instance Segmentation), a self-supervised framework that learns instance masks from unlabelled video. Our method uses the movement in videos to define masks, which are then used to train a YOLO model to segment these instances. This makes the method end-to-end with minimal assumptions. Unlike methods that require motion at inference, MoSIS performs single-image instance segmentation, so it can detect objects that are currently static (e.g., vehicles waiting at a red light). To systematically evaluate our approach, we developed a controllable synthetic dataset with ground truth masks and motion fields. Overall, MoSIS shows that it can train an instance-segmentation model from unlabelled video while requiring only a single RGB frame at inference. While supervised training still attains higher BBox mAP, our label-free approach creates usable instance masks and points to a practical route for reducing annotation cost in perception systems. ... Read more

Accurate sensor calibration is a critical challenge in the development of automated vehicles, especially in dynamic and modular sensor configurations. Traditional target-based methods, while precise, are limited in scalability and adaptability. In this work, we propose a modular, targetless, ego-motion-based calibration framework for multi-modal sensors, including a monocular camera, LiDAR, and 4D radar. The framework leverages odometry trajectories for extrinsic calibration, incorporating temporal alignment, trajectory scaling, and both pairwise and joint optimization techniques to achieve robust and accurate sensor alignment. Experimental validation using the View-of-Delft (VoD) dataset demonstrates the framework’s
robustness across diverse sensor setups, adaptability to real-world conditions. Our results underscore the potential of scalable, targetless calibration approaches to enhance the reliability and flexibility of automated systems, supporting implementation in real-world scenarios. ... Read more

In the context of open-world scenarios in autonomous vehicles (AVs), previously unseen classes may arise. To address this, effective extraction of well generalizable features is essential for AV downstream tasks, especially in the context of zero-shot learning. This can be achieved using transformers, and Swin Transformer in particular (vision backbone of most Vision-Language Models). However, to enable on-board applications, knowledge distillation must be utilized to create a lightweight model capable of real-time processing. We explore self-supervised knowledge distillation, given that AV datasets need to generalize to previously unseen classes. Our contributions include adapting existing CNN-to-CNN output-based self-supervised knowledge distillation algorithms to Transformer-to-CNN for benchmarking and enhancing them with a cross-architecture loss function. By leveraging DisCo, the best performing output-based self-supervised knowledge distillation method, and using EfficientNetB0 as the student model, we achieve a 3.9% relative improvement in top-1 accuracy over the supervised Swin-T teacher on our modified ImageNet for open-world classification, up to 5.0% with our loss. ... Read more

Unsupervised 3D object detection methods can reduce the reliance on human-annotations by leveraging raw sensor data directly for supervision. Recent approaches combine density-based spatial clustering with motion and appearance cues to extract object proposals from the scene, which serve as pseudo-annotations. However, density-based methods struggle with the uneven data densities seen in LiDAR point clouds, and fail to distinguish between foreground and background objects effectively. To address this issue, this thesis introduces MobileClusterNet, a learnable framework designed for 3D spatial clustering. MobileClusterNet incorporates a novel loss module which utilizes appearance embeddings alongside scene flow information, thereby learning to generate high-quality clusters consisting of both static and dynamic mobile objects. Annotations generated by MobileClusterNet can be used for training any existing supervised detector, without the need for extensive self-training. Experimental results on the Waymo Open Dataset demonstrate that MobileClusterNet outperforms traditional density-based methods like HDBSCAN in clustering performance by a large margin, and provides high quality proposals for training supervised detectors. ... Read more

Automated driving has immense potential for improving road safety. Over the past decades, extensive research has been conducted in this field. Although the technological capability for highly automated driving exists today, its widespread application is not yet present. One major limiting factor of current automated driving solutions is that vehicle localization heavily relies on high-definition maps (HD maps), which are highly expensive to construct and maintain. This dissertation focuses on developing a more scalable solution for vehicle localization. It explores a novel technique that estimates the ego vehicle’s pose (location and orientation) by matching ground-level images captured by the vehicle’s onboard camera to publicly available geo-referenced aerial imagery... ... Read more

Logistics and transportation can greatly benefit from the use of autonomous robots, such as self-driving vehicles. Robots can help to move goods or people without human supervision. One of the main components that enable autonomous navigation among humans is motion planning. Motion planning is responsible for computing a collision-free trajectory that moves the robot to its destination, based on the perceived information. The motion planner should be efficient, robust, and safe. This thesis contributes towards this goal by investigating the design of motion planning algorithms for autonomous navigation of mobile robots near humans.

Traditional motion planners for dynamic environments have two key limitations that this thesis aims to address. First, they assume that their model of dynamic obstacles (e.g., humans) is exactly correct, capturing it with a single deterministic prediction. In practice, the robot cannot observe human intentions and must account for its uncertainty about the human's future behavior. Second, motion planners usually compute a single trajectory around an obstacle as a result of previously taken decisions without exploring alternative options. They react slowly or even fail to find a solution when unpredicted changes make this path undesirable. This results in poor planning performance in dynamic environments.

The goal of this thesis is to develop motion planners that account for the uncertainty of human motion predictions and that are consistent and robust in their decision-making in order to deal with unpredicted changes in dynamic environments. To accomplish this goal, this thesis proposes two motion planning frameworks: scenario-based and topology-driven trajectory optimization.

The first contribution of this thesis is Scenario-based Model Predictive Contouring Control (S-MPCC), a real-time capable probabilistic planning framework that incorporates any uncertainty associated with the motion predictions of dynamic obstacles. Contrary to existing methods that only account for small variations around a single predicted trajectory (unimodal uncertainty), the proposed planner accounts for multiple possible trajectories (multi-modal uncertainty). The planner therefore safely accounts for several outcomes, for instance, to express that a pedestrian may or may not cross in front of the robot.

S-MPCC bounds the probability of collision in each time step with all obstacles through Chance-Constrained Optimization (CCO). The CCO is reformulated as an optimization without uncertainty by sampling trajectories from the predicted distribution, known as scenarios. Each scenario represents a possible position of all obstacles in one time step, and the planner avoids collisions with all scenarios. This Scenario Program (SP), through a tailored linearization, can be solved efficiently online. S-MPCC therefore plans probabilistic safe trajectories independent of the underlying distribution of the uncertainty.

S-MPCC considers the probability of collision separately for each time instance in the planned trajectory. The second contribution of this thesis, Safe Horizon Model Predictive Control (SH-MPC), builds on S-MPCC to constrain the joint probability of collision with all obstacles over the duration of the planned trajectory. Existing methods that separately constrain the probability of collision in each time step (temporal marginal) and with each obstacle (obstacle marginal) lead to overly cautious motion planning when safety constraints are enforced. SH-MPC formulates a single chance constraint to bound the overall probability of collision. This CCO is reformulated as an SP where each scenario represents a possible trajectory for all obstacles. To certify the joint probability of collision with the SP, the number of scenarios that affect the motion plan needs to be identified. SH-MPC estimates this quantity at a negligible computational cost during optimization. Consequently, SH-MPC plans trajectories in real-time under generic uncertainties that are less cautious than existing methods without compromising on safety.

The probabilistic safety of S-MPCC and SH-MPC is linked to the underlying accuracy of the prediction model of the obstacles that provide the scenarios. As a third contribution, a joint prediction and planning framework, Partitioned Scenario Replay (PSR), is proposed that replays past observations of human motion as scenarios for scenario-based planning. PSR does not fit a distribution on observed data but directly uses the data as empirical evidence of the underlying uncertainty and thereby provides a real-world safety guarantee.

A key limitation of the developed scenario-based planners and other optimization-based planners is that they locally refine an initial trajectory. This initial trajectory largely determines the quality of the final trajectory, while it does not consider other options. The fourth contribution of this thesis is Topology-driven Model Predictive Control (T-MPC) that concurrently optimizes trajectories, each attempting a different way to pass the obstacles. T-MPC is composed of a guidance planner and several parallel local planners. The guidance planner identifies guidance trajectories for several distinct maneuvers, relying on results from topology to distinguish trajectories. Each local planner is composed of an existing optimization-based planner (e.g., a scenario-based planner) and an additional set of constraints that are derived from one of the guidance trajectories. The guidance trajectories are optimized by the local planners in parallel, and the results are compared to determine which trajectory gets executed. T-MPC is faster, more consistent, and safer than several state-of-the-art planners. Contrary to similar existing work, it does not rely on an explicit lane structure and therefore enables both urban driving and mobile robotic applications.

The motion planners developed in this thesis are extensively validated in simulation and in experiments with a small-scale mobile robot and a full-scale self-driving vehicle navigating among pedestrians. The robot-agnostic implementation of the proposed planners that were developed for this thesis is available open source. ... Read more

3D object detection models that exploit both LiDAR and camera sensor features are top performers in large-scale autonomous driving benchmarks. A transformer is a popular network architecture used for this task, in which so-called object queries act as candidate objects. Initializing these object queries based on current sensor inputs leads to state-of-the-art performance. Existing methods rely strongly on LiDAR data however, and do not fully exploit image features. Besides, they introduce significant latency.

To overcome these limitations we propose EfficientQ3M, an efficient, modular, and multimodal solution for object query initialization for transformer-based 3D object detection models. Using both the LiDAR and camera modalities as input, we use efficient grid sampling and a lightweight detection head to predict a set of initial object query locations and corresponding query feature vectors. The proposed initialization method is combined with a “modality-balanced” transformer decoder where the queries can access all sensor modalities throughout the decoder.

We achieve state-of-the-art performance for both LiDAR-camera and LiDAR-only sensor setups on the competitive nuScenes benchmark while being up to 15 times more efficient than the closest related method. The proposed initialization can be applied with any combination of sensor modalities as input, demonstrating its modularity.
... Read more

This work addresses visual localization of intelligent vehicles as an alternative to traditional GPS- of HD map-based localization options. Specifically, the problem of Cross-View Pose Estimation (CVPE) is explored, which involves estimating the vehicle pose within an encompassing aerial patch, given a ground image from the on-board camera feed. The aerial patch containing the ground truth pose can be obtained through a rough localization prior, such as GPS. We find that existing CVPE methods start with a location prior that is too coarse given both the GPS performance and the required localization error. Therefore, we define a fine-grained localization setting and propose three approaches, targeting performance, interpretability, and data efficiency. Furthermore, the approaches have a unique capacity to predict a 6-DoF camera pose. Two approaches involve matching point-level local features in 3D space using a novel point cross-attention, while the last one aims to tailor an existing dense feature matching method to the fine-grained setting. Despite quantitative performance of the local feature matching approaches being inferior to the state-of-the-art, we establish a new state-of-the-art on the fine-grained setting with the improved dense-feature baseline. Nevertheless, we show the key limitations of the local feature matching, namely the influence of the “unmatchable” queries. Furthermore, using a 6-DoF projective transformation we discover severe issues with the ground truth quality on the KITTI dataset, commonly used in CVPE literature, potentially accounting to the large degree to the substandard performance of most available CVPE methods. Finally, our local feature matching methods demonstrate the capability of predicting pitch and roll angles of the camera, estimating which has not yet been attempted in CVPE. ... Read more

With recent advancements in autonomous driving, the demand for precise and accurate perception systems has increased. Perception of the vehicle’s environment is a key element in ensuring safe operation. Due to their wide aperture angle and low cost, ultrasonic sensors are a viable option for achieving close-range 360° perception around the vehicle. This thesis investigates and evaluates the use of ultrasonic sensors for detection and tracking of objects and vulnerable road users in an SAE-level 4 autonomous vehicle. The thesis focuses on optimizing ultrasonic sensors for their designated measurement task and optimizing the positions of ultrasonic sensors on the vehicle. Furthermore, it explores different detection and tracking methods to investigate whether an ultrasonic detection system is indeed a viable candidate for close-range detection and tracking. ... Read more

Critical to the safe application of autonomous vehicles is the ability to accurately predict the future motion of agents surrounding the vehicle. This is especially important - and challenging - in urban traffic, where vehicles share the road with Vulnerable Road Users (VRUs) such as pedestrians and cyclists. However, the majority of the existing on-board prediction datasets focus on predicting future trajectories of vehicles. We therefore present the View-of-Delft Prediction dataset, an extension of the recently-released urban View-of-Delft (VoD) dataset. The proposed dataset contains a large proportion of VRUs and has a good class balance, consisting of 844 prediction scenarios in the city of Delft, with 228 prediction instances for vehicles, 159 for cyclists, and 444 for pedestrians in dense urban traffic. Since state-of-the-art trajectory prediction approaches are primarily developed on car-dominated traffic with little interaction with VRUs, we analyse if the same methodology is suitable for mixed-traffic urban environments with VRUs and vehicles in close proximity. As our baseline for this analysis, we select the graph-based PGP model, for which we propose the addition of encoding motion of surrounding cyclists separately to facilitate its application in dense urban traffic. Since PGP relies on the lane graph topology, we provide novel rich map annotations for the VoD dataset, including lane polylines. Our analysis shows that there is a significant domain gap between the vehicle-dominated nuScenes and VRU-dominated View-of-Delft Prediction datasets, as training only on nuScenes results in a 107.79% higher minADE10 on the VoD Prediction test set than training the model on VoD Prediction. Furthermore, we modify the model by adding target agent class information, to make it suited for multi-class trajectory prediction. Our analysis shows that this yields a significant performance improvement of 13.92% in minADE10 for a six-second prediction horizon. The View-of-Delft Prediction dataset will be publicly released, enabling novel research on urban mixed-traffic trajectory prediction. ... Read more

Multi-class road user detection using the next- generation, 3+1D (range, azimuth, elevation, and Doppler) radars has been shown feasible, thanks to the increased density of their point clouds and the inclusion of elevation information. However, object detection networks using LiDAR (64-layer) point clouds still dominate the performance metrics. In this work, we explore the potential of fusing a 3+1D radar point cloud and a monocular image to further close this performance gap in 3D object detection. We propose a generic and modular fusion architecture to extract both spatial and semantic cues from an RGB image to complement the radar point cloud. In a two-stage approach, we first generate a 3D point cloud representation of the input monocular image appended with semantic information through our proposed RAID (RAdar guided Instance-aware Depth) network, which takes monocular depth map and panoptic masks predicted from any pre-trained state-of-the-art networks, and a radar depth map as input. We then append the resulting point cloud to the 3+1D radar point cloud in a straightforward fusion scheme and train a point cloud-based object detection network. Results on the View-of-Delft dataset [1] show that our fusion approach significantly outperforms multiple state-of-the-art radar-camera fusion methods (proposed fusion vs. best baseline: 53.6 mAP vs. 50.8 mAP), and yields comparable performance to a network trained on LiDAR input when evaluated in the safety-critical driving corridor (80.5 mAP vs. 81.6 mAP). ... Read more

Knowledge Distillation (KD) is a well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student. KD has proven to be an effective technique to significantly improve the student's performance for various tasks including object detection. As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an L_p-norm distance between teacher and student activations during training. In this paper, we propose a replacement for the pixel-wise independent L_p-norm based on structural similarity (SSIM). By taking into account additional contrast and structural cues, more knowledge within intermediate feature maps can be preserved. Extensive experiments on MSCOCO demonstrate the effectiveness of our method across different training schemes and architectures.
Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard L_p-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model. ... Read more

Pedestrian trajectory prediction is essential for developing safe autonomous driving systems. Such trajectories depend on various contextual cues, among which surrounding objects.

This work proposes the first pedestrian trajectory prediction method in the 2D on-board domain that models interactions between the pedestrian and surrounding static- and dynamic- contextual objects using a graph-based approach. Our two-stream model separately encodes past motion history and interactions. The encoded information from both streams is fused and decoded to generate future pedestrian trajectories. The interactions are modeled using spatial graphs, which are temporally connected using a Gated Recurrent Unit. The graph nodes represent the pedestrian and contextual objects, and the edges represent the interaction importance between nodes.

In experiments on the PIE and JAAD_full dataset, it is shown that our graph-based interaction-aware trajectory prediction method outperforms all considered baselines on nearly all metrics. Moreover, the performance gain on JAAD_full is most significant for the close-by pedestrians. Finally, modeling the interactions with all considered contextual objects, i.e. vehicles, crosswalks, and traffic lights, improves trajectory prediction performance most compared to only using a subset of these objects. ... Read more

The aim of the thesis is to develop methods and algorithms for the development of a robust perception system that is capable of dealing with adverse weather conditions. Robust environmental perception is important in order to guarantee safety for the automated vehicle and the road users in the neighborhood. To create a robust perception system, a sensor setup should be selected with multiple sensing modalities. Commonly used sensing modalities in the field of intelligent vehicles are lidar, camera and radar sensors. This thesis addresses three subjects that are important for robust perception, namely sensor selection, extrinsic calibration and object tracking.... ... Read more

Problem Definition According to the World Health Organization, traffic injuries have become the eighth cause of death and the leading cause among children and young adults. Human error, and in particular perceptual error, is among the most frequently reported causes of road fatalities. The desire to reduce traffic fatalities has led to the development of automated driving, which promises revolutionary advances in driver safety, traffic capacity and driver convenience. Since true autonomy in mixed traffic has not yet been achieved, today's automated vehicles require the driver to continuously supervise the automation and to capably intervene when necessary. However, simulator studies and experiences from disciplines such as aviation and factories have demonstrated that humans are generally ill-equipped to monitor automation for longer periods. This raises the concern that partial automation may harm rather than help traffic safety if not designed to adequately support the drivers in their supervisory tasks. Research objectives To address this concern, further insights are needed in how drivers monitor automation in complex real-world traffic, and how their behaviour and performance change with long-term automated driving experience. This dissertation sets out to investigate how real-world automation changes the availability of attentional resources, to establish where and how drivers use automation in naturalistic conditions, and evaluate how these change with experience. While these objectives investigate periods of automated driving, vehicles with automated driving functionalities will often be driven manually, when outside the operational design domain or at the driver’s preference. In these conditions, the available automation may still outperform the driver on particular tasks, such as detecting and tracking surrounding road users without bias or distraction. This dissertation therefore also contributes to the search for ways in which automation can provide meaningful support to the traffic monitoring task in manual and supervised driving. To evaluate if and when supervised automated driving negatively affects the driver’s ability to monitor, mental workload is evaluated in a Tesla model S on public roads (Chapter 2). Voluntary automation use and attention are examined in a naturalistic driving study on public roads (Chapter 3). To evaluate the effect of experience with automated driving, Chapter 2 compares drivers with and without prior automation use, whereas Chapter 3 examines how behaviour changes over a two-month period, compared to one month of manual driving. Two studies are performed to examine how driving automation can support the driver with the monitoring task, for which an instrumented vehicle was extended with cameras which track the driver’s gaze and associate it to surrounding road users as detected by the vehicle perception. The first study (Chapter 4) investigates how well gaze behaviour can indicate driver awareness toward individual road users, and proposes a recognition task to obtain a ground truth for awareness of multiple other road-users. The second study (Chapter 5) evaluates if driver gaze and head pose can provide earlier predictions for emergency alerting and intervention systems. A crossing pedestrian collision risk prediction system is used as a case study where gaze and contextual cues are evaluated in their contribution to path and risk prediction using a dynamic Bayesian network. Findings & recommendations Chapter 2 found that workload differed between roads with high and low traffic complexity, both for manual and automated driving, which indicates that drivers remain sensitive to changes in task demand while supervising automated driving. Drivers with prior experience in automated driving perceived a lower workload while supervising automation compared to manual driving. No workload difference was perceived for first-time users. In contrast, attentional demand as measured by a detection-response task was higher during automation use compared to manual driving regardless of experience. This indicates that monitoring automation (SAE2) requires more mental capacity compared to manual driving, which suggests that in contrast to a wide range of studies, SAE2 can increase workload. Supervising automation may therefore be beneficial for driver attention, but perception of workload during supervision may be too low for this to occur naturally. Future work should consider calibrating workload perception and system limitation understanding rather than actual task demand to encourage attentive supervision. Chapter 3 shows that automation is mostly used on road types generally considered suitable for automated driving with only incidental use on urban roads. This suggests that users are adhering to the operational design domain of these vehicles. On highways, automation is used at all speeds, but less during short periods of slow driving. No time-in-drive, time-of-day or experience effects were found for automation use. On the highway, head pose deviation was smaller during automation use compared to manual driving but tended to increase over the first six weeks of use, which may indicate a change in monitoring strategy. Further research is needed to assess if this difference indicates better or worse monitoring behaviour. Chapter 4 found that drivers performed better on the recognition task when road users were relevant for the driven manoeuvre and when drivers had directed their gaze within 10 degrees of these road users. However, at least 18% of road users were recognised while only observed peripherally, suggesting that peripheral vision should not be neglected in attention monitoring. Recognition performance was not predicted by gaze metrics and requires further development to reduce forget rates. Further analysis is needed to compare the recognition task to established situation awareness measures after these improvements are obtained. Chapter 5 demonstrates that driver and pedestrian attention monitoring can provide a benefit to pedestrian crossing collision risk prediction when predicting further than 0.75 seconds ahead. The higher workload during supervised automation and the general adherence to the operational design domain in naturalistic driving indicate that supervising driving automation can be beneficial to driver attention and traffic safety, but literature and recent accidents demonstrate that challenges remain in encouraging such attentive behaviour. Strategies to encourage attentive supervision should therefore be further developed, as well as ways to maintain these strategies while automation technology improves in pursuit of the opposite objective to reduce engagement in the driving task. The joint analysis of driver gaze and road scene may improve driver support during manual driving and supervised automation, and benefit the development of automated driving. But care should be taken that systems which use driver attention or rely on other contextual cues do not become susceptible to the same mistakes as drivers tend to make. While careful design approaches can reduce the risk of mimicking human error, validation will ultimately require a reliable way to distinguish between awareness and inattentional blindness. The instrumentation and conducted studies with on-road automation demonstrate that on-road research is becoming more practical and accessible than ever before, thanks to recent developments in automation. The observation that during on-road automation, inexperienced drivers perceive higher workload compared to in simulators testifies for the importance of on-road driving research. Challenges encountered during the naturalistic study and attention study demonstrate that the instrumentation and processing have to be designed and tested carefully for on-road research to be effective. ... Read more