Visual tracking of vulnerable road users for automated driving

Abstract

This thesis addresses the problem of vision-based tracking of Vulnerable Road Users (VRUs) such as pedestrians and riders in the context of Automated Driving (AD). The work primarily builds on camera-based perception, while also considering LiDAR as a complementary sensor. Robust tracking of persons is a critical perception task, as it enables downstream modules for motion prediction and safe planning. The motivation stems from the disproportionate involvement of VRUs in traffic fatalities, as globally pedestrians and riders account for more than half of all road deaths, particularly in urban environments. By providing temporally consistent trajectories and identity preservation across occlusion and re-entries, tracking contributes to safer and more anticipatory automated driving systems.

The thesis first covers an extended review of the state-of-the-art in multi-object tracking. A modular framework is outlined to describe traditional tracking pipelines. The review then focuses on deep learning-based methods. Early methods typically extended classical pipelines with deep features, while later works moved toward end-to-end formulations and transformer-based architectures. This survey provides the methodological foundation and highlights the transition toward data-driven approaches.

As these approaches rely on large amounts of data for both training and evaluation, fully manual annotation becomes increasingly challenging and costly, as the required effort grows rapidly with dataset size. To address this challenge, the thesis develops a semi-supervised trajectory generation pipeline. The approach leverages camera detections, LiDAR-based estimation of 3D locations from 2D boxes, and vehicle inertial sensing for ego-motion compensation. Based on this data, the approach generates temporally dense pseudo ground-truth (i.e. 2D bounding boxes, 3D person locations) from sparse, manual annotations at keyframes. Two complementary strategies are employed to generate tracks within each batch, whereby each batch is defined to begin and end at a keyframe. The first is a graph-based optimization for global association that leverages a spatio-temporal graph. The second is a tracklet-based approach that combines tracklet generation with subsequent tracklet merging. For both strategies, the outputs obtained within each batch are subsequently aggregated into complete trajectories, guided by the keyframe annotations. An experimental evaluation of these approaches, including comparisons to baselines, ablation studies, and ceiling analyses, demonstrates that the tracklet-based pipeline achieves the most robust overall performance. This approach significantly reduces annotation effort while producing temporally consistent trajectories suitable for training deep tracking models.

The EuroCity Persons (ECP) 2.0 dataset is introduced as a large-scale image dataset for person detection, tracking, and prediction in traffic. The dataset was collected onboard a vehicle driving through 29 cities in 11 European countries. It contains more than 220K unique person trajectories, in more than 2.0M images and comes with a size of 11 TB. ECP2.0 is about one order of magnitude larger than previous state-of-the art person datasets in automotive context. It offers remarkable diversity in terms of geographical coverage, time of day, weather and seasons. The pipeline proposed in this thesis is used to generate the dense annotations, providing temporally consistent pseudo ground-truth on top of the recorded image data. Experiments analyze the resulting annotation quality and object-dependent effects. The benefits of using the pseudo ground-truth for training and testing of state-of-the-art tracking methods are further demonstrated, showing improved tracking accuracy and robustness compared to training on sparsely annotated or purely detector-based supervision, while also highlighting remaining challenges in handling long-term occlusions and identity switches. The ECP2.0 dataset is made freely available for non-commercial research use.

Finally, this thesis improves person appearance embeddings as a means to strengthen identity association in multi-object tracking. To this end, a novel method for creating appearance embeddings to identify individual persons using an object Re- Identification (ReID) framework is introduced. CLFormer (Camera LiDAR Transformer) is presented, a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. The 3D Cuboid-Inclusive Point Embedding (3D-CIPE) is introduced, which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, various strategies for the early and late fusion of multi-modal input data are explored and analyzed. To evaluate the proposed CLFormer, the nuScenes dataset [14] is reinterpreted for ReID purposes and used for the experiments. CLFormer demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).

In conclusion, this thesis advances the state of-the-art in VRU tracking for automated driving by providing an extended methodological review, novel semi-supervised dataset creation techniques, a large-scale benchmark dataset, and a complementary ReID method that provides stronger identity cues for tracking. Together, these contributions offer new methodological insights, practical tools, and benchmarks that can support future research and development in the field of automated driving perception and tracking. Future work may focus on further improving the semi-supervised annotation method, particularly by reducing or even moving beyond the reliance on manual keyframe annotations, toward full unsupervised or self-supervised methods. In general, the long-term vision for multi-object tracking in automated driving points toward systems that unify perception, prediction, and decision-making in end-to-end frameworks, enabling robust and reliable operation in diverse and challenging real-world environments.