SK
S.A. Krebs
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
5 records found
1
This thesis addresses the problem of vision-based tracking of Vulnerable Road Users (VRUs) such as pedestrians and riders in the context of Automated Driving (AD). The work primarily builds on camera-based perception, while also considering LiDAR as a complementary sensor. Robust tracking of persons is a critical perception task, as it enables downstream modules for motion prediction and safe planning. The motivation stems from the disproportionate involvement of VRUs in traffic fatalities, as globally pedestrians and riders account for more than half of all road deaths, particularly in urban environments. By providing temporally consistent trajectories and identity preservation across occlusion and re-entries, tracking contributes to safer and more anticipatory automated driving systems.
The thesis first covers an extended review of the state-of-the-art in multi-object tracking. A modular framework is outlined to describe traditional tracking pipelines. The review then focuses on deep learning-based methods. Early methods typically extended classical pipelines with deep features, while later works moved toward end-to-end formulations and transformer-based architectures. This survey provides the methodological foundation and highlights the transition toward data-driven approaches.
As these approaches rely on large amounts of data for both training and evaluation, fully manual annotation becomes increasingly challenging and costly, as the required effort grows rapidly with dataset size. To address this challenge, the thesis develops a semi-supervised trajectory generation pipeline. The approach leverages camera detections, LiDAR-based estimation of 3D locations from 2D boxes, and vehicle inertial sensing for ego-motion compensation. Based on this data, the approach generates temporally dense pseudo ground-truth (i.e. 2D bounding boxes, 3D person locations) from sparse, manual annotations at keyframes. Two complementary strategies are employed to generate tracks within each batch, whereby each batch is defined to begin and end at a keyframe. The first is a graph-based optimization for global association that leverages a spatio-temporal graph. The second is a tracklet-based approach that combines tracklet generation with subsequent tracklet merging. For both strategies, the outputs obtained within each batch are subsequently aggregated into complete trajectories, guided by the keyframe annotations. An experimental evaluation of these approaches, including comparisons to baselines, ablation studies, and ceiling analyses, demonstrates that the tracklet-based pipeline achieves the most robust overall performance. This approach significantly reduces annotation effort while producing temporally consistent trajectories suitable for training deep tracking models.
The EuroCity Persons (ECP) 2.0 dataset is introduced as a large-scale image dataset for person detection, tracking, and prediction in traffic. The dataset was collected onboard a vehicle driving through 29 cities in 11 European countries. It contains more than 220K unique person trajectories, in more than 2.0M images and comes with a size of 11 TB. ECP2.0 is about one order of magnitude larger than previous state-of-the art person datasets in automotive context. It offers remarkable diversity in terms of geographical coverage, time of day, weather and seasons. The pipeline proposed in this thesis is used to generate the dense annotations, providing temporally consistent pseudo ground-truth on top of the recorded image data. Experiments analyze the resulting annotation quality and object-dependent effects. The benefits of using the pseudo ground-truth for training and testing of state-of-the-art tracking methods are further demonstrated, showing improved tracking accuracy and robustness compared to training on sparsely annotated or purely detector-based supervision, while also highlighting remaining challenges in handling long-term occlusions and identity switches. The ECP2.0 dataset is made freely available for non-commercial research use.
Finally, this thesis improves person appearance embeddings as a means to strengthen identity association in multi-object tracking. To this end, a novel method for creating appearance embeddings to identify individual persons using an object Re- Identification (ReID) framework is introduced. CLFormer (Camera LiDAR Transformer) is presented, a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. The 3D Cuboid-Inclusive Point Embedding (3D-CIPE) is introduced, which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, various strategies for the early and late fusion of multi-modal input data are explored and analyzed. To evaluate the proposed CLFormer, the nuScenes dataset [14] is reinterpreted for ReID purposes and used for the experiments. CLFormer demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).
In conclusion, this thesis advances the state of-the-art in VRU tracking for automated driving by providing an extended methodological review, novel semi-supervised dataset creation techniques, a large-scale benchmark dataset, and a complementary ReID method that provides stronger identity cues for tracking. Together, these contributions offer new methodological insights, practical tools, and benchmarks that can support future research and development in the field of automated driving perception and tracking. Future work may focus on further improving the semi-supervised annotation method, particularly by reducing or even moving beyond the reliance on manual keyframe annotations, toward full unsupervised or self-supervised methods. In general, the long-term vision for multi-object tracking in automated driving points toward systems that unify perception, prediction, and decision-making in end-to-end frameworks, enabling robust and reliable operation in diverse and challenging real-world environments.
...
The thesis first covers an extended review of the state-of-the-art in multi-object tracking. A modular framework is outlined to describe traditional tracking pipelines. The review then focuses on deep learning-based methods. Early methods typically extended classical pipelines with deep features, while later works moved toward end-to-end formulations and transformer-based architectures. This survey provides the methodological foundation and highlights the transition toward data-driven approaches.
As these approaches rely on large amounts of data for both training and evaluation, fully manual annotation becomes increasingly challenging and costly, as the required effort grows rapidly with dataset size. To address this challenge, the thesis develops a semi-supervised trajectory generation pipeline. The approach leverages camera detections, LiDAR-based estimation of 3D locations from 2D boxes, and vehicle inertial sensing for ego-motion compensation. Based on this data, the approach generates temporally dense pseudo ground-truth (i.e. 2D bounding boxes, 3D person locations) from sparse, manual annotations at keyframes. Two complementary strategies are employed to generate tracks within each batch, whereby each batch is defined to begin and end at a keyframe. The first is a graph-based optimization for global association that leverages a spatio-temporal graph. The second is a tracklet-based approach that combines tracklet generation with subsequent tracklet merging. For both strategies, the outputs obtained within each batch are subsequently aggregated into complete trajectories, guided by the keyframe annotations. An experimental evaluation of these approaches, including comparisons to baselines, ablation studies, and ceiling analyses, demonstrates that the tracklet-based pipeline achieves the most robust overall performance. This approach significantly reduces annotation effort while producing temporally consistent trajectories suitable for training deep tracking models.
The EuroCity Persons (ECP) 2.0 dataset is introduced as a large-scale image dataset for person detection, tracking, and prediction in traffic. The dataset was collected onboard a vehicle driving through 29 cities in 11 European countries. It contains more than 220K unique person trajectories, in more than 2.0M images and comes with a size of 11 TB. ECP2.0 is about one order of magnitude larger than previous state-of-the art person datasets in automotive context. It offers remarkable diversity in terms of geographical coverage, time of day, weather and seasons. The pipeline proposed in this thesis is used to generate the dense annotations, providing temporally consistent pseudo ground-truth on top of the recorded image data. Experiments analyze the resulting annotation quality and object-dependent effects. The benefits of using the pseudo ground-truth for training and testing of state-of-the-art tracking methods are further demonstrated, showing improved tracking accuracy and robustness compared to training on sparsely annotated or purely detector-based supervision, while also highlighting remaining challenges in handling long-term occlusions and identity switches. The ECP2.0 dataset is made freely available for non-commercial research use.
Finally, this thesis improves person appearance embeddings as a means to strengthen identity association in multi-object tracking. To this end, a novel method for creating appearance embeddings to identify individual persons using an object Re- Identification (ReID) framework is introduced. CLFormer (Camera LiDAR Transformer) is presented, a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. The 3D Cuboid-Inclusive Point Embedding (3D-CIPE) is introduced, which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, various strategies for the early and late fusion of multi-modal input data are explored and analyzed. To evaluate the proposed CLFormer, the nuScenes dataset [14] is reinterpreted for ReID purposes and used for the experiments. CLFormer demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).
In conclusion, this thesis advances the state of-the-art in VRU tracking for automated driving by providing an extended methodological review, novel semi-supervised dataset creation techniques, a large-scale benchmark dataset, and a complementary ReID method that provides stronger identity cues for tracking. Together, these contributions offer new methodological insights, practical tools, and benchmarks that can support future research and development in the field of automated driving perception and tracking. Future work may focus on further improving the semi-supervised annotation method, particularly by reducing or even moving beyond the reliance on manual keyframe annotations, toward full unsupervised or self-supervised methods. In general, the long-term vision for multi-object tracking in automated driving points toward systems that unify perception, prediction, and decision-making in end-to-end frameworks, enabling robust and reliable operation in diverse and challenging real-world environments.
...
This thesis addresses the problem of vision-based tracking of Vulnerable Road Users (VRUs) such as pedestrians and riders in the context of Automated Driving (AD). The work primarily builds on camera-based perception, while also considering LiDAR as a complementary sensor. Robust tracking of persons is a critical perception task, as it enables downstream modules for motion prediction and safe planning. The motivation stems from the disproportionate involvement of VRUs in traffic fatalities, as globally pedestrians and riders account for more than half of all road deaths, particularly in urban environments. By providing temporally consistent trajectories and identity preservation across occlusion and re-entries, tracking contributes to safer and more anticipatory automated driving systems.
The thesis first covers an extended review of the state-of-the-art in multi-object tracking. A modular framework is outlined to describe traditional tracking pipelines. The review then focuses on deep learning-based methods. Early methods typically extended classical pipelines with deep features, while later works moved toward end-to-end formulations and transformer-based architectures. This survey provides the methodological foundation and highlights the transition toward data-driven approaches.
As these approaches rely on large amounts of data for both training and evaluation, fully manual annotation becomes increasingly challenging and costly, as the required effort grows rapidly with dataset size. To address this challenge, the thesis develops a semi-supervised trajectory generation pipeline. The approach leverages camera detections, LiDAR-based estimation of 3D locations from 2D boxes, and vehicle inertial sensing for ego-motion compensation. Based on this data, the approach generates temporally dense pseudo ground-truth (i.e. 2D bounding boxes, 3D person locations) from sparse, manual annotations at keyframes. Two complementary strategies are employed to generate tracks within each batch, whereby each batch is defined to begin and end at a keyframe. The first is a graph-based optimization for global association that leverages a spatio-temporal graph. The second is a tracklet-based approach that combines tracklet generation with subsequent tracklet merging. For both strategies, the outputs obtained within each batch are subsequently aggregated into complete trajectories, guided by the keyframe annotations. An experimental evaluation of these approaches, including comparisons to baselines, ablation studies, and ceiling analyses, demonstrates that the tracklet-based pipeline achieves the most robust overall performance. This approach significantly reduces annotation effort while producing temporally consistent trajectories suitable for training deep tracking models.
The EuroCity Persons (ECP) 2.0 dataset is introduced as a large-scale image dataset for person detection, tracking, and prediction in traffic. The dataset was collected onboard a vehicle driving through 29 cities in 11 European countries. It contains more than 220K unique person trajectories, in more than 2.0M images and comes with a size of 11 TB. ECP2.0 is about one order of magnitude larger than previous state-of-the art person datasets in automotive context. It offers remarkable diversity in terms of geographical coverage, time of day, weather and seasons. The pipeline proposed in this thesis is used to generate the dense annotations, providing temporally consistent pseudo ground-truth on top of the recorded image data. Experiments analyze the resulting annotation quality and object-dependent effects. The benefits of using the pseudo ground-truth for training and testing of state-of-the-art tracking methods are further demonstrated, showing improved tracking accuracy and robustness compared to training on sparsely annotated or purely detector-based supervision, while also highlighting remaining challenges in handling long-term occlusions and identity switches. The ECP2.0 dataset is made freely available for non-commercial research use.
Finally, this thesis improves person appearance embeddings as a means to strengthen identity association in multi-object tracking. To this end, a novel method for creating appearance embeddings to identify individual persons using an object Re- Identification (ReID) framework is introduced. CLFormer (Camera LiDAR Transformer) is presented, a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. The 3D Cuboid-Inclusive Point Embedding (3D-CIPE) is introduced, which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, various strategies for the early and late fusion of multi-modal input data are explored and analyzed. To evaluate the proposed CLFormer, the nuScenes dataset [14] is reinterpreted for ReID purposes and used for the experiments. CLFormer demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).
In conclusion, this thesis advances the state of-the-art in VRU tracking for automated driving by providing an extended methodological review, novel semi-supervised dataset creation techniques, a large-scale benchmark dataset, and a complementary ReID method that provides stronger identity cues for tracking. Together, these contributions offer new methodological insights, practical tools, and benchmarks that can support future research and development in the field of automated driving perception and tracking. Future work may focus on further improving the semi-supervised annotation method, particularly by reducing or even moving beyond the reliance on manual keyframe annotations, toward full unsupervised or self-supervised methods. In general, the long-term vision for multi-object tracking in automated driving points toward systems that unify perception, prediction, and decision-making in end-to-end frameworks, enabling robust and reliable operation in diverse and challenging real-world environments.
The thesis first covers an extended review of the state-of-the-art in multi-object tracking. A modular framework is outlined to describe traditional tracking pipelines. The review then focuses on deep learning-based methods. Early methods typically extended classical pipelines with deep features, while later works moved toward end-to-end formulations and transformer-based architectures. This survey provides the methodological foundation and highlights the transition toward data-driven approaches.
As these approaches rely on large amounts of data for both training and evaluation, fully manual annotation becomes increasingly challenging and costly, as the required effort grows rapidly with dataset size. To address this challenge, the thesis develops a semi-supervised trajectory generation pipeline. The approach leverages camera detections, LiDAR-based estimation of 3D locations from 2D boxes, and vehicle inertial sensing for ego-motion compensation. Based on this data, the approach generates temporally dense pseudo ground-truth (i.e. 2D bounding boxes, 3D person locations) from sparse, manual annotations at keyframes. Two complementary strategies are employed to generate tracks within each batch, whereby each batch is defined to begin and end at a keyframe. The first is a graph-based optimization for global association that leverages a spatio-temporal graph. The second is a tracklet-based approach that combines tracklet generation with subsequent tracklet merging. For both strategies, the outputs obtained within each batch are subsequently aggregated into complete trajectories, guided by the keyframe annotations. An experimental evaluation of these approaches, including comparisons to baselines, ablation studies, and ceiling analyses, demonstrates that the tracklet-based pipeline achieves the most robust overall performance. This approach significantly reduces annotation effort while producing temporally consistent trajectories suitable for training deep tracking models.
The EuroCity Persons (ECP) 2.0 dataset is introduced as a large-scale image dataset for person detection, tracking, and prediction in traffic. The dataset was collected onboard a vehicle driving through 29 cities in 11 European countries. It contains more than 220K unique person trajectories, in more than 2.0M images and comes with a size of 11 TB. ECP2.0 is about one order of magnitude larger than previous state-of-the art person datasets in automotive context. It offers remarkable diversity in terms of geographical coverage, time of day, weather and seasons. The pipeline proposed in this thesis is used to generate the dense annotations, providing temporally consistent pseudo ground-truth on top of the recorded image data. Experiments analyze the resulting annotation quality and object-dependent effects. The benefits of using the pseudo ground-truth for training and testing of state-of-the-art tracking methods are further demonstrated, showing improved tracking accuracy and robustness compared to training on sparsely annotated or purely detector-based supervision, while also highlighting remaining challenges in handling long-term occlusions and identity switches. The ECP2.0 dataset is made freely available for non-commercial research use.
Finally, this thesis improves person appearance embeddings as a means to strengthen identity association in multi-object tracking. To this end, a novel method for creating appearance embeddings to identify individual persons using an object Re- Identification (ReID) framework is introduced. CLFormer (Camera LiDAR Transformer) is presented, a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. The 3D Cuboid-Inclusive Point Embedding (3D-CIPE) is introduced, which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, various strategies for the early and late fusion of multi-modal input data are explored and analyzed. To evaluate the proposed CLFormer, the nuScenes dataset [14] is reinterpreted for ReID purposes and used for the experiments. CLFormer demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).
In conclusion, this thesis advances the state of-the-art in VRU tracking for automated driving by providing an extended methodological review, novel semi-supervised dataset creation techniques, a large-scale benchmark dataset, and a complementary ReID method that provides stronger identity cues for tracking. Together, these contributions offer new methodological insights, practical tools, and benchmarks that can support future research and development in the field of automated driving perception and tracking. Future work may focus on further improving the semi-supervised annotation method, particularly by reducing or even moving beyond the reliance on manual keyframe annotations, toward full unsupervised or self-supervised methods. In general, the long-term vision for multi-object tracking in automated driving points toward systems that unify perception, prediction, and decision-making in end-to-end frameworks, enabling robust and reliable operation in diverse and challenging real-world environments.
In this paper, we introduce a novel method for creating appearance embeddings to identify individual persons using an object re-identification (ReID) framework. We present CLFormer (Camera LiDAR Transformer), a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. We introduce the 3D Cuboid-Inclusive Point Embedding (3D-CIPE), which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, we explore and analyze various strategies for the early and late fusion of multi-modal input data. To evaluate our proposed CLFormer, we reinterpret the nuScenes dataset [1] for ReID purposes and use it for our experiments. Our method demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).
...
In this paper, we introduce a novel method for creating appearance embeddings to identify individual persons using an object re-identification (ReID) framework. We present CLFormer (Camera LiDAR Transformer), a transformer-based architecture that incorporates multi-modal data from both camera and LiDAR sensors. We introduce the 3D Cuboid-Inclusive Point Embedding (3D-CIPE), which leverages rich data from LiDAR point clouds and 3D cuboids to add a learnable embedding into the transformer structure. Additionally, through ablation studies, we explore and analyze various strategies for the early and late fusion of multi-modal input data. To evaluate our proposed CLFormer, we reinterpret the nuScenes dataset [1] for ReID purposes and use it for our experiments. Our method demonstrates a significant improvement in performance, outperforming the image-only baseline with an increase of 2.3 in mean Average Precision (mAP).
Conference paper
(2021)
-
Markus Braun, Fabian B. Flohr, Sebastian Krebs, Ulrich Kresse, Dariu M. Gavrila
Despite the success of deep learning, human pose estimation remains a challenging problem in particular in dense urban traffic scenarios. Its robustness is important for followup tasks like trajectory prediction and gesture recognition. We are interested in human pose estimation in crowded scenes with overlapping pedestrians, in particular pairwise constellations. We propose a new top-down method that relies on pairwise detections as input and jointly estimates the two poses of such pairs in a single forward pass within a deep convolutional neural network. As availability of automotive datasets providing poses and a fair amount of crowded scenes is limited, we extend the EuroCity Persons dataset by additional images and pose annotations. With 46, 975 images and poses of 279, 329 persons our new EuroCity Persons Dense Pose dataset is the largest pose dataset recorded from a moving vehicle. In our experiments using this dataset we show improved performance for poses of pedestrian pairs in comparison with a state of the art method for human pose estimation in crowds.
...
Despite the success of deep learning, human pose estimation remains a challenging problem in particular in dense urban traffic scenarios. Its robustness is important for followup tasks like trajectory prediction and gesture recognition. We are interested in human pose estimation in crowded scenes with overlapping pedestrians, in particular pairwise constellations. We propose a new top-down method that relies on pairwise detections as input and jointly estimates the two poses of such pairs in a single forward pass within a deep convolutional neural network. As availability of automotive datasets providing poses and a fair amount of crowded scenes is limited, we extend the EuroCity Persons dataset by additional images and pose annotations. With 46, 975 images and poses of 279, 329 persons our new EuroCity Persons Dense Pose dataset is the largest pose dataset recorded from a moving vehicle. In our experiments using this dataset we show improved performance for poses of pedestrian pairs in comparison with a state of the art method for human pose estimation in crowds.
This paper presents an approach to generate dense person 3D trajectories from sparse image annotations on-board a moving platform. Our approach leverages the additional information that is typically available in an intelligent vehicle setting, such as LiDAR sensor measurements (to obtain 3D positions from detected 2D image bounding boxes) and inertial sensing (to perform ego-motion compensation). The sparse manual 2D person annotations that are available at regular time intervals (key-frames) are augmented with the output of a state-of-the-art 2D person detector, to obtain frame-wise data. A graph-based batch optimization approach is subsequently performed to find the best 3D trajectories, accounting for erroneous person detector output (false positives, false negatives, imprecise localization) and unknown temporal correspondences. Experiments on the EuroCity Persons dataset show promising results.
...
This paper presents an approach to generate dense person 3D trajectories from sparse image annotations on-board a moving platform. Our approach leverages the additional information that is typically available in an intelligent vehicle setting, such as LiDAR sensor measurements (to obtain 3D positions from detected 2D image bounding boxes) and inertial sensing (to perform ego-motion compensation). The sparse manual 2D person annotations that are available at regular time intervals (key-frames) are augmented with the output of a state-of-the-art 2D person detector, to obtain frame-wise data. A graph-based batch optimization approach is subsequently performed to find the best 3D trajectories, accounting for erroneous person detector output (false positives, false negatives, imprecise localization) and unknown temporal correspondences. Experiments on the EuroCity Persons dataset show promising results.
EuroCity persons
A novel benchmark for person detection in traffic scenes
Big data has had a great share in the success of deep learning in computer vision. Recent works suggest that there is significant further potential to increase object detection performance by utilizing even bigger datasets. In this paper, we introduce the EuroCity Persons dataset, which provides a large number of highly diverse, accurate and detailed annotations of pedestrians, cyclists and other riders in urban traffic scenes. The images for this dataset were collected on-board a moving vehicle in 31 cities of 12 European countries. With over 238,200 person instances manually labeled in over 47,300 images, EuroCity Persons is nearly one order of magnitude larger than datasets used previously for person detection in traffic scenes. The dataset furthermore contains a large number of person orientation annotations (over 211,200). We optimize four state-of-the-art deep learning approaches (Faster R-CNN, R-FCN, SSD and YOLOv3) to serve as baselines for the new object detection benchmark. In experiments with previous datasets we analyze the generalization capabilities of these detectors when trained with the new dataset. We furthermore study the effect of the training set size, the dataset diversity (day- versus night-time, geographical region), the dataset detail (i.e., availability of object orientation information) and the annotation quality on the detector performance. Finally, we analyze error sources and discuss the road ahead.
...
Big data has had a great share in the success of deep learning in computer vision. Recent works suggest that there is significant further potential to increase object detection performance by utilizing even bigger datasets. In this paper, we introduce the EuroCity Persons dataset, which provides a large number of highly diverse, accurate and detailed annotations of pedestrians, cyclists and other riders in urban traffic scenes. The images for this dataset were collected on-board a moving vehicle in 31 cities of 12 European countries. With over 238,200 person instances manually labeled in over 47,300 images, EuroCity Persons is nearly one order of magnitude larger than datasets used previously for person detection in traffic scenes. The dataset furthermore contains a large number of person orientation annotations (over 211,200). We optimize four state-of-the-art deep learning approaches (Faster R-CNN, R-FCN, SSD and YOLOv3) to serve as baselines for the new object detection benchmark. In experiments with previous datasets we analyze the generalization capabilities of these detectors when trained with the new dataset. We furthermore study the effect of the training set size, the dataset diversity (day- versus night-time, geographical region), the dataset detail (i.e., availability of object orientation information) and the annotation quality on the detector performance. Finally, we analyze error sources and discuss the road ahead.