JK
J.F.P. Kooij
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
44 records found
1
Within Reach
Enabling Embodiment-Aware Affordance Prediction Through Synthetic Dataset Generation
Affordance is a relational property that depends on both an object and the agent acting on it, yet most 3D affordance detection methods predict a single geometry-only heatmap that ignores which robot will act. We make embodiment an explicit conditioning variable for affordance prediction. Using Isaac Sim, we record contact between three morphologically distinct robots, the ANYmal-C quadruped, the Ridgeback-Franka mobile manipulator, and the Unitree H1 humanoid, and a set of household objects, and aggregate the raw contacts into per-point, per-body-part maps annotated with the displacement and rotation each contact creates. On this dataset we train a PointNet++-based model that conditions on object geometry and a natural-language description of the robot through cross-attention, producing different affordance maps for the same object depending on the acting morphology. The model recovers the reachability mask, body-part class, and interaction pose across embodiments, with near-convex objects such as bottles and vases the most predictable and geometrically varied chairs and lamps the hardest. The results show that embodiment is a learnable conditioning variable for affordance prediction, and provide a dataset and baseline for the task.
...
Affordance is a relational property that depends on both an object and the agent acting on it, yet most 3D affordance detection methods predict a single geometry-only heatmap that ignores which robot will act. We make embodiment an explicit conditioning variable for affordance prediction. Using Isaac Sim, we record contact between three morphologically distinct robots, the ANYmal-C quadruped, the Ridgeback-Franka mobile manipulator, and the Unitree H1 humanoid, and a set of household objects, and aggregate the raw contacts into per-point, per-body-part maps annotated with the displacement and rotation each contact creates. On this dataset we train a PointNet++-based model that conditions on object geometry and a natural-language description of the robot through cross-attention, producing different affordance maps for the same object depending on the acting morphology. The model recovers the reachability mask, body-part class, and interaction pose across embodiments, with near-convex objects such as bottles and vases the most predictable and geometrically varied chairs and lamps the hardest. The results show that embodiment is a learnable conditioning variable for affordance prediction, and provide a dataset and baseline for the task.
Object-level maps represent objects in an environment as a set of persistent instances; they support questions such as which objects are present, what they are, where they are, and how large they are. This thesis presents an online system that builds such a map for indoor mobile robots from RGB-D images and 2D LiDAR scans under imperfect detection, localization uncertainty, and changing object presence. Each object is represented by recursive beliefs over pose, physical extent, semantic class, and existence. Measurement uncertainty is calibrated empirically per class and expressed in a direction dependent, sensor-aligned frame. Observations are associated one-to-one by Mahalanobis distance for robustness to localization and measurement noise, and a viewpoint-novelty gate restricts geometric updates to sufficiently unique viewpoints, reducing overconfidence caused by treating repeated same-viewpoint observations as independent. Object existence is updated as an evidence-driven log-odds belief: detections provide positive evidence, while LiDAR free-space ray tracing provides negative evidence only when the predicted object location is confirmed empty. Evaluated on held-out recordings against motion-capture ground truth, the system maintains the correct object set under increasing synthetic Gaussian localization noise while achieving centimeter-level geometric accuracy and approximately consistent position covariances in the tested indoor setting.
...
Object-level maps represent objects in an environment as a set of persistent instances; they support questions such as which objects are present, what they are, where they are, and how large they are. This thesis presents an online system that builds such a map for indoor mobile robots from RGB-D images and 2D LiDAR scans under imperfect detection, localization uncertainty, and changing object presence. Each object is represented by recursive beliefs over pose, physical extent, semantic class, and existence. Measurement uncertainty is calibrated empirically per class and expressed in a direction dependent, sensor-aligned frame. Observations are associated one-to-one by Mahalanobis distance for robustness to localization and measurement noise, and a viewpoint-novelty gate restricts geometric updates to sufficiently unique viewpoints, reducing overconfidence caused by treating repeated same-viewpoint observations as independent. Object existence is updated as an evidence-driven log-odds belief: detections provide positive evidence, while LiDAR free-space ray tracing provides negative evidence only when the predicted object location is confirmed empty. Evaluated on held-out recordings against motion-capture ground truth, the system maintains the correct object set under increasing synthetic Gaussian localization noise while achieving centimeter-level geometric accuracy and approximately consistent position covariances in the tested indoor setting.
Conditional variational auto-encoders showed improved anomaly detection abilities over standard variational auto-encoders in literature. This paper explores the effectiveness of robot pose conditioning on thermal anomaly detection in the context of electrical substations. We introduce a multi-modal conditional variational auto-encoder framework, capable of reconstructing thermal images and robot poses. It utilises a multi-objective loss function consisting of mean squared error image and pose reconstruction loss and Kullback-Leibler divergence. Orientation showed to be the most effective conditioning pose, in the context of anomaly detection. A well performing network effectively reconstructs the original assets based on the latent space representation, contains only slightly blurred reconstructions in cases of uncertainty, has a structured latent space as principal component analysis reveals and shows high separability between the distributions of the image reconstruction errors for normal and anomalous samples.
...
...
Conditional variational auto-encoders showed improved anomaly detection abilities over standard variational auto-encoders in literature. This paper explores the effectiveness of robot pose conditioning on thermal anomaly detection in the context of electrical substations. We introduce a multi-modal conditional variational auto-encoder framework, capable of reconstructing thermal images and robot poses. It utilises a multi-objective loss function consisting of mean squared error image and pose reconstruction loss and Kullback-Leibler divergence. Orientation showed to be the most effective conditioning pose, in the context of anomaly detection. A well performing network effectively reconstructs the original assets based on the latent space representation, contains only slightly blurred reconstructions in cases of uncertainty, has a structured latent space as principal component analysis reveals and shows high separability between the distributions of the image reconstruction errors for normal and anomalous samples.
This work presents a real-time context-awareness frame work for mobile robots navigating dynamic indoor environments. The system provides reliable semantic information that is mapped onto Risk Priority Numbers (RPNs) using Failure Mode and Effects Analysis (FMEA), enabling risk-aware navigation decisions that prioritise both safety and efficiency. The pipeline combines RGB-D perception with large language model (LLM)-based classification, assigning each detected object a subtype and context-aware attributes such as animacy, movability, obstruction potential, and attentiveness. These attributes are dynamically converted into RPN scores, guiding the robot to proceed, slow down, reroute, or stop based on the current scene. A modular ROS 2 architecture manages real-time data fusion, including a Context Conduit Node (CCN) that synchronises semantic outputs, fallback detections, and depth data into a unified, time-aligned world state. The framework incorporates a fallback mechanism that ensures continuous operation when LLM inference is delayed. Evaluation in simulation demonstrates that unified prompting improves attribute consistency and risk estimation compared to split prompting strategies, supporting responsive and risk-aware navigation decisions.
...
This work presents a real-time context-awareness frame work for mobile robots navigating dynamic indoor environments. The system provides reliable semantic information that is mapped onto Risk Priority Numbers (RPNs) using Failure Mode and Effects Analysis (FMEA), enabling risk-aware navigation decisions that prioritise both safety and efficiency. The pipeline combines RGB-D perception with large language model (LLM)-based classification, assigning each detected object a subtype and context-aware attributes such as animacy, movability, obstruction potential, and attentiveness. These attributes are dynamically converted into RPN scores, guiding the robot to proceed, slow down, reroute, or stop based on the current scene. A modular ROS 2 architecture manages real-time data fusion, including a Context Conduit Node (CCN) that synchronises semantic outputs, fallback detections, and depth data into a unified, time-aligned world state. The framework incorporates a fallback mechanism that ensures continuous operation when LLM inference is delayed. Evaluation in simulation demonstrates that unified prompting improves attribute consistency and risk estimation compared to split prompting strategies, supporting responsive and risk-aware navigation decisions.
Artificial Intelligence (AI) put an increasing amount of strain on our total energy consumption and CO2 production. Not only is AI becoming increasingly more popular, but also AI models keep growing and thus need an increasing amount of computational resources. Recent research tries to mitigate this effect by creating more efficient hardware and by using Green AI, which is research in AI with additional focus on the computational resources a model requires. During this research one such Green AI method will be studied. This research focusses on the effects of memory usage on Canoncial Polyadic Decomposition (CPD) and Tensor Train (TT) decomposed Convolutional Neural Networks (CNN)s. These decomposed kinds of CNNs reduce the amount of parameters in the model, but may increase the amount of memory that is required to run the model. Therefore, first a theoretical analysis will be done on the memory used by these models. This analysis will then be validated by doing a real life scenario test and the effects of memory usage on inference time will be explored. Finally, regressions models will be made to see whether it is possible to predict the inference time. The results of these tests show that decomposed CNNs require more memory and the memory required in the real-life scenarios is higher than that was expected in the theoretical analysis. For the tested systems, it is shown that memory does influence the inference time negatively. Additionally, it was found that for very small kernels some initialization bias seems to be present, which makes the inference time larger,despite the CNN having less parameters and requiring less memory. Finally, it is shown that despite this inference bias, it is possible to predict whether the use of decomposed CNNs is beneficial to use compared to a regular CNN in terms of inference time.
...
Artificial Intelligence (AI) put an increasing amount of strain on our total energy consumption and CO2 production. Not only is AI becoming increasingly more popular, but also AI models keep growing and thus need an increasing amount of computational resources. Recent research tries to mitigate this effect by creating more efficient hardware and by using Green AI, which is research in AI with additional focus on the computational resources a model requires. During this research one such Green AI method will be studied. This research focusses on the effects of memory usage on Canoncial Polyadic Decomposition (CPD) and Tensor Train (TT) decomposed Convolutional Neural Networks (CNN)s. These decomposed kinds of CNNs reduce the amount of parameters in the model, but may increase the amount of memory that is required to run the model. Therefore, first a theoretical analysis will be done on the memory used by these models. This analysis will then be validated by doing a real life scenario test and the effects of memory usage on inference time will be explored. Finally, regressions models will be made to see whether it is possible to predict the inference time. The results of these tests show that decomposed CNNs require more memory and the memory required in the real-life scenarios is higher than that was expected in the theoretical analysis. For the tested systems, it is shown that memory does influence the inference time negatively. Additionally, it was found that for very small kernels some initialization bias seems to be present, which makes the inference time larger,despite the CNN having less parameters and requiring less memory. Finally, it is shown that despite this inference bias, it is possible to predict whether the use of decomposed CNNs is beneficial to use compared to a regular CNN in terms of inference time.
3D reconstruction for Digital Twin has become crucial, providing controlled and scalable environments for developing and validating advanced sensor algorithms before real-world deployment. Traditional reconstruction approaches have several limitations: they often overlook physical rendering properties of materials, lack detailed textures, and rely on lidar sensors that require complex calibration and perform poorly with capturing retro-reflective and transparent surfaces. We propose a modular, camera-centric, and material-informed 3D Gaussian Splatting (3DGS) pipeline for reconstructing large 3D scenes. Our approach extracts semantic material masks using segmentation models, converts Gaussian representations to explicit mesh surfaces, and automatically projects 2D material labels onto 3D geometry. This combines photorealistic reconstruction with physics-based material assignment for accurate sensor simulation and rendering in modern graphics engines and simulators. Evaluation on an internal Siemens autonomous driving dataset demonstrates that our material-informed approach achieves sensor simulation fidelity comparable to lidar-based ground truth while providing high visual fidelity, as validated through image similarity metrics and lidar reflectivity analysis.
...
3D reconstruction for Digital Twin has become crucial, providing controlled and scalable environments for developing and validating advanced sensor algorithms before real-world deployment. Traditional reconstruction approaches have several limitations: they often overlook physical rendering properties of materials, lack detailed textures, and rely on lidar sensors that require complex calibration and perform poorly with capturing retro-reflective and transparent surfaces. We propose a modular, camera-centric, and material-informed 3D Gaussian Splatting (3DGS) pipeline for reconstructing large 3D scenes. Our approach extracts semantic material masks using segmentation models, converts Gaussian representations to explicit mesh surfaces, and automatically projects 2D material labels onto 3D geometry. This combines photorealistic reconstruction with physics-based material assignment for accurate sensor simulation and rendering in modern graphics engines and simulators. Evaluation on an internal Siemens autonomous driving dataset demonstrates that our material-informed approach achieves sensor simulation fidelity comparable to lidar-based ground truth while providing high visual fidelity, as validated through image similarity metrics and lidar reflectivity analysis.
In the context of open-world scenarios in autonomous vehicles (AVs), previously unseen classes may arise. To address this, effective extraction of well generalizable features is essential for AV downstream tasks, especially in the context of zero-shot learning. This can be achieved using transformers, and Swin Transformer in particular (vision backbone of most Vision-Language Models). However, to enable on-board applications, knowledge distillation must be utilized to create a lightweight model capable of real-time processing. We explore self-supervised knowledge distillation, given that AV datasets need to generalize to previously unseen classes. Our contributions include adapting existing CNN-to-CNN output-based self-supervised knowledge distillation algorithms to Transformer-to-CNN for benchmarking and enhancing them with a cross-architecture loss function. By leveraging DisCo, the best performing output-based self-supervised knowledge distillation method, and using EfficientNetB0 as the student model, we achieve a 3.9% relative improvement in top-1 accuracy over the supervised Swin-T teacher on our modified ImageNet for open-world classification, up to 5.0% with our loss.
...
In the context of open-world scenarios in autonomous vehicles (AVs), previously unseen classes may arise. To address this, effective extraction of well generalizable features is essential for AV downstream tasks, especially in the context of zero-shot learning. This can be achieved using transformers, and Swin Transformer in particular (vision backbone of most Vision-Language Models). However, to enable on-board applications, knowledge distillation must be utilized to create a lightweight model capable of real-time processing. We explore self-supervised knowledge distillation, given that AV datasets need to generalize to previously unseen classes. Our contributions include adapting existing CNN-to-CNN output-based self-supervised knowledge distillation algorithms to Transformer-to-CNN for benchmarking and enhancing them with a cross-architecture loss function. By leveraging DisCo, the best performing output-based self-supervised knowledge distillation method, and using EfficientNetB0 as the student model, we achieve a 3.9% relative improvement in top-1 accuracy over the supervised Swin-T teacher on our modified ImageNet for open-world classification, up to 5.0% with our loss.
Multi-Robot Exploration in Network-Uncertain Indoor Environments
An approach based on adaptive signal strength
In this thesis, an autonomous multi-robot system for indoor exploration in limited network environments is proposed. The specific use case is search and rescue where the operators must have access to the most up-to-date information, necessitating the requirement for communication maintenance. This requirement is satisfied in a novel way by using signal strength measurements collected by the robots to update the network model, thus reducing the overly conservative nature of current approaches. The network model chosen is an adaptation of existing path loss models as they balance complexity and performance given the information available during multi-robot exploration.
The proposed system consists of a central server with multiple robots which perform tasks that are readily distributable, whereas the central server performs complex iterative optimizations such as merging the local maps and running task assignment. While fully distributed architectures offer theoretical benefits such as scalability and robustness, in practice, these benefits are currently not achieved due to the complexity of splitting iterative optimizations such as mapping and task assignment among multiple nodes. This results in distributed architectures requiring more time and bandwidth to solve these problems, which is why this work implements a central architecture. While iterative optimizations are more efficient when performed centrally, scalability is limited due to the branching of possible options during task assignment. To combat this, an action space formulation, called an action graph, is proposed that is unique to each robot and reduces the number of actions by merging similar ones.
Both the dynamic network prediction model and the action graph formulation are shown to hold promise by experimentation. However, more work is needed before real-world use is feasible.
...
The proposed system consists of a central server with multiple robots which perform tasks that are readily distributable, whereas the central server performs complex iterative optimizations such as merging the local maps and running task assignment. While fully distributed architectures offer theoretical benefits such as scalability and robustness, in practice, these benefits are currently not achieved due to the complexity of splitting iterative optimizations such as mapping and task assignment among multiple nodes. This results in distributed architectures requiring more time and bandwidth to solve these problems, which is why this work implements a central architecture. While iterative optimizations are more efficient when performed centrally, scalability is limited due to the branching of possible options during task assignment. To combat this, an action space formulation, called an action graph, is proposed that is unique to each robot and reduces the number of actions by merging similar ones.
Both the dynamic network prediction model and the action graph formulation are shown to hold promise by experimentation. However, more work is needed before real-world use is feasible.
...
In this thesis, an autonomous multi-robot system for indoor exploration in limited network environments is proposed. The specific use case is search and rescue where the operators must have access to the most up-to-date information, necessitating the requirement for communication maintenance. This requirement is satisfied in a novel way by using signal strength measurements collected by the robots to update the network model, thus reducing the overly conservative nature of current approaches. The network model chosen is an adaptation of existing path loss models as they balance complexity and performance given the information available during multi-robot exploration.
The proposed system consists of a central server with multiple robots which perform tasks that are readily distributable, whereas the central server performs complex iterative optimizations such as merging the local maps and running task assignment. While fully distributed architectures offer theoretical benefits such as scalability and robustness, in practice, these benefits are currently not achieved due to the complexity of splitting iterative optimizations such as mapping and task assignment among multiple nodes. This results in distributed architectures requiring more time and bandwidth to solve these problems, which is why this work implements a central architecture. While iterative optimizations are more efficient when performed centrally, scalability is limited due to the branching of possible options during task assignment. To combat this, an action space formulation, called an action graph, is proposed that is unique to each robot and reduces the number of actions by merging similar ones.
Both the dynamic network prediction model and the action graph formulation are shown to hold promise by experimentation. However, more work is needed before real-world use is feasible.
The proposed system consists of a central server with multiple robots which perform tasks that are readily distributable, whereas the central server performs complex iterative optimizations such as merging the local maps and running task assignment. While fully distributed architectures offer theoretical benefits such as scalability and robustness, in practice, these benefits are currently not achieved due to the complexity of splitting iterative optimizations such as mapping and task assignment among multiple nodes. This results in distributed architectures requiring more time and bandwidth to solve these problems, which is why this work implements a central architecture. While iterative optimizations are more efficient when performed centrally, scalability is limited due to the branching of possible options during task assignment. To combat this, an action space formulation, called an action graph, is proposed that is unique to each robot and reduces the number of actions by merging similar ones.
Both the dynamic network prediction model and the action graph formulation are shown to hold promise by experimentation. However, more work is needed before real-world use is feasible.
4D millimeter-wave radar is increasingly important in advanced driver-assistance systems due to its ability to capture Doppler/velocity information and robustness in low-light or adverse weather conditions. Unlike traditional 3D radar, 4D radar provides elevation information, enhancing 3D spatial perception. Most of the perception tasks using 4D radar tensors focus on target classification using bounding boxes. In contrast, although semantic segmentation is typically used to process images and LiDAR point clouds, it has not been well explored for 4D radar tensors.
This MSc thesis aims to bridge the gap in using 4D radar tensors for semantic segmentation and in generating the required labels for supervision. Specifically, it proposes an automatic approach for generating multi-class point-wise labels for automotive datasets by leveraging the complementary information from the synchronized camera and LiDAR data. Then, 4D radar tensors are used as inputs, supervised by the generated labels, to design a radar semantic segmentation network. Promising results are shown by applying both developed parts to the publicly shared RaDelft dataset. The automatic labeling process demonstrates satisfactory quantitative and qualitative results compared with manual labeling results obtained on randomly chosen scenes. The outputs of the semantic segmentation network achieve more than 65% in overall detection probability, improving by +13.1% in terms of vehicle class detection probability, and reducing 0.54 m in terms of Chamfer distance compared to the variants inspired by the literature.
...
This MSc thesis aims to bridge the gap in using 4D radar tensors for semantic segmentation and in generating the required labels for supervision. Specifically, it proposes an automatic approach for generating multi-class point-wise labels for automotive datasets by leveraging the complementary information from the synchronized camera and LiDAR data. Then, 4D radar tensors are used as inputs, supervised by the generated labels, to design a radar semantic segmentation network. Promising results are shown by applying both developed parts to the publicly shared RaDelft dataset. The automatic labeling process demonstrates satisfactory quantitative and qualitative results compared with manual labeling results obtained on randomly chosen scenes. The outputs of the semantic segmentation network achieve more than 65% in overall detection probability, improving by +13.1% in terms of vehicle class detection probability, and reducing 0.54 m in terms of Chamfer distance compared to the variants inspired by the literature.
...
4D millimeter-wave radar is increasingly important in advanced driver-assistance systems due to its ability to capture Doppler/velocity information and robustness in low-light or adverse weather conditions. Unlike traditional 3D radar, 4D radar provides elevation information, enhancing 3D spatial perception. Most of the perception tasks using 4D radar tensors focus on target classification using bounding boxes. In contrast, although semantic segmentation is typically used to process images and LiDAR point clouds, it has not been well explored for 4D radar tensors.
This MSc thesis aims to bridge the gap in using 4D radar tensors for semantic segmentation and in generating the required labels for supervision. Specifically, it proposes an automatic approach for generating multi-class point-wise labels for automotive datasets by leveraging the complementary information from the synchronized camera and LiDAR data. Then, 4D radar tensors are used as inputs, supervised by the generated labels, to design a radar semantic segmentation network. Promising results are shown by applying both developed parts to the publicly shared RaDelft dataset. The automatic labeling process demonstrates satisfactory quantitative and qualitative results compared with manual labeling results obtained on randomly chosen scenes. The outputs of the semantic segmentation network achieve more than 65% in overall detection probability, improving by +13.1% in terms of vehicle class detection probability, and reducing 0.54 m in terms of Chamfer distance compared to the variants inspired by the literature.
This MSc thesis aims to bridge the gap in using 4D radar tensors for semantic segmentation and in generating the required labels for supervision. Specifically, it proposes an automatic approach for generating multi-class point-wise labels for automotive datasets by leveraging the complementary information from the synchronized camera and LiDAR data. Then, 4D radar tensors are used as inputs, supervised by the generated labels, to design a radar semantic segmentation network. Promising results are shown by applying both developed parts to the publicly shared RaDelft dataset. The automatic labeling process demonstrates satisfactory quantitative and qualitative results compared with manual labeling results obtained on randomly chosen scenes. The outputs of the semantic segmentation network achieve more than 65% in overall detection probability, improving by +13.1% in terms of vehicle class detection probability, and reducing 0.54 m in terms of Chamfer distance compared to the variants inspired by the literature.
Detecting nearby vehicles involves utilizing data from various sensors installed on a car as it moves. Common sensors for identifying nearby vehicles include LiDAR, cameras, and RADAR. However, all of these sensors suffer from the same issue -- they cannot detect an approaching vehicle that is not yet visible. Hence, this thesis explores the potential of using a microphone array -- an array of sensors capable of detecting vehicles that are out of sight. Exploring prior research on detecting obstructed vehicles using sound reveals an existing model capable of detecting nearby vehicles approaching from behind blind corners. However, as the local geometry around the ego vehicle affects the perceived sound patterns, this model was only designed to work within a specific set of T-junctions. Therefore, the thesis aims to take a step further and develop a detection model capable of detecting vehicles behind blind corners in environments not included in the training set of the deployed model. This is challenging for multiple reasons. First, literature review revealed a lack of suitable datasets comprising sounds from approaching vehicles behind blind corners within various road junctions. In addition, microphones, like other sensors, come with limitations. Sound inherently provides less spatial information compared to commonly used sensors in autonomous driving, such as LiDAR or cameras. Considering sound propagation variations in different road junction geometries, building a model adaptable across diverse junction types presents a challenge. To overcome the data scarcity and sound's inherent spatial limitations, the study investigates the potential of employing simulated acoustic responses within artificial road environments as training data for real-world vehicle detection. Simultaneously, to complement the sounds inherent advantage of detecting objects that are out of sight, the thesis proposes to use a Bird's Eye View (BEV) encoding of the top-down map from the driving vehicle's perspective. Having an encoding of the top-down map of the current driving environment would allow a detection model to expect sound signatures commonly observed within a given setting. Overall, the assessment of acoustic simulations could not outline a singular configuration of simulation properties allowing realistic sound propagation for any kind of considered junctions when hearing an approaching vehicle. However, it was observed that the utilization of specific simulation parameters can result in realistic sound propagation within the given junction. Subsequently, evaluating a novel BEV encoding within the newly proposed acoustic detection pipeline demonstrated either equivalent or superior performance compared to a model relying solely on sound. Overall, this research underscores the potential of incorporating BEV encoding in non-line-of-sight acoustic detection and suggests the promise of acoustic simulations within the field. This study contributes to advancing the integration of sound as an additional data modality in vehicle detection.
...
Detecting nearby vehicles involves utilizing data from various sensors installed on a car as it moves. Common sensors for identifying nearby vehicles include LiDAR, cameras, and RADAR. However, all of these sensors suffer from the same issue -- they cannot detect an approaching vehicle that is not yet visible. Hence, this thesis explores the potential of using a microphone array -- an array of sensors capable of detecting vehicles that are out of sight. Exploring prior research on detecting obstructed vehicles using sound reveals an existing model capable of detecting nearby vehicles approaching from behind blind corners. However, as the local geometry around the ego vehicle affects the perceived sound patterns, this model was only designed to work within a specific set of T-junctions. Therefore, the thesis aims to take a step further and develop a detection model capable of detecting vehicles behind blind corners in environments not included in the training set of the deployed model. This is challenging for multiple reasons. First, literature review revealed a lack of suitable datasets comprising sounds from approaching vehicles behind blind corners within various road junctions. In addition, microphones, like other sensors, come with limitations. Sound inherently provides less spatial information compared to commonly used sensors in autonomous driving, such as LiDAR or cameras. Considering sound propagation variations in different road junction geometries, building a model adaptable across diverse junction types presents a challenge. To overcome the data scarcity and sound's inherent spatial limitations, the study investigates the potential of employing simulated acoustic responses within artificial road environments as training data for real-world vehicle detection. Simultaneously, to complement the sounds inherent advantage of detecting objects that are out of sight, the thesis proposes to use a Bird's Eye View (BEV) encoding of the top-down map from the driving vehicle's perspective. Having an encoding of the top-down map of the current driving environment would allow a detection model to expect sound signatures commonly observed within a given setting. Overall, the assessment of acoustic simulations could not outline a singular configuration of simulation properties allowing realistic sound propagation for any kind of considered junctions when hearing an approaching vehicle. However, it was observed that the utilization of specific simulation parameters can result in realistic sound propagation within the given junction. Subsequently, evaluating a novel BEV encoding within the newly proposed acoustic detection pipeline demonstrated either equivalent or superior performance compared to a model relying solely on sound. Overall, this research underscores the potential of incorporating BEV encoding in non-line-of-sight acoustic detection and suggests the promise of acoustic simulations within the field. This study contributes to advancing the integration of sound as an additional data modality in vehicle detection.
A scenario-based testing approach can reduce the time required to obtain statistically significant evidence of the safety of Automated Driving Systems (ADS). Identifying these scenarios in an automated manner is a challenging task. Most methods on scenario classification do not work for complex scenarios with diverse environments (highways, urban) and interaction with other traffic agents. This is mirrored in their approaches which model an individual vehicle in relation to its environment, but neglect the interaction between multiple vehicles (e.g. cut-ins, stationary lead vehicle). Furthermore, existing datasets lack diversity and do not have per-frame annotations to accurately learn the start and end time of a scenario. We propose a method for complex traffic scenario classification that is able to model the interaction of a vehicle with the environment, as well as other agents. We use Graph Convolutional Networks to model spatial and temporal aspects of these scenarios. Expanding the nuScenes and Argoverse 2 driving datasets, we introduce a scenario-labeled dataset, which covers different driving environments and is annotated per frame. Training our method on this dataset, we present a promising baseline for future research on perframe complex scenario classification
...
A scenario-based testing approach can reduce the time required to obtain statistically significant evidence of the safety of Automated Driving Systems (ADS). Identifying these scenarios in an automated manner is a challenging task. Most methods on scenario classification do not work for complex scenarios with diverse environments (highways, urban) and interaction with other traffic agents. This is mirrored in their approaches which model an individual vehicle in relation to its environment, but neglect the interaction between multiple vehicles (e.g. cut-ins, stationary lead vehicle). Furthermore, existing datasets lack diversity and do not have per-frame annotations to accurately learn the start and end time of a scenario. We propose a method for complex traffic scenario classification that is able to model the interaction of a vehicle with the environment, as well as other agents. We use Graph Convolutional Networks to model spatial and temporal aspects of these scenarios. Expanding the nuScenes and Argoverse 2 driving datasets, we introduce a scenario-labeled dataset, which covers different driving environments and is annotated per frame. Training our method on this dataset, we present a promising baseline for future research on perframe complex scenario classification
VPR describes a task where an agent (e.g., a robot) attempts to recognize its current location by comparing the incoming visual data from its sensor(s) (query images), usually a camera, to geotagged reference images. Both query and reference images are described using a feature extractor, and the query descriptor is matched to its closest reference descriptor in the feature space. Within VPR there are many different VPR techniques that have been proposed throughout the years with many different types of architecture and trained on different datasets. With the many test datasets available, there exists no VPR technique that is able to reach state-of-the-art performance on all these datasets. For this reason, existing work has argued it can be beneficial to utilize an ensemble-based method to combine multiple VPR techniques and achieve better VPR performance. Some of these Ensemble-based methods have already been proposed. These ensemble-methods combine individual VPR techniques and weigh their predictions using these same predictions to give an indication of their confidence. This calculation, however, is strictly based on predictions obtained from applying the VPR techniques on test data at inference time. Generally within VPR research, the dataset that was used to train the VPR technique is often different from the dataset it is tested on. This means there is a domain shift between the training and test data. This domain shift is not taken into account when weighting the predictions of VPR techniques in an ensemble using these existing methods. In this work, we analyze how this degree of domain shift between train and test data, which can be observed by looking at the relative location of descriptors in the feature space, impacts downstream VPR performance. Intuitively, one would expect better VPR performance in a situation where the degree of train-test domain shift is minimal. Our analysis shows that this is indeed the case. We propose two different methods that utilize this degree of domain shift to calculate the weights given to the VPR techniques in an ensemble. First, we propose a generative method. Here weights are given to the VPR techniques based on the likelihood that the query sample originated from the same distribution as the training dataset of the technique and is in distribution. This way each individual technique is given a weight. Secondly, we propose a discriminative method. Here weights are given to the training datasets used to train the techniques in the ensemble. These training datasets are given weights based on relative proximity to a query sample in the feature space, an indicator for the degree of domain-shift between the training dataset and the query sample. all VPR techniques are given the weight corresponding to their training dataset. We compare these proposed approaches to other ensemble-based baselines and individual VPR techniques. The quantitative results show that our proposed methods generally outperform the ensemblebased baselines and the individual VPR techniques. We also propose further future work. One of the generative methods still delivers lower performance than could be possible, caused by applying this method to high-dimensionality descriptors. A solution for this issue should lead to higher VPR performance using this method. Additionally, we suggest future work to expand on the datasets used in this research, to strengthen the claims made, and verify that results and trends found to hold up when testing and training using other datasets.
...
VPR describes a task where an agent (e.g., a robot) attempts to recognize its current location by comparing the incoming visual data from its sensor(s) (query images), usually a camera, to geotagged reference images. Both query and reference images are described using a feature extractor, and the query descriptor is matched to its closest reference descriptor in the feature space. Within VPR there are many different VPR techniques that have been proposed throughout the years with many different types of architecture and trained on different datasets. With the many test datasets available, there exists no VPR technique that is able to reach state-of-the-art performance on all these datasets. For this reason, existing work has argued it can be beneficial to utilize an ensemble-based method to combine multiple VPR techniques and achieve better VPR performance. Some of these Ensemble-based methods have already been proposed. These ensemble-methods combine individual VPR techniques and weigh their predictions using these same predictions to give an indication of their confidence. This calculation, however, is strictly based on predictions obtained from applying the VPR techniques on test data at inference time. Generally within VPR research, the dataset that was used to train the VPR technique is often different from the dataset it is tested on. This means there is a domain shift between the training and test data. This domain shift is not taken into account when weighting the predictions of VPR techniques in an ensemble using these existing methods. In this work, we analyze how this degree of domain shift between train and test data, which can be observed by looking at the relative location of descriptors in the feature space, impacts downstream VPR performance. Intuitively, one would expect better VPR performance in a situation where the degree of train-test domain shift is minimal. Our analysis shows that this is indeed the case. We propose two different methods that utilize this degree of domain shift to calculate the weights given to the VPR techniques in an ensemble. First, we propose a generative method. Here weights are given to the VPR techniques based on the likelihood that the query sample originated from the same distribution as the training dataset of the technique and is in distribution. This way each individual technique is given a weight. Secondly, we propose a discriminative method. Here weights are given to the training datasets used to train the techniques in the ensemble. These training datasets are given weights based on relative proximity to a query sample in the feature space, an indicator for the degree of domain-shift between the training dataset and the query sample. all VPR techniques are given the weight corresponding to their training dataset. We compare these proposed approaches to other ensemble-based baselines and individual VPR techniques. The quantitative results show that our proposed methods generally outperform the ensemblebased baselines and the individual VPR techniques. We also propose further future work. One of the generative methods still delivers lower performance than could be possible, caused by applying this method to high-dimensionality descriptors. A solution for this issue should lead to higher VPR performance using this method. Additionally, we suggest future work to expand on the datasets used in this research, to strengthen the claims made, and verify that results and trends found to hold up when testing and training using other datasets.
Master thesis
(2023)
-
M.M. Şabanoğlu, Nergis Tömen, J.C.F. de Winter, Martijn Souman, J.C. van Gemert, J.F.P. Kooij, M.R. Araghi
An event-based camera enables capturing a video at a high temporal resolution, high dynamical range, reduced power consumption and minimal data bandwidth while the camera has minimal physical dimensions compared to a frame-based camera with the same vision properties. The limiting factor, however, of an event-based camera is the spatial resolution which ranges between 40 × 40 and 1280 × 960. To counter this deficiency, a method is researched to super resolve event-based vision in order to enhance spatial resolution. A selection of different neural network types and configurations are researched in a step-by-step fashion. Subsequent experiments tested the selected networks on their ability to process event-based data and extract features from it. Followed by experiments that exploited the limitations of the networks to super resolve at different ratios, lengths of eventstreams and more complex event-based data. Results of various experiments showed that a network configuration that utilizes a transformer architecture was best able to super resolve event-based vision. This type of network leverages the ability to extract features based on dependencies between events which aligns with the characteristics of event- based vision. Based on the obtained results from the exper- iments, a pipeline is proposed to super resolve event-based vision and consists of a combination of a transformer network, multilayer perceptrons and a k-nearest-neighbor algorithm. Using this pipeline, eventstreams can be super resolved in the spatial resolution at a scaling ratio of 4. Visually, these super resolved eventstreams resemble more detailed and enhanced version to the low-resolution input. This proposed pipeline can be considered as a starting point in further research toward the super-resolution of event-based data and thereby contributes to the extension of application possibilities of event-based vision.
...
An event-based camera enables capturing a video at a high temporal resolution, high dynamical range, reduced power consumption and minimal data bandwidth while the camera has minimal physical dimensions compared to a frame-based camera with the same vision properties. The limiting factor, however, of an event-based camera is the spatial resolution which ranges between 40 × 40 and 1280 × 960. To counter this deficiency, a method is researched to super resolve event-based vision in order to enhance spatial resolution. A selection of different neural network types and configurations are researched in a step-by-step fashion. Subsequent experiments tested the selected networks on their ability to process event-based data and extract features from it. Followed by experiments that exploited the limitations of the networks to super resolve at different ratios, lengths of eventstreams and more complex event-based data. Results of various experiments showed that a network configuration that utilizes a transformer architecture was best able to super resolve event-based vision. This type of network leverages the ability to extract features based on dependencies between events which aligns with the characteristics of event- based vision. Based on the obtained results from the exper- iments, a pipeline is proposed to super resolve event-based vision and consists of a combination of a transformer network, multilayer perceptrons and a k-nearest-neighbor algorithm. Using this pipeline, eventstreams can be super resolved in the spatial resolution at a scaling ratio of 4. Visually, these super resolved eventstreams resemble more detailed and enhanced version to the low-resolution input. This proposed pipeline can be considered as a starting point in further research toward the super-resolution of event-based data and thereby contributes to the extension of application possibilities of event-based vision.
Adaptive Cruise Control (ACC) relieves human drivers’ tasks by taking over the control of the throttle and braking of the vehicles automatically. However, it has been demonstrated in many empirical studies that current production ACC systems fail to guarantee string stability. It is believed that if vehicles can take the longitudinal dynamics further downstream into account and react to the propagating disturbance earlier, the string stability in the platoon may be improved. Instead of relying on inter-vehicle communication technologies, the ego-vehicle should be able to detect the second leading vehicle by leveraging the power of on-board sensors. Still, the second leader measurements can be highly erroneous. Therefore, it is important to consider the entailed measurement uncertainties when designing and evaluating such ACC systems. This study proposes several ACC systems which possess the property of multi-anticipation and uncertainty handling.
The possible sensor technology which can collect the second leader measurements is first investigated. Based on the considered setup, the measurement uncertainties are modelled to reflect the real-world conditions. The ACC system architecture and control system design method are then proposed. Deep reinforcement learning is applied for the controller design in light of its great potential in describing the complex non-linear control task and handling the uncertainties. Kalman filters and recurrent policies with a Long-Short-Term-Memory network are applied to cope with uncertain measurements. The first method estimates the state information before feeding it back to the controller agent, while the latter incorporates the state estimator into the controller to actively consider the uncertainties while making decisions.
A numerical simulation approach is adopted to theoretically assess the performance of the proposed ACC systems. A traffic disturbance event and multiple levels of measurement noise are considered in the experiment. To analyze the performance in terms of string stability and ride comfort and understand the car-following behavior mechanism resulted from the proposed systems, a quantitative analysis framework is developed.
The evaluation results demonstrate the applied learning-based approach succeeds to train ACC control policies which can ensure string stability. It is also found that the multi-anticipation ability significantly improves the string stability and ride comfort performance. In the scenarios with measurement noise, systems using the tuned Kalman filters exhibit the ideal level of string stability performance. However, ride comfort cannot be guaranteed in scenarios with large measurement noise. On the other hand, systems using recurrent policies can better ensure ride comfort performance while maintaining string stability at certain levels. Based on the results, the performance limits of the proposed ACC systems in the handling of measurement uncertainties are explored. In addition, with the different policy training setups, the trade-off between these two performance aspects is shown.
The findings of this study are anticipated to trigger the development of advanced multi-leader ACC system by automakers, sensor manufacturers, and traffic engineers. Future work can be directed to an enhanced controller design. Robustness of the systems with respect to other sources of measurement uncertainties, more types of traffic disturbance, and platoon heterogeneity is worth further design consideration and analysis. ...
The possible sensor technology which can collect the second leader measurements is first investigated. Based on the considered setup, the measurement uncertainties are modelled to reflect the real-world conditions. The ACC system architecture and control system design method are then proposed. Deep reinforcement learning is applied for the controller design in light of its great potential in describing the complex non-linear control task and handling the uncertainties. Kalman filters and recurrent policies with a Long-Short-Term-Memory network are applied to cope with uncertain measurements. The first method estimates the state information before feeding it back to the controller agent, while the latter incorporates the state estimator into the controller to actively consider the uncertainties while making decisions.
A numerical simulation approach is adopted to theoretically assess the performance of the proposed ACC systems. A traffic disturbance event and multiple levels of measurement noise are considered in the experiment. To analyze the performance in terms of string stability and ride comfort and understand the car-following behavior mechanism resulted from the proposed systems, a quantitative analysis framework is developed.
The evaluation results demonstrate the applied learning-based approach succeeds to train ACC control policies which can ensure string stability. It is also found that the multi-anticipation ability significantly improves the string stability and ride comfort performance. In the scenarios with measurement noise, systems using the tuned Kalman filters exhibit the ideal level of string stability performance. However, ride comfort cannot be guaranteed in scenarios with large measurement noise. On the other hand, systems using recurrent policies can better ensure ride comfort performance while maintaining string stability at certain levels. Based on the results, the performance limits of the proposed ACC systems in the handling of measurement uncertainties are explored. In addition, with the different policy training setups, the trade-off between these two performance aspects is shown.
The findings of this study are anticipated to trigger the development of advanced multi-leader ACC system by automakers, sensor manufacturers, and traffic engineers. Future work can be directed to an enhanced controller design. Robustness of the systems with respect to other sources of measurement uncertainties, more types of traffic disturbance, and platoon heterogeneity is worth further design consideration and analysis. ...
Adaptive Cruise Control (ACC) relieves human drivers’ tasks by taking over the control of the throttle and braking of the vehicles automatically. However, it has been demonstrated in many empirical studies that current production ACC systems fail to guarantee string stability. It is believed that if vehicles can take the longitudinal dynamics further downstream into account and react to the propagating disturbance earlier, the string stability in the platoon may be improved. Instead of relying on inter-vehicle communication technologies, the ego-vehicle should be able to detect the second leading vehicle by leveraging the power of on-board sensors. Still, the second leader measurements can be highly erroneous. Therefore, it is important to consider the entailed measurement uncertainties when designing and evaluating such ACC systems. This study proposes several ACC systems which possess the property of multi-anticipation and uncertainty handling.
The possible sensor technology which can collect the second leader measurements is first investigated. Based on the considered setup, the measurement uncertainties are modelled to reflect the real-world conditions. The ACC system architecture and control system design method are then proposed. Deep reinforcement learning is applied for the controller design in light of its great potential in describing the complex non-linear control task and handling the uncertainties. Kalman filters and recurrent policies with a Long-Short-Term-Memory network are applied to cope with uncertain measurements. The first method estimates the state information before feeding it back to the controller agent, while the latter incorporates the state estimator into the controller to actively consider the uncertainties while making decisions.
A numerical simulation approach is adopted to theoretically assess the performance of the proposed ACC systems. A traffic disturbance event and multiple levels of measurement noise are considered in the experiment. To analyze the performance in terms of string stability and ride comfort and understand the car-following behavior mechanism resulted from the proposed systems, a quantitative analysis framework is developed.
The evaluation results demonstrate the applied learning-based approach succeeds to train ACC control policies which can ensure string stability. It is also found that the multi-anticipation ability significantly improves the string stability and ride comfort performance. In the scenarios with measurement noise, systems using the tuned Kalman filters exhibit the ideal level of string stability performance. However, ride comfort cannot be guaranteed in scenarios with large measurement noise. On the other hand, systems using recurrent policies can better ensure ride comfort performance while maintaining string stability at certain levels. Based on the results, the performance limits of the proposed ACC systems in the handling of measurement uncertainties are explored. In addition, with the different policy training setups, the trade-off between these two performance aspects is shown.
The findings of this study are anticipated to trigger the development of advanced multi-leader ACC system by automakers, sensor manufacturers, and traffic engineers. Future work can be directed to an enhanced controller design. Robustness of the systems with respect to other sources of measurement uncertainties, more types of traffic disturbance, and platoon heterogeneity is worth further design consideration and analysis.
The possible sensor technology which can collect the second leader measurements is first investigated. Based on the considered setup, the measurement uncertainties are modelled to reflect the real-world conditions. The ACC system architecture and control system design method are then proposed. Deep reinforcement learning is applied for the controller design in light of its great potential in describing the complex non-linear control task and handling the uncertainties. Kalman filters and recurrent policies with a Long-Short-Term-Memory network are applied to cope with uncertain measurements. The first method estimates the state information before feeding it back to the controller agent, while the latter incorporates the state estimator into the controller to actively consider the uncertainties while making decisions.
A numerical simulation approach is adopted to theoretically assess the performance of the proposed ACC systems. A traffic disturbance event and multiple levels of measurement noise are considered in the experiment. To analyze the performance in terms of string stability and ride comfort and understand the car-following behavior mechanism resulted from the proposed systems, a quantitative analysis framework is developed.
The evaluation results demonstrate the applied learning-based approach succeeds to train ACC control policies which can ensure string stability. It is also found that the multi-anticipation ability significantly improves the string stability and ride comfort performance. In the scenarios with measurement noise, systems using the tuned Kalman filters exhibit the ideal level of string stability performance. However, ride comfort cannot be guaranteed in scenarios with large measurement noise. On the other hand, systems using recurrent policies can better ensure ride comfort performance while maintaining string stability at certain levels. Based on the results, the performance limits of the proposed ACC systems in the handling of measurement uncertainties are explored. In addition, with the different policy training setups, the trade-off between these two performance aspects is shown.
The findings of this study are anticipated to trigger the development of advanced multi-leader ACC system by automakers, sensor manufacturers, and traffic engineers. Future work can be directed to an enhanced controller design. Robustness of the systems with respect to other sources of measurement uncertainties, more types of traffic disturbance, and platoon heterogeneity is worth further design consideration and analysis.
Master thesis
(2022)
-
H. Haifawi, F. Fioranelli, Alexander Yarovoy , Rob van der Meer, J.F.P. Kooij
Drone detection and tracking systems are nowadays a requirement in most public, private and political events, because of the increasing risk of unintentional or malicious misuse of these platforms. Moreover, in order to ensure adequate protection, full spatial coverage is a must for every such system. However, the research literature focuses on staring radars that have a limited field of view, but which yield rich target information via time-frequency distributions that facilitate the target recognition task. In this thesis, surveillance radars that offer full spatial coverage are presented, albeit their usage for classification is made more complex because of the rotating nature of their antennas which limits the dwell time on targets.
Additionally, due to the incredible fast growth of the drone market, novel counter-drone radars that are able to jointly localize and classify small targets while on-the-move now represent a highly in-demand remote sensing system. Nonetheless, surveillance sensors anchored on moving vehicles are a brand-new technology that is currently being developed. This work therefore investigates surveillance systems in a novel scenario, and presents the technological challenges alongside the proposed solutions to achieve reliable object detection via grounded counter-drone radars on-the-move. Specifically, the required pre-processing steps to remove the clutter from the data while the radar is rotating and moving on the ground are developed and discussed.
In the end, the joint detection and classification problem is traditionally solved separately by different algorithms due to the computational complexity of the task. This thesis project presents a novel framework that localizes and labels drones in an unified pipeline under the umbrella of object detection via computer vision, and that is able to operate while being static or on-the-move. Thus, an end-to-end radar data processing architecture that is robust against homogeneity constraints and based on You Only Look Once (YOLO) model is used to perform object detection in real-time. In brief, this work opens new avenues towards multi-class and multi-instance plot-based target detection and classification by transferring cross-disciplinary algorithms from computer vision into remote sensing.
...
Additionally, due to the incredible fast growth of the drone market, novel counter-drone radars that are able to jointly localize and classify small targets while on-the-move now represent a highly in-demand remote sensing system. Nonetheless, surveillance sensors anchored on moving vehicles are a brand-new technology that is currently being developed. This work therefore investigates surveillance systems in a novel scenario, and presents the technological challenges alongside the proposed solutions to achieve reliable object detection via grounded counter-drone radars on-the-move. Specifically, the required pre-processing steps to remove the clutter from the data while the radar is rotating and moving on the ground are developed and discussed.
In the end, the joint detection and classification problem is traditionally solved separately by different algorithms due to the computational complexity of the task. This thesis project presents a novel framework that localizes and labels drones in an unified pipeline under the umbrella of object detection via computer vision, and that is able to operate while being static or on-the-move. Thus, an end-to-end radar data processing architecture that is robust against homogeneity constraints and based on You Only Look Once (YOLO) model is used to perform object detection in real-time. In brief, this work opens new avenues towards multi-class and multi-instance plot-based target detection and classification by transferring cross-disciplinary algorithms from computer vision into remote sensing.
...
Drone detection and tracking systems are nowadays a requirement in most public, private and political events, because of the increasing risk of unintentional or malicious misuse of these platforms. Moreover, in order to ensure adequate protection, full spatial coverage is a must for every such system. However, the research literature focuses on staring radars that have a limited field of view, but which yield rich target information via time-frequency distributions that facilitate the target recognition task. In this thesis, surveillance radars that offer full spatial coverage are presented, albeit their usage for classification is made more complex because of the rotating nature of their antennas which limits the dwell time on targets.
Additionally, due to the incredible fast growth of the drone market, novel counter-drone radars that are able to jointly localize and classify small targets while on-the-move now represent a highly in-demand remote sensing system. Nonetheless, surveillance sensors anchored on moving vehicles are a brand-new technology that is currently being developed. This work therefore investigates surveillance systems in a novel scenario, and presents the technological challenges alongside the proposed solutions to achieve reliable object detection via grounded counter-drone radars on-the-move. Specifically, the required pre-processing steps to remove the clutter from the data while the radar is rotating and moving on the ground are developed and discussed.
In the end, the joint detection and classification problem is traditionally solved separately by different algorithms due to the computational complexity of the task. This thesis project presents a novel framework that localizes and labels drones in an unified pipeline under the umbrella of object detection via computer vision, and that is able to operate while being static or on-the-move. Thus, an end-to-end radar data processing architecture that is robust against homogeneity constraints and based on You Only Look Once (YOLO) model is used to perform object detection in real-time. In brief, this work opens new avenues towards multi-class and multi-instance plot-based target detection and classification by transferring cross-disciplinary algorithms from computer vision into remote sensing.
Additionally, due to the incredible fast growth of the drone market, novel counter-drone radars that are able to jointly localize and classify small targets while on-the-move now represent a highly in-demand remote sensing system. Nonetheless, surveillance sensors anchored on moving vehicles are a brand-new technology that is currently being developed. This work therefore investigates surveillance systems in a novel scenario, and presents the technological challenges alongside the proposed solutions to achieve reliable object detection via grounded counter-drone radars on-the-move. Specifically, the required pre-processing steps to remove the clutter from the data while the radar is rotating and moving on the ground are developed and discussed.
In the end, the joint detection and classification problem is traditionally solved separately by different algorithms due to the computational complexity of the task. This thesis project presents a novel framework that localizes and labels drones in an unified pipeline under the umbrella of object detection via computer vision, and that is able to operate while being static or on-the-move. Thus, an end-to-end radar data processing architecture that is robust against homogeneity constraints and based on You Only Look Once (YOLO) model is used to perform object detection in real-time. In brief, this work opens new avenues towards multi-class and multi-instance plot-based target detection and classification by transferring cross-disciplinary algorithms from computer vision into remote sensing.
Knowledge Distillation (KD) is a well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student. KD has proven to be an effective technique to significantly improve the student's performance for various tasks including object detection. As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an L_p-norm distance between teacher and student activations during training. In this paper, we propose a replacement for the pixel-wise independent L_p-norm based on structural similarity (SSIM). By taking into account additional contrast and structural cues, more knowledge within intermediate feature maps can be preserved. Extensive experiments on MSCOCO demonstrate the effectiveness of our method across different training schemes and architectures.
Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard L_p-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model. ...
Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard L_p-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model. ...
Knowledge Distillation (KD) is a well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student. KD has proven to be an effective technique to significantly improve the student's performance for various tasks including object detection. As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an L_p-norm distance between teacher and student activations during training. In this paper, we propose a replacement for the pixel-wise independent L_p-norm based on structural similarity (SSIM). By taking into account additional contrast and structural cues, more knowledge within intermediate feature maps can be preserved. Extensive experiments on MSCOCO demonstrate the effectiveness of our method across different training schemes and architectures.
Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard L_p-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model.
Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard L_p-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model.
Pedestrian trajectory prediction is essential for developing safe autonomous driving systems. Such trajectories depend on various contextual cues, among which surrounding objects.
This work proposes the first pedestrian trajectory prediction method in the 2D on-board domain that models interactions between the pedestrian and surrounding static- and dynamic- contextual objects using a graph-based approach. Our two-stream model separately encodes past motion history and interactions. The encoded information from both streams is fused and decoded to generate future pedestrian trajectories. The interactions are modeled using spatial graphs, which are temporally connected using a Gated Recurrent Unit. The graph nodes represent the pedestrian and contextual objects, and the edges represent the interaction importance between nodes.
In experiments on the PIE and JAAD_full dataset, it is shown that our graph-based interaction-aware trajectory prediction method outperforms all considered baselines on nearly all metrics. Moreover, the performance gain on JAAD_full is most significant for the close-by pedestrians. Finally, modeling the interactions with all considered contextual objects, i.e. vehicles, crosswalks, and traffic lights, improves trajectory prediction performance most compared to only using a subset of these objects. ...
This work proposes the first pedestrian trajectory prediction method in the 2D on-board domain that models interactions between the pedestrian and surrounding static- and dynamic- contextual objects using a graph-based approach. Our two-stream model separately encodes past motion history and interactions. The encoded information from both streams is fused and decoded to generate future pedestrian trajectories. The interactions are modeled using spatial graphs, which are temporally connected using a Gated Recurrent Unit. The graph nodes represent the pedestrian and contextual objects, and the edges represent the interaction importance between nodes.
In experiments on the PIE and JAAD_full dataset, it is shown that our graph-based interaction-aware trajectory prediction method outperforms all considered baselines on nearly all metrics. Moreover, the performance gain on JAAD_full is most significant for the close-by pedestrians. Finally, modeling the interactions with all considered contextual objects, i.e. vehicles, crosswalks, and traffic lights, improves trajectory prediction performance most compared to only using a subset of these objects. ...
Pedestrian trajectory prediction is essential for developing safe autonomous driving systems. Such trajectories depend on various contextual cues, among which surrounding objects.
This work proposes the first pedestrian trajectory prediction method in the 2D on-board domain that models interactions between the pedestrian and surrounding static- and dynamic- contextual objects using a graph-based approach. Our two-stream model separately encodes past motion history and interactions. The encoded information from both streams is fused and decoded to generate future pedestrian trajectories. The interactions are modeled using spatial graphs, which are temporally connected using a Gated Recurrent Unit. The graph nodes represent the pedestrian and contextual objects, and the edges represent the interaction importance between nodes.
In experiments on the PIE and JAAD_full dataset, it is shown that our graph-based interaction-aware trajectory prediction method outperforms all considered baselines on nearly all metrics. Moreover, the performance gain on JAAD_full is most significant for the close-by pedestrians. Finally, modeling the interactions with all considered contextual objects, i.e. vehicles, crosswalks, and traffic lights, improves trajectory prediction performance most compared to only using a subset of these objects.
This work proposes the first pedestrian trajectory prediction method in the 2D on-board domain that models interactions between the pedestrian and surrounding static- and dynamic- contextual objects using a graph-based approach. Our two-stream model separately encodes past motion history and interactions. The encoded information from both streams is fused and decoded to generate future pedestrian trajectories. The interactions are modeled using spatial graphs, which are temporally connected using a Gated Recurrent Unit. The graph nodes represent the pedestrian and contextual objects, and the edges represent the interaction importance between nodes.
In experiments on the PIE and JAAD_full dataset, it is shown that our graph-based interaction-aware trajectory prediction method outperforms all considered baselines on nearly all metrics. Moreover, the performance gain on JAAD_full is most significant for the close-by pedestrians. Finally, modeling the interactions with all considered contextual objects, i.e. vehicles, crosswalks, and traffic lights, improves trajectory prediction performance most compared to only using a subset of these objects.
Wi-Closure: wireless sensing for multi-robot map matching
Enabling fast and reliable search of inter-robot loop closures in repetitive environments
This thesis proposes a novel algorithm, Wi-Closure, to improve computational efficiency and robustness of map matching in multi-robot SLAM. Current state-of-the-art techniques connect maps with inter-robot loop closures, that are usually found through place recognition. Wi-Closure decreases the computational overhead of these approaches by pruning the search space of potential loop closures, prior to evaluation by a typical place recognition algorithm. Wi-Closure achieves this by identifying where trajectories are close to each other through sensing spatial information directly from the wireless communication signal. Then, place recognition is only performed on scans taken at locations close to each other. Wireless sensing provides information even when operating in non-line-of-sight or without existing communication infrastructure. The validity of Wi-Closure is demonstrated in simulation and hardware experiments. Results show that using Wi-closure greatly reduces computation time, by 54% in simulation and by 77% in hardware, compared with a multi-robot SLAM baseline. Importantly, this is achieved without sacrificing accuracy. Using Wi-Closure reduces absolute trajectory estimation error by 99% in simulation and 89% in hardware experiments. This improvement is due in part to Wi-Closure’s ability to avoid catastrophic optimization failure that typically occurs with classical approaches in challenging repetitive environments.
...
This thesis proposes a novel algorithm, Wi-Closure, to improve computational efficiency and robustness of map matching in multi-robot SLAM. Current state-of-the-art techniques connect maps with inter-robot loop closures, that are usually found through place recognition. Wi-Closure decreases the computational overhead of these approaches by pruning the search space of potential loop closures, prior to evaluation by a typical place recognition algorithm. Wi-Closure achieves this by identifying where trajectories are close to each other through sensing spatial information directly from the wireless communication signal. Then, place recognition is only performed on scans taken at locations close to each other. Wireless sensing provides information even when operating in non-line-of-sight or without existing communication infrastructure. The validity of Wi-Closure is demonstrated in simulation and hardware experiments. Results show that using Wi-closure greatly reduces computation time, by 54% in simulation and by 77% in hardware, compared with a multi-robot SLAM baseline. Importantly, this is achieved without sacrificing accuracy. Using Wi-Closure reduces absolute trajectory estimation error by 99% in simulation and 89% in hardware experiments. This improvement is due in part to Wi-Closure’s ability to avoid catastrophic optimization failure that typically occurs with classical approaches in challenging repetitive environments.
This thesis research proposes a new method for a controlling an agricultural robot using computer vision. The robot has to follow and simultaneously reel in a hose, which lies on a grass field. The hose that has to be followed, is attached to the robot itself. The trajectory of the hose is captured by a monocular camera and is extracted from the image by using a vision pipeline. The vision pipeline consists out of semantic segmenting in combination with a clustering algorithm and polynomial regression to find the trajectory of the hose. This trajectory is used as an input for the PID controller to control the motion of the robot. The presented method of controlling an agricultural robot by using vision control is tested and validated and performs sufficient, when the robot drives with a velocity of 0.2 m/s and when the hose is not partial occluded by long grass.
...
This thesis research proposes a new method for a controlling an agricultural robot using computer vision. The robot has to follow and simultaneously reel in a hose, which lies on a grass field. The hose that has to be followed, is attached to the robot itself. The trajectory of the hose is captured by a monocular camera and is extracted from the image by using a vision pipeline. The vision pipeline consists out of semantic segmenting in combination with a clustering algorithm and polynomial regression to find the trajectory of the hose. This trajectory is used as an input for the PID controller to control the motion of the robot. The presented method of controlling an agricultural robot by using vision control is tested and validated and performs sufficient, when the robot drives with a velocity of 0.2 m/s and when the hose is not partial occluded by long grass.
Bathymetry SLAM using reduced rank Gaussian Processes and DVL range measurements
For real-time underwater position estimation
Underwater position estimation is challenging due to the absence of Global Navigation Satellite System (GNSS) signals. Underwater vehicles are typically equipped with a Doppler Velocity Log (DVL) that measures the velocity relative to the seafloor. Aside from the velocity, the DVL also measures the range of each of the four beams. When compared against a bathymetry height map, these measured ranges provide additional information enabling improving position estimates. Unfortunately, surveys often occur in areas where detailed bathymetry maps are unavailable. In these cases, bathymetry Simultaneous Localization and Mapping (SLAM) could be used to improve the position estimates compared to the velocity integration position. With SLAM, the map is being estimated during the mission while at the same time using the map for position determination.
In this thesis, reduced rank Gaussian processes (GPs) are used as map representation for SLAM. The downside of regular GPs is a time complexity of O(n^3) , reduced rank GPs improve the computation performance. GPs provide Gaussian distributions at any point, leading to a neat integration with probabilistic SLAM algorithms. To the best of the author’s knowledge, this has not been used in bathymetry SLAM. This report investigates how reduced rank approximated GPs, representing the bathymetry, can be integrated into a SLAM algorithm to improve the position estimates of an underwater vehicle equipped with a DVL and low-quality gyroscopes.
A squared exponential kernel is used as GPs model of the bathymetry. The reduced rank approximation is vital for real-time SLAM performance. This map representation is integrated with a Rao Blackwellized particle filter (RBPF) that estimates both the underwater vehicle’s trajectory and the bathymetry map.
The SLAM algorithm is evaluated using data from an underwater vehicle operated at the surface such that a GNSS reference position is available. Experiments of the SLAM algorithm show a reduced position error compared to the GNSS reference. The resulting algorithm has a computation time of up to 30 times faster than the Autonomous Underwater Vehicle (AUV) collects data while improving position estimates. This concludes that the RBPF using reduced rank GPs is capable of onboard improved position estimation on underwater vehicles. ...
In this thesis, reduced rank Gaussian processes (GPs) are used as map representation for SLAM. The downside of regular GPs is a time complexity of O(n^3) , reduced rank GPs improve the computation performance. GPs provide Gaussian distributions at any point, leading to a neat integration with probabilistic SLAM algorithms. To the best of the author’s knowledge, this has not been used in bathymetry SLAM. This report investigates how reduced rank approximated GPs, representing the bathymetry, can be integrated into a SLAM algorithm to improve the position estimates of an underwater vehicle equipped with a DVL and low-quality gyroscopes.
A squared exponential kernel is used as GPs model of the bathymetry. The reduced rank approximation is vital for real-time SLAM performance. This map representation is integrated with a Rao Blackwellized particle filter (RBPF) that estimates both the underwater vehicle’s trajectory and the bathymetry map.
The SLAM algorithm is evaluated using data from an underwater vehicle operated at the surface such that a GNSS reference position is available. Experiments of the SLAM algorithm show a reduced position error compared to the GNSS reference. The resulting algorithm has a computation time of up to 30 times faster than the Autonomous Underwater Vehicle (AUV) collects data while improving position estimates. This concludes that the RBPF using reduced rank GPs is capable of onboard improved position estimation on underwater vehicles. ...
Underwater position estimation is challenging due to the absence of Global Navigation Satellite System (GNSS) signals. Underwater vehicles are typically equipped with a Doppler Velocity Log (DVL) that measures the velocity relative to the seafloor. Aside from the velocity, the DVL also measures the range of each of the four beams. When compared against a bathymetry height map, these measured ranges provide additional information enabling improving position estimates. Unfortunately, surveys often occur in areas where detailed bathymetry maps are unavailable. In these cases, bathymetry Simultaneous Localization and Mapping (SLAM) could be used to improve the position estimates compared to the velocity integration position. With SLAM, the map is being estimated during the mission while at the same time using the map for position determination.
In this thesis, reduced rank Gaussian processes (GPs) are used as map representation for SLAM. The downside of regular GPs is a time complexity of O(n^3) , reduced rank GPs improve the computation performance. GPs provide Gaussian distributions at any point, leading to a neat integration with probabilistic SLAM algorithms. To the best of the author’s knowledge, this has not been used in bathymetry SLAM. This report investigates how reduced rank approximated GPs, representing the bathymetry, can be integrated into a SLAM algorithm to improve the position estimates of an underwater vehicle equipped with a DVL and low-quality gyroscopes.
A squared exponential kernel is used as GPs model of the bathymetry. The reduced rank approximation is vital for real-time SLAM performance. This map representation is integrated with a Rao Blackwellized particle filter (RBPF) that estimates both the underwater vehicle’s trajectory and the bathymetry map.
The SLAM algorithm is evaluated using data from an underwater vehicle operated at the surface such that a GNSS reference position is available. Experiments of the SLAM algorithm show a reduced position error compared to the GNSS reference. The resulting algorithm has a computation time of up to 30 times faster than the Autonomous Underwater Vehicle (AUV) collects data while improving position estimates. This concludes that the RBPF using reduced rank GPs is capable of onboard improved position estimation on underwater vehicles.
In this thesis, reduced rank Gaussian processes (GPs) are used as map representation for SLAM. The downside of regular GPs is a time complexity of O(n^3) , reduced rank GPs improve the computation performance. GPs provide Gaussian distributions at any point, leading to a neat integration with probabilistic SLAM algorithms. To the best of the author’s knowledge, this has not been used in bathymetry SLAM. This report investigates how reduced rank approximated GPs, representing the bathymetry, can be integrated into a SLAM algorithm to improve the position estimates of an underwater vehicle equipped with a DVL and low-quality gyroscopes.
A squared exponential kernel is used as GPs model of the bathymetry. The reduced rank approximation is vital for real-time SLAM performance. This map representation is integrated with a Rao Blackwellized particle filter (RBPF) that estimates both the underwater vehicle’s trajectory and the bathymetry map.
The SLAM algorithm is evaluated using data from an underwater vehicle operated at the surface such that a GNSS reference position is available. Experiments of the SLAM algorithm show a reduced position error compared to the GNSS reference. The resulting algorithm has a computation time of up to 30 times faster than the Autonomous Underwater Vehicle (AUV) collects data while improving position estimates. This concludes that the RBPF using reduced rank GPs is capable of onboard improved position estimation on underwater vehicles.