Autonomous exploration enables unmanned aerial vehicles (UAVs) to map unknown environments without human intervention. While state-of-the-art algorithms primarily focus on maximizing coverage or minimizing exploration time, they often overlook energy efficiency, a critical constraint for battery-powered UAVs. This thesis introduces EAAE, a modular Energy-Aware Autonomous Exploration framework that explicitly incorporates energy consumption into the exploration decision-making process. By combining frontier detection, a K-means divisive clustering algorithm, energy-aware target selection, and dual-layer planning, the framework balances information gain with energy cost. The algorithm is evaluated in simulation using the Agilicious control stack, with a physically realistic power model based on rotor speeds. Two 3D environments of varying complexity are used to compare EAAE against two baselines, an information-gain-based frontier method and distance-based frontier methods. Results show that EAAE consistently achieves the lowest total energy consumption while maintaining competitive or lower exploration times. Moreover, it demonstrates similar performance in terms of information entropy and reduced power variability, contributing to more efficient and robust mapping. This work highlights the importance of integrating energy-awareness into exploration pipelines and provides a foundation for further research into long-endurance autonomous aerial missions.

Generative AI has revolutionized domains such as language, vision, and audio; yet, its application to the field of haptics, specifically signals for friction modulation devices, remains barely explored. A generative model could alleviate the issues associated with recording friction-based texture signals, such as the expenses of recording equipment and the limitation to lab environments, which significantly constrain the diversity of texture signals that can be rendered on friction modulation haptic devices. We propose a generative latent diffusion model called DreamTexture. The model is conditioned on a feature vector derived from a psychophysical perceptual space, where each dimension corresponds to an adjective pair (e.g., Rough–Smooth, Sticky–Slippery). We investigate whether DreamTexture can synthesize friction signals that align with users’ perceptual expectations, despite the subjective nature of tactile experiences, influenced by individual skin properties and linguistic interpretation. Moreover, DreamTexture is optimized for real-time inference on commercially available hardware, making haptic content creation more scalable and accessible. Our findings indicate that the diffusion process lends itself well to the efficient generation of one-dimensional friction signals and produces realistic signals, but it exhibits limitations in fully capturing the variability inherent in the input space.

When a robot is navigating through an environment crowded by obstacles and other robots, it can get stuck while trying to go from point A to point B. If they get stuck in place, this is often called a deadlock. If they rather get stuck in a pattern of repetitive movements, this is called a livelock. Those locking situations are detrimental to the efficiency of the robot as it loses time and possibly energy. Locking situations should therefore be avoided as much as possible or be recovered from as quickly as possible. Some planning algorithms are more prone to locking situations than others, in particular, decentralised learning-based planners such as DCC, PRIMAL and MAPPER. This is caused by their lack of a mech- anism that explicitly coordinates different agents and their limited perception of the state of the envi- ronment. It would be beneficial if the tendency to get dead- or livelocked could be curbed with simple techniques that can be applied to any planner that works in largely the same manner. Hence, this research proposes and evaluates four methods: punishing actions that lead to an agent getting locked during training (Lock Punishment or LP), training on locking situations encountered during training to teach the model to recover from such a situation (Targeted Training or TT), teach an already trained model to recover from locking situation (Targeted Fine-tuning or TF), and placing obstacles that are only visible to an agent that has just entered a locking situation to encourage it to move away from that location (Phantom Obstacles or PO). All of these methods are evaluated on the same benchmark to see which singular or combination of techniques performs best. In the end, the most effective methods turned out to be PO and a combination of TT and PO. PO demonstrates an increase of 1.6% in its mean success-rate and a decrease of 3% in its computation times over the baseline, and TT+PO’s mean Sum of Costs, Sum of Fuel, and Makespan are lower than the baseline by 0.8%, 1.2% and 2.1% respectively. Although LP+PO has relatively poor performance in most metrics, it reduces the number of locks the baseline encounters by 66%, thus achieving the lowest number of locks.

Soft continuum robots provide a compelling solution for safe interactions with unknown and unstructured environments, owing to their compliance and infinite degrees of freedom. However, the non-linear deformation and inherent underactuation pose a persistent challenge for real-time control and state estimation. To date, continuum structures are typically discretized using a fixed-parameter kinematic model, introducing a trade-off between model accuracy and computational efficiency.

This letter presents an adaptive kinematic modeling framework to address the challenge of underactuation in a different way - not by increasing model resolution or complexity - but by making the model parametric with respect to a set of parameters that are dynamically adapted using a novel inverse kinematic adaptive controller.

We formally prove stability of the adaptive controller and validate its performance through simulations and experiments, considering setpoint reaching tasks with variable end-effector payloads. Even though the approach introduces additional challenges, in comparison to the conventional fixed-parameter models presented in literature, the proposed solution enhances shape representation, redundancy resolution, and state inference while mitigating model complexity.

Three-year-old toddlers can effortlessly guide a toy train along a wooden track, whereas this slide-in-the-groove position tracking task requires a skilled operator using a teleoperated robot arm due to the lack of direct contact and force feedback. Although an autonomous robot can perform this task in a fixed setup, telerobotics is crucial for unknown environments where human control is essential, as humans provide the adaptability needed to handle unpredictable conditions. The introduction of torque-controlled motors and haptic devices has enhanced teleoperation by improving telepresence and immersion. Operators can perceive interaction forces through the primary position control input via the haptic device, while a secondary control input allows them to adjust the robot arm's impedance. This ability, known as teleimpedance, allows operators to control the robot’s physical interaction based on environmental context. A toddler naturally remains relaxed in the plane perpendicular to the train's forward direction, where gravity and the groove sides provide stability, preventing derailment and wheel damage. At the same time, they maintain firmness along the track for smooth forward movement. Tele-impedance enables the operator to achieve a similar balance. It allows adaptation of an optimal balance of low and high impedance in different axes of Cartesian space to match task demands. Current impedance control interfaces rely on complex muscle activity measurements, requiring long calibration procedures to map the operator’s arm stiffness to the robot arm. Other interfaces use hand-controlled input devices that must be operated in addition to the haptic device, reducing the operator's cognitive bandwidth for the position tracking task. Existing interfaces typically provide only partial stiffness control or introduce visual distractions. In contrast, we propose a novel visio-verbal interface that leverages gaze and speech, natural modes of interaction, to enable hands-free semi-autonomous control of translational stiffness in all three dimensions while maintaining visual attention on the position tracking task. The interface’s vision-language model (VLM) determines the three-dimensional robot endpoint stiffness by combining the operator's verbal intent with gaze estimates from a mobile eye tracker. We demonstrate a proof of concept for this approach. The hardware includes Tobii Pro Glasses 2 eye trackers, a Force Dimension sigma7 haptic position input interface, and a KUKA LBR iiwa collaborative robot arm equipped with a custom-built endpoint camera mount for the Realsense D455 camera and a 3D-printed peg to evaluate the interface in a 3D-printed U-shaped slot for a slide-in-the-groove task similar to guiding a toy train.

Quadrupedal robots have great potential for deployment in challenging environments. However, one of the most significant challenges these robots face is maintaining stability on slippery surfaces due to inaccurate friction estimation. This thesis investigates the role of accurate friction estimation in improving locomotion and reducing slip in quadrupedal robots. A Model Predictive Controller (MPC) with a friction-aware constraint update is proposed and evaluated in a simulation environment using a physics-based simulation with Open Dynamics Engine (ODE).

The experimental results demonstrate that incorporating real-time friction measurement and constraint in the MPC framework significantly reduces slip occurrences, decreases energy consumption, and improves overall locomotion stability. Statistical hypothesis tests, including paired t-tests and Bonferroni corrections, confirm the significance of these improvements. The findings suggest that integrating real-time friction estimation into quadrupedal robot controllers can enhance their robustness and reduce the need for explicit slip recovery strategies. Future work should focus on extending this approach to real-world scenarios, incorporating actual friction sensors, and testing in diverse terrains.

Trajectory prediction is a key element of autonomous vehicle systems, enabling them to anticipate and react to the movements of other road users. Robustness testing through adversarial methods is essential for evaluating the reliability of these prediction models. However, current approaches tend to focus solely on manipulating model inputs, which can generate unrealistic scenarios and overlook critical vulnerabilities. This limitation may result in incomplete assessments of model performance in real-world conditions. The specific effects of more comprehensive adversarial attacks on trajectory prediction models have not been thoroughly investigated. In this work, we demonstrate that by perturbing both model inputs and anticipated future states, we can uncover previously undetected weaknesses and provide a more realistic evaluation of model robustness. Our novel approach incorporates dynamical constraints and preserves tactical behaviors, enabling more effective and realistic adversarial attacks. We introduce new performance measures to assess the realism and impact of these adversarial trajectories. Testing our method on a state-of-the-art prediction model reveals significant increases in prediction errors and collision rates under adversarial conditions. Qualitative analysis further shows that our attacks can expose critical weaknesses, such as the model’s inability to detect potential collisions in what appear to be safe predictions. These results underscore the need for more comprehensive adversarial testing to better evaluate and improve the reliability of trajectory prediction models for autonomous vehicles. To support further research in this area, we provide an open-source framework for studying adversarial robustness in trajectory prediction. This work advances adversarial testing techniques, contributing to the safety and reliability of autonomous driving systems.

Currently, self-driving vehicles have trouble detecting partially and fully occluded objects such as pedestrians, vehicles, and static obstacles. It has been proven that a drone surveilling the area around the vehicle improves the vehicle's awareness of its surroundings. This work explores planning strategies for the drone and evaluates how much the strategies assist the vehicle. Previous work proposed a metric called PKL, which describes the awareness of a vehicle as a function of the detections of the vehicle using a planner model. Subtracting this metric calculated using detections done by the drone from the metric without these detections results in an awareness improvement metric. This metric was used to evaluate drone positions and therefore the drone planning strategies. It was found that aiming to hover directly above the autonomous vehicle improves the awareness of the vehicle measured by median PKL improvement by 1.8%. Using only the relative position of the autonomous vehicle to the drone, the velocity of the vehicle in x and y directions and the number of vehicles visible to the drone, an imitation learning-based strategy performed the best with 11.6% median PKL improvement. If the drone's planner knows the future position of the autonomous vehicle or the positions of all vehicles in the area, the median PKL improvement is 74.5% and 80.8%, respectively. Optimizing the trajectories using a genetic algorithm further improves the performance to 96.7%. From these numbers, we can conclude that aiming to stay directly above the vehicle does not benefit the autonomous vehicle the most. We show that intelligent drone trajectory planning strategies can be learned which improve the awareness of the autonomous vehicle and therefore the safety of the people in and around this vehicle.

Autonomous vehicles rely on prediction modules, in order to plan collision-free trajectories. Vehicle trajectory prediction models are multimodal, to account for the multiple route options and the inherent uncertainty in human behavior. The state-of-the-art prediction models are deep-learning models, which are susceptible to mode collapse, a phenomenon in which the model fails to output the full distribution of modes and only predicts the most likely one. Mode collapsing poses safety concerns for autonomous driving, as missed predictions could result in collisions. Most works have focused on addressing this issue by generating diverse predictions that cover various route options at the environmental level. However, there are no metrics for mode-collapse. Furthermore, little attention has been given to generating diversity in the interaction modes among agent trajectories. Additionally, the traditional distance-based metrics are heavily dependent on datasets and do not evaluate interactions between agents. To this end, we propose a novel evaluation framework that assesses the interaction modes of joint trajectory predictions, focusing only on the safety-critical interactions in a dataset. We introduce a metric for mode-collapse and time-based metrics for mode correctness and coverage, shedding light on the temporal dimension of the predictions. We test four multi-agent trajectory prediction models on the widely used nuScenes dataset and conclude that mode collapse happens. While the rate of correctly predicted interaction modes increases closer to the interaction event, there are still cases where the models are unable to predict the interaction mode even right before the interaction happens. With the introduction of our novel framework, researchers can now benchmark their models’ performance in predicting critical interactions. This provides new insights and perspectives, helping the holistic evaluation and interpretation of a model’s performance. Additionally, our work offers a new developmental direction for prediction models, aiming for greater consistency and accuracy in predicting agent interactions, thereby advancing the safety of autonomous driving systems. Our evaluation framework is available online at: https://github.com/MaartenHugenholtz/InteractionEval

While learning-based control techniques often outperform classical controller designs, safety requirements limit the acceptance of such methods in many applications. Recent developments address this issue through Certified Learning (CL), which combines a learning-based controller with formal methods to provide safety guarantees. This thesis focuses on the CL based on Control Barrier Functions (CBFs), as CBFs have been widely used for safety-critical systems. However, it is non-trivial to design a CBF. Utilizing neural networks as CBFs has
shown great success, but it necessitates their certification as CBFs. In this work, we leverage bound propagation techniques and the Branch-and-Bound scheme to efficiently verify that a neural network satisfies the conditions to be a CBF over the continuous state space. To accelerate training, we further present a framework that embeds the verification scheme into the training loop to synthesize and verify a neural CBF simultaneously. In particular, we
employ the verification scheme to identify partitions of the state space that are not guaranteed to satisfy the CBF conditions and expand the training dataset by incorporating additional data from these partitions. The neural network is then optimized using the augmented dataset to meet the CBF conditions. We show that for a non-linear control-affine system, our framework can efficiently certify a neural network as a CBF and render a larger safe set than state-of-the-art neural CBF works. We further employ our learned neural CBF to derive a safe controller to illustrate the practical use of our framework.

Unmanned aerial vehicles (UAVs), particularly quadcopters, are increasingly employed in diverse applications due to their manoeuvrability and affordability. However, their susceptibility to GPS jamming and the need for skilled operators limit their operational resilience. This paper presents a new design of an Image-Based Visual Servoing (IBVS) control system for autonomous quadcopter interception, utilizing only a monocular camera and an Inertial Measurement Unit (IMU). The system features a Multi-Axis PID Controller with acceleration limiting and a virtual plane projection method to decouple pitch and vertical motions, thereby enhancing interception accuracy and maintaining target visibility within the drone’s field of view (FOV). Furthermore, a perception module is designed to run Yolo-v8n and CSRT in parallel, followed by a Kalman filter, to deliver an accurate and robust representation of the target within the image, operating at 18 frames per second in Simulation-in-the-Loop (SITL) environments. Comprehensive SITL experiments and comparative analyses against recent IBVS algorithms demonstrate that the proposed proposed system achieves a 19% reduction in circular error probable (CEP) for static targets and superior performance in diverse scenarios involving moving targets. These findings validate the effectiveness of the proposed IBVS approach, offering a reliable and scalable solution for autonomous UAV interception in GPS-denied environments with potential applications in security, surveillance, and conflict zones.

A scenario-based testing approach can reduce the time required to obtain statistically significant evidence of the safety of Automated Driving Systems (ADS). Identifying these scenarios in an automated manner is a challenging task. Most methods on scenario classification do not work for complex scenarios with diverse environments (highways, urban) and interaction with other traffic agents. This is mirrored in their approaches which model an individual vehicle in relation to its environment, but neglect the interaction between multiple vehicles (e.g. cut-ins, stationary lead vehicle). Furthermore, existing datasets lack diversity and do not have per-frame annotations to accurately learn the start and end time of a scenario. We propose a method for complex traffic scenario classification that is able to model the interaction of a vehicle with the environment, as well as other agents. We use Graph Convolutional Networks to model spatial and temporal aspects of these scenarios. Expanding the nuScenes and Argoverse 2 driving datasets, we introduce a scenario-labeled dataset, which covers different driving environments and is annotated per frame. Training our method on this dataset, we present a promising baseline for future research on perframe complex scenario classification

Recent progress in multiple micro aerial vehicle (MAV) systems has demonstrated autonomous navigation in static environments. Yet, there are limited works regarding the autonomous navigation of multiple MAVs in dynamic and unknown environments. The challenge arises from the complexity of the motion planning problem, which requires the MAVs to coordinate with each other while avoiding dynamic obstacles. This thesis presents a novel risk-aware decentralized multi-agent motion planning framework to address this issue. For perception, we rely on a particle-based dynamic map, which utilizes particles to represent dynamic obstacles and predict future states of dynamic obstacles. Leveraging this map representation and the shared trajectory from other agents, we evaluate the future collision risk with dynamic obstacles and other agents in a coupled manner. During the planning phase, a risk-aware kino-dynamic A* algorithm tailored to the particle-based map representation is developed, ensuring dynamically feasible paths with risks under a given safe level. Subsequently, spatio-temporal safety corridors with maximum volume are optimized by inflating from path segments, taking map particles as constraints. These corridors act as constraints for the trajectory optimization problem, which is simplified to a convex optimization problem by using Bézier splines. The proposed method is thoroughly evaluated in simulation environments featuring various quantities and shapes of dynamic obstacles. Comparative results with state-of-the-art multi-agent planners that rely on precise obstacle observations demonstrate the efficiency and safety of the proposed method. Furthermore, the effectiveness of the proposed method is validated in a more realistic simulation environment with pedestrians, using depth cameras onboard for perception.

This research introduces a novel approach for 3D object detection utilizing language models, with a particular focus on addressing the challenges that have been encountered in the autonomous vehicle domain. The primary objective revolves around addressing the constraints associated with object detection models that rely heavily on labeled data and are resource-intensive, while also showing limited proficiency in recognizing new, unseen objects. The thesis presents the PointGLIP model, a novel integration of zero-shot and few-shot learning methodologies, that utilizes language models to augment object detection in 3D point cloud data. PointGLIP utilizes GLIP encoders, which are known for their capability to combine textual and visual data. This methodology facilitates the transfer of pre-trained information from a 2D domain to a 3D domain by converting point clouds into depth maps. This conversion process enables object detection without the need for extensive or any prior 3D training. The model’s ability to generalize and transfer learning from 2D to 3D environments is demonstrated by experiments conducted utilizing the nuScenes dataset. The results revealed that the model shows poor performance in the context of both zero-shot and few-shot learning. This shows that the model struggles to perform detections with depth maps which indicates a struggle in the transfer of knowledge from 2D to 3D contexts. In 2D, the integration of descriptive text through language models provides a unique approach to contextual understanding, though the outcomes demonstrated that further refinements are necessary for consistent reliability.