Skid-steer mobile robots present a unique challenge for reinforcement learning (RL) due to their nonholonomic constraints, dynamic wheel coupling, and high susceptibility to slip. Standard RL methods often struggle to achieve stable torque control in such settings, particularly under long-horizon tasks with sparse or delayed rewards. This thesis introduces a knowledge-assisted RL framework that systematically integrates expert demonstrations and curriculum learning to improve sample efficiency, training stability, and final policy performance.

The foundation of the method is a torque-based Deep Deterministic Policy Gradient (DDPG) agent, augmented through two key innovations: (1) KAMMA (Knowledge-Assisted Mixed Mode Actioning), a probabilistic switching mechanism that alternates between expert and learned actions to avoid interference artifacts and accelerate early-stage convergence; and (2) Curriculum-Driven System Identification, where the task complexity is gradually increased via staged velocity profiles to reveal underlying terrain-robot dynamics in a structured manner.

Experiments conducted in Isaac Sim demonstrate that this integrated KAMMA + Curriculum approach outperforms both baseline KA-DDPG and imitation-only variants across key metrics, including trajectory tracking error, policy smoothness, and convergence speed. The results confirm that combining staged learning with adaptive knowledge infusion enables robust torque-level control and offers a scalable template for learning-driven system identification in robotics.

This study presents the development and evaluation of an automated laser-based measurement system for determining the frequencies of Flexous oscillators while still in the wafer, using a Prusa I3 MK3 3D printer as a motion platform. Emphasis is placed on assessing the system's precision, repeatability, and edge detection accuracy. A lead screw-driven positioning mechanism with aluminium frame construction was designed to ensure high mechanical stability. Factors affecting positional accuracy, including microstepping, backlash, and structural deflection, are analysed and addressed. Calibration and homing tests confirmed sub-micrometre repeatability, with the Y-axis showing superior consistency. Movement accuracy tests revealed axis-dependent errors, influenced by mechanical and sensor-related factors. Edge detection performance was found to be sensitive to calibration speed, with slower movements significantly improving accuracy. The study concludes with recommendations for enhancing measurement precision through closed-loop control and predictive stopping logic, demonstrating the system’s potential for scalable, high-precision oscillator testing.

Force/torque sensors are essential tools that enable robots to effectively interact with their environments. Existing calibration methods often fail to capture inter-axis nonlinearities and coupling effects, particularly when available calibration data are sparse and discrete. To address this challenge, the presented approach employs a Deep Neural Network (DNN) that learns both the scaling and the direction of the input-output relationship. The method works by extracting the absolute magnitude and unit vector from the raw N-dimensional sensor output values, which can vary among sensors. The DNN takes this N-dimensional input and produces a 7-dimensional output—comprising a corrected 6D unit vector representing the desired force-torque direction and a scaling factor. The final measurement is then constructed by combining the output unit vector, the learned scaling factor, and the original input magnitude. This approach simplifies the calibration problem to a linear mapping along one axis, enabling the model to generalize well under limited training conditions while leveraging the DNN’s strength in capturing nonlinear inter-axis relationships. The proposed DNN was trained and evaluated on both artificially generated and real-world datasets, and its performance was compared to two baseline models: a commonly used linear transformation model and a comparative DNN approach from the literature. On generated data, the proposed DNN achieved an RMSE of 36.9 ± 3.44, outperforming the comparative DNN (48.3 ± 4.47) and the linear transformation model (62.3±0.76). Similar improvements were observed on the real-world dataset. Although these results are promising, they are based on artificially generated data and a single real-world dataset from one specific sensor. Further validation and more extensive testing are necessary. Nonetheless, the gains indicated here suggest meaningful potential for improved calibration strategies in force-controlled robotic applications, even under limited training conditions.

Reinforcement learning has emerged as a promising approach for enabling robots to learn from interactions with their environments, without relying on predefined behaviors. However, robots face significant challenges when learning directly from real-world interactions. Real-world learning is time-consuming and resource-intensive, often requiring extensive data collection over long periods. Additionally, the risks involved in trial-and-error learning in physical settings are high, as faulty policies can lead to safety issues or system damage. Simulations offer a safer and more efficient alternative, allowing robots to learn in simulated environments at faster-than-real-time speeds. Despite these benefits, simulations often serve as imperfect approximations of reality. As a result, robots may learn behaviors that exploit simulation-specific quirks, which may not perform well in real-world settings, creating difficulties in transferring learned behaviors from simulation to real environments—a challenge known as the sim-to-real gap. Several factors contribute to the sim-to-real gap, such as unmodeled physical phenomena like friction and deformation, and the asynchronous nature of real-world systems that simulations often fail to capture accurately. Additionally, using separate software stacks for simulation and deployment can unintentionally lead to discrepancies. Finally, simulating at faster-than-real-time speeds with asynchronous frameworks that distribute computation across multiple cores may also introduce inaccuracies without proper synchronization.

This thesis focuses on improving simulation tools and methodologies to enhance the efficiency and effectiveness of learning-based approaches in robotics. The work addresses key trade-offs between flexibility, speed, and accuracy in robotic simulations, which are critical for successfully transferring learned policies from simulation to real-world environments. Additionally, it introduces a strategy to improve resilience, ensuring that learned behaviors are robust to irrelevant and unknown dynamics.
By tackling these challenges, this thesis provides insights into the design of effective robotic simulators and presents contributions that help bridge the gap between simulated and real-world robotic learning.

Reinforcement Learning (RL) shows great potential for robotic manipulation tasks, yet it suffers from low sample efficiency and needs extensive exploration of state-action spaces. Some recent methods leverage the commonsense knowledge and reasoning abilities of Large Language Models (LLMs) to guide RL exploration toward more meaningful states. However, LLMs may generate semantically correct but physically infeasible plans, leading to unreliable solutions. In this paper, we propose \textit{Language-Guided exploration for Reinforcement Learning} (LGRL), a novel framework that utilizes LLMs' planning capability to directly guide RL exploration. This approach utilizes LLM planning at both the task and affordance levels, enhancing learning efficiency by directing RL agents toward semantically meaningful actions. Unlike previous methods that rely on the optimality of LLM-generated plans or rewards, LGRL corrects sub-optimality and explores multimodal affordance-level plans without human intervention.
We evaluated LGRL on pick-and-place tasks within standard RL benchmark environments, demonstrating significant improvements in both sample efficiency and success rates.

This paper introduces a new imitation learning framework based on energy-based generative models capable of generating complex, life-like, physics-dependent motions, through state-only expert motion trajectories. Our algorithm, called Noise-conditioned Energy-based Annealed Rewards (NEAR), constructs several perturbed versions of the expert's motion data distribution and learns smooth, and well-defined representations of the data distribution's energy function using denoising score matching. We propose to use these learnt energy functions as reward functions to learn imitation policies via reinforcement learning. We also present a strategy to gradually switch between the learnt energy functions, ensuring that the learnt rewards are always well-defined in the manifold of policy-generated samples, thereby improving the learnt policies. We evaluate our algorithm on complex humanoid tasks such as locomotion and martial arts and compare it with state-only adversarial imitation learning algorithms like Adversarial Motion Priors (AMP). Our framework sidesteps the optimisation challenges of conventional generative imitation learning techniques and produces results comparable to AMP in several quantitative metrics across multiple tasks. Finally, a portion of this paper also analyses the optimisation challenges of adversarial imitation learning algorithms, and discusses some previously under-explored failure modes, providing rigorous empirical results to back our argumentation. Code and videos are available at anishhdiwan.github.io/noise-conditioned-energy-based-annealed-rewards/

Training models for autonomous vehicles (AVs) necessitates substantial volumes of high-quality data due to the strong correlation between dataset size and model performance. However, acquiring such datasets is labor-intensive and expensive, requiring significant resources for collection and labeling. To optimize the utility of available data, augmenting the dataset or generating synthetic data presents a cost-effective and efficient solution. Traditional methods that operate within the RGB domain frequently overlook crucial information, such as object frequency, scene composition, and agent trajectories. To address these limitations, a pipeline employing controllable diffusion models and vehicle simulation software is proposed. This approach involves loading collected data into a physics-based simulator, which allows for augmentation beyond the pixel space into the structural space. The augmented simulated data is subsequently transformed back into the photorealistic domain using generative artificial intelligence. This process generates high-fidelity synthetic data, thereby enabling models to train effectively on an expanded and varied dataset, enhancing robustness through the increased variation. The proposed method is evaluated in both image and video domains to assess its effectiveness.

Autonomous vehicles rely on prediction modules, in order to plan collision-free trajectories. Vehicle trajectory prediction models are multimodal, to account for the multiple route options and the inherent uncertainty in human behavior. The state-of-the-art prediction models are deep-learning models, which are susceptible to mode collapse, a phenomenon in which the model fails to output the full distribution of modes and only predicts the most likely one. Mode collapsing poses safety concerns for autonomous driving, as missed predictions could result in collisions. Most works have focused on addressing this issue by generating diverse predictions that cover various route options at the environmental level. However, there are no metrics for mode-collapse. Furthermore, little attention has been given to generating diversity in the interaction modes among agent trajectories. Additionally, the traditional distance-based metrics are heavily dependent on datasets and do not evaluate interactions between agents. To this end, we propose a novel evaluation framework that assesses the interaction modes of joint trajectory predictions, focusing only on the safety-critical interactions in a dataset. We introduce a metric for mode-collapse and time-based metrics for mode correctness and coverage, shedding light on the temporal dimension of the predictions. We test four multi-agent trajectory prediction models on the widely used nuScenes dataset and conclude that mode collapse happens. While the rate of correctly predicted interaction modes increases closer to the interaction event, there are still cases where the models are unable to predict the interaction mode even right before the interaction happens. With the introduction of our novel framework, researchers can now benchmark their models’ performance in predicting critical interactions. This provides new insights and perspectives, helping the holistic evaluation and interpretation of a model’s performance. Additionally, our work offers a new developmental direction for prediction models, aiming for greater consistency and accuracy in predicting agent interactions, thereby advancing the safety of autonomous driving systems. Our evaluation framework is available online at: https://github.com/MaartenHugenholtz/InteractionEval

How can robots without expressive faces or bodies convey emotions? Why would it be useful if robots could express emotion? In the context of human-robot interaction, could emotional expression lead to a greater comprehension of robotic behaviors and intents? These are questions addressed by the field of affective robotics, which seeks to develop and establish naturalistic social interaction between robots and humans. Emotions can provide a natural communication modality to augment the multi-modal capabilities of social robots in a variety of domains.

Historically, the emphasis in the field has been on facial and bodily expressions, relying heavily on anthropomorphic or zoomorphic robot appearances. This presents a challenge, as most robots are designed with functionality in mind, often lacking expressive faces and bodies, which limits their ability to effectively convey emotions. This study investigates the potential for appearance-constrained robots to convey emotions through variations in motion, light, and sound parameters.

We conducted an experiment where participants rated the emotional qualities of a non-humanoid, faceless robot’s behaviors, which were manipulated through variations in motion, light, and sound parameters. Our approach is unique in that it adopts a bottom-up methodology similar to the work of Jack et al. on facial expressions. By systematically varying individual features and observing the resultant emotional perceptions, we aimed to discern the specific affective contributions of each parameter. Using machine-learning based regression models, we sought to predict the perceived emotional qualities based on these systematically varied parameters.

Our findings reveal that variations in motion parameters, particularly speed, significantly influence the perceived intensity of arousal, joy, and dominance. Light temperature was found to affect the perceived intensity of anger and joy, while sound pitch influenced perceptions of surprise and fear. The regression models showed varying degrees of success, with the random forest models often outperforming linear models but also exhibiting a higher tendency to overfit the training data. The linear models, while less prone to overfitting, struggled to capture the full complexity of the emotional responses. These findings suggest that non-anthropomorphic robots can indeed convey emotional qualities through controlled variations in their behaviors, though the strength and clarity of these emotions remain limited. Future research should focus on enhancing the expressiveness of these parameters and testing the models with new data to better understand their generalizability and effectiveness.

Surgical navigation involves transferring preoperative imaging data, along with preplanned information, onto the patient in the operating theater without using constant radiation. This technique has proven effective and is widely adopted across various surgical specialties. Research has shown a consistent trend in surgical robotics, with numerous initiatives using this technology to navigate the robot’s end-effector within the patient’s anatomy. For this purpose, commercially available surgical navigation systems are often employed. However, these systems, which are primarily dominated by optical tracking, are not necessarily suited for robotic systems and exhibit limitations such as low update frequency and line-of-sight issues. Additionally, performance reporting in current surgical robotic research is highly inconsistent, and clear guidelines are lacking. This research aims to develop a surgical robotic navigation system to work towards establishing a performance benchmark and systematically assess various error components as a first step toward guiding the field of surgical robotic navigation. To this end, two systems, the Haply System and the Dual-Robot System, have been developed and evaluated for technical accuracy and registration accuracy in both static and dynamic environments. Furthermore, sensor fusion methods have been explored to enhance performance in the Haply system. The results and analysis indicate that the Dual-Robot System is the most accurate in dynamic navigation and presents a viable alternative to optical tracking systems in terms of performance. However, its clinical adoptability remains questionable.

Autonomy in traffic (e.g., autonomous vehicles) could potentially benefit mobility, safety, accessibility and sustainability. However, the realisation of these advancements is highly dependent on how effective these autonomous vehicles interact with vulnerable road users such as pedestrians. Before we can understand how pedestrians will interact with autonomous vehicles, it is essential to understand how pedestrians interact among themselves in interactive traffic scenarios. Previous studies have focused on describing these scenarios with probabilistic trajectory prediction methods such as TrajFlow. However, these approaches often fall short in capturing the nuances of mutual interactions. Simple interaction models have been proposed that can describe these interactions, but neglect the influence of another person's intentions. To address this issue, in existing work the Communication-Enabled-Interaction (CEI) framework was proposed that describes interactions by modelling communication and a belief of another person's intentions. The idea of using beliefs in interaction modelling is based on the concept that people have a general but uncertain idea about the plans of other people. These beliefs are one of the fundamental aspects of the CEI framework and must therefore contain valuable information about possible decisions. That is why this study investigates the use of the probabilistic trajectory prediction method TrajFlow for the belief construction of the CEI framework. TrajFlow is trained on the belief-based Forking Paths dataset, integrated into the CEI framework, and tested in four simulated pedestrian interaction scenarios. The analysis shows that the framework is able to simulate plausible interaction behaviour, dealing with conflicting goals and trajectories in multiple simulations. By doing so, this study takes a positive step towards modelling pedestrian interactions and contributes to the broader goal of realising the benefits linked to autonomy in traffic.

This paper presents a hierarchical reinforcement learning framework for efficient robotic manipulation in sequential contact tasks. We leverage this hierarchical structure to sequentially execute behavior primitives with variable stiffness control capabilities for contact tasks. Our proposed approach relies on three key components: an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance. Through comprehensive training and evaluation, our framework learns efficient stiffness control capabilities and demonstrates improvements in learning efficiency, compositionality in primitive selection, and success rates compared to the state-of-the-art. The training environments include block lifting, door opening, object pushing, and surface cleaning. Real world evaluations further confirm the framework’s sim2real capability. This work lays the foundation for more adaptive and versatile robotic manipulation systems, with potential applications in more complex contact-based tasks.

This paper presents a novel Graph Optimal Transport (Graph OT) framework for analyzing and aligning plant structures across different growth stages and transformations. Our method extends existing graph matching techniques by incorporating domain-specific botanical features and employing a multi-scale matching strategy that captures both local and global structural characteristics. The framework combines multiple feature representations, including node descriptors, spectral embeddings, Node2Vec embeddings, and relative positions, to construct an augmented cost matrix for optimal transport based matching. We evaluated our approach on a dataset of 50 distinct plant structures under various transformations, including rotation, deformation, and partial matching scenarios.

The results indicate that our Graph OT framework significantly outperforms traditional optimal transport (OT) methods, achieving node-matching accuracy scores of 0.75 for rotated,
0.74 for deformed structures, 0.67 for cut structures, and 0.71 for structures with skipped nodes. Our approach demonstrates particular robustness in handling complex transformations. This method provides a powerful tool for botany applications such as crop management, growth modeling, and automated pruning systems.

This thesis explores enhancing track generalization in motorsport driver models through image-based feature sets, drawing inspiration from autonomous driving applications in urban settings. Traditional motorsport models often rely on numeric features, which excel on known tracks but face limita- tions when adapting to new, unseen environments. To address this, I introduce a CNN-based model that integrates bird’s- eye-view images with vehicle states and path-planning data, allowing a more holistic perception of track layouts and sur- roundings. Through open-loop evaluations on unseen tracks, the proposed model demonstrates superior generalization, achieving significantly lower RMSE compared to boundary point-based models, with improvements observed across steering, braking, and acceleration actions. Additionally, I apply novelty detection using Mahalanobis Distance to isolate Out-of-Distribution(OoD) scenarios, providing a precise measure of the generalization gap. This work establishes a baseline for image feature design in motorsport driver modeling, emphasizing the role of spatial and contextual information in achieving adaptable and high- performance autonomous racing agents.

Real-world environments require robots to continuously acquire new skills while retain-ing previously learned abilities, all without the need for clearly defined task boundaries. Storing all past data to prevent forgetting is impractical due to storage and privacy con-cerns. To address this, we propose a method that efficiently restores a robot’s proficiency in previously learned tasks over its lifespan. Using an Episodic Memory M, our approach enables experience replay during training and retrieval during testing for local fine-tuning, allowing rapid adaptation to previously encountered problems without explicit task iden-tifiers. Additionally, we introduce a selective weighting mechanism that emphasizes the most challenging segments of retrieved demonstrations, focusing local adaptation where it is most needed. This framework offers a scalable solution for lifelong learning in dy-namic, task-unaware environments, combining retrieval-based local adaptation with selec-tive weighting to enhance robot performance in open-ended scenarios.

Soft robots offer a variety of useful applications. However, the nature of their design makes them challenging to control using traditional techniques. Many applications therefore rely on machine learning-based methods, learning opaque control policies or dynamic models of the robot. This often results in overly complex and uninterpretable solutions for the problem at hand.
In this paper, a simplified approach is considered: Using a common feedforward neural network as a target adapter module, learned adaptations are added to the desired targets that are fed to a simple PID controller, essentially “deceiving” it into a better performance. This adapter is completely separate from the controller itself, allowing for relatively simple controllers to control and adapt to complex and evolving systems.
The approach is tested in simulation on a simple mass-springdamper control problem, where qualitative results show nearimmediate improvement in controller performance. The approach is then tested on a demonstrator origami robot, which relies for its functionality on the dynamics of the Kresling origami spring. Results show that this simple adaptation approach is robust to changes in controller tuning and changes in system dynamics. However, the method is sensitive to the sampling frequency at which training data is recorded.

Human Theory of Mind (ToM), the ability to infer others’ mental states, is essential for effective social interaction. It allows us to predict behavior and make decisions accordingly.In Human Robot Interaction (HRI), however, this remains a significant challenge, especially in dynamic, real-world scenarios. Enabling robots to possess ToM-like capabilities has the potential to greatly improve their interaction with humans. Recent advancements have introduced Large Language Models (LLMs) as robot controllers, leveraging their strengths in generalization, reasoning, and code comprehension. Some have claimed that LLMs may exhibit emergent ToM capabilities, but these claims have yet to be substantiated with rigorous evidence. This study investigates the ToM-like abilities of Multimodal Large Language Models (MLLMs) by creating a benchmark dataset from humans performing object rearrangement tasks in a simulated environment. The dataset visually captures the participants’ behavior and textually captures their internal monologues. Based on this dataset (text, video, or hybrid) three state-of-the-art models made predictions about the participants’ belief updates. While the results do not conclusively establish ToM capabilities in MLLMs, they offer promising insights into mental model inference and suggest future directions for research in this domain.

A widely held assumption within the field of Trajectory Prediction is the perfect and complete observation of agents’ pasts. While this assumption allows for a simpler representation of the prediction problem, it no longer holds true when prediction models are expected to operate on histories generated by upstream perception systems, which are susceptible to fail. Occlusions are a particularly important cause of perception failures. They fragment tracked agents’ trajectories, and can often hide their most recent position(s) from the perception system. While most prediction models that are currently being researched cannot account for the possible incompleteness of agents’ past histories, we devise a prediction model that is designed to directly operate on partially missing histories caused by occlusions. We present OcclusionFormer, a TransFormer based prediction model, which predicts agent’s futures from their last observed position, without requiring imputation of missing past positions. Experiments show that our design is occlusion capable, as it can predict from trajectories with partially missing data, while remaining performant in the ideal, fully observed scenario. We also conduct research on the integration of an occlusion map within our model, which could help narrow the region of plausible prediction for occluded agents. We observe that, while the addition of such a map does improve the coherence of predictions with respect to the configuration of the occluded space, it results in a degradation of prediction performance.

Robots have the potential to assume tasks across various real-world scenarios. To achieve this, we require adaptable and reactive robots that can robustly deal with products and environments that present variability. For example, in the agro-food sector, each tomato plant inside a greenhouse is unique; hence, different robotic motions are required when interacting with different plants. Unfortunately, due to their simplicity, most robotic solutions currently employed are rigid and rely on hand-crafted rules. Such solutions perform well in controlled and repetitive environments; however, they fall short when these conditions are not met. As a consequence, a large family of problems remains unsolved.....

Physical human-robot cooperation (pHRC) has the potential to combine human and robot strengths in a team that can achieve more than a human and a robot working on the task separately. However, how much of the potential can be realized depends on the quality of cooperation, in which awarenes of the partner’s intention and preferences plays an important role. Preferences tend to be highly personal, and additionally depend on the cooperation partner and the cooperation itself. They can be hard to define in terms a robot would understand, and may change over time. This thesis focuses on learning ‘useful models’ from observed behavior, to let our robot adapt its behavior to better match its human partner’s preferences, and thus improve the cooperation.
The aim is to capture personalized approximate models of human preferences –how a person likes to do something– from very few interactive observations, providing only small amounts of imprecise data, such that the robot can use the model to improve each user’s comfort. First, we learn a model to predict and optimize the human ergonomics in a pHRC task, such that our robot can ropose a plan, for both the human and itself, to solve the task in a way that is more ergonomic for its human partner. However, people do not necessarily prefer to act ergonomically, nor do we want to impose on them what a robot thinks best. Therefore, next, we apply inverse reinforcement learning (IRL), to capture less restrictive preference models: 1) path and velocity preferences for motion planning, and 2) on a higher level of abstraction, which (grasp or motion) action to initiate for proactive physical support. For learning to take the correct action in cooperation, we developed the disagreement-aware variable impedance (DAVI) controller to smoothly transition between providing active guidance and allowing the human to demonstrate alternative behavior.....