Reliable 3D object detection is essential for autonomous driving, yet adverse weather conditions such as rain continue to degrade perception performance. This thesis investigates the robustness of Camera-LiDAR fusion for 3D object detection under rainy conditions within the BEVFusion framework. The study first examines how training-data composition affects performance by comparing clear-only and mixed-weather training. It then evaluates whether explicit robustness modeling can improve detection beyond this baseline through two lightweight extensions: spatial reliability-based feature modulation and short-term temporal consistency. The results show that exposure to rainy scenes during training already provides a substantial robustness gain under rainy evaluation. In contrast, the proposed module-based extensions do not produce a clear and consistent class-wise improvement over this mixed-weather baseline. Additional analyses indicate that the remaining limitation under rain is not mainly explained by a large loss of object presence, but more by changes in detector confidence and ranking behavior. The reliability-map experiments suggest that weakly supervised feature gating is difficult to align with meaningful local sensor degradation, while the temporal-consistency experiments show that changes in internal feature behavior do not necessarily translate into improved final detection performance. The main conclusion is that, within the present BEVFusion setup, the evaluated robustness modules are not sufficient to improve detection accuracy beyond the gains obtained from rainy-weather training exposure. The findings nevertheless provide insight into where the remaining bottlenecks under rain are likely to lie and help clarify which directions may be more promising for future robustness research in Camera-LiDAR 3D detection. ... Read more

Leg movements of dairy cows during automatic milking are relevant for robot safety, animal welfare and health monitoring. Detecting the exact onset of these movements inside an Automatic Milking System is difficult because the legs are often occluded by the cow, fences or the milking arm. This study investigates to what extent safe and dangerous leg movement onsets can be detected and classified from 3D pose sequences reconstructed from multi-view RGB-D data. Three RGB-D cameras were used to record cows during milking. DeepLabCut was used to estimate 2D keypoints of the cow and the milking arm, after which the detections were lifted to 3D using depth data, fused across camera views and optimized using measurement, temporal, and structural constraints. The resulting 3D pose sequence was represented as a graph containing cow joints, AMS points and cow-AMS interaction information. A graph-based model was then trained to detect movement onsets per leg and classify them as safe or dangerous. The graph-based model clearly outperformed a threshold-based baseline, reaching an F1 score of $0.787 \pm 0.012$ on the unseen test set over three random seeds. The ablation study showed that velocity, observed cow leg points and mirror augmentation were most important for performance. The error analysis showed that many remaining errors originated from occlusion, keypoint detection failures and pose inaccuracy, rather than from the movement onset detection model alone. These results show that multi-view RGB-D 3D pose is a promising representation for leg movement detection in an AMS, but further work is needed. ... Read more

Large Language Models (LLMs) promise intuitive robot control through natural language, yet the gap between vague human intent and safe physical execution remains significant. This thesis investigates how the distribution of planning responsibility relates to the reasoning architecture of the model. In a study involving 226 participants who were new users in LLM-based robot prompting, a non-reasoning model (Gemini 2.0 Flash-Lite) and a reasoning model (Gemini 2.5 Pro) were compared on a baseline navigation task, followed by an evaluation of the reasoning model across tasks of increasing logical complexity. Results indicate a clear divergence in safety profiles: non-reasoning models showed more collision-prone goal-seeking behavior, whereas reasoning models demonstrated stricter adherence to the safety constraints in the system prompt, preferring to refuse a request rather than generating an unsafe plan when the task exceeded the model’s capabilities. However, even reasoning models showed declining performance in high-complexity tasks, which hyperparameter tuning (temperature/tokens) did not resolve. Analysis of user interaction reveals that effective prompting is less about linguistic precision and more about "distributed cognition": while models can autonomously plan simple tasks, complex scenarios require the human to reclaim the task planning effort and provide low-level guidance to reduce the solution space. These findings suggest that safe language-driven robotics depends on a dynamic partnership where the distribution of task planning effort shifts based on task difficulty and the capabilities of the specific LLM. ... Read more

A patrol robot drives the same route repeatedly; which of the changes between one visit and the next a supervisor should see depends on the surroundings, earlier moments, and local rules. We present a hybrid geometry-VLM pipeline for evaluating operational surveillance norms. LiDAR change detection and person tracking feed a spatio-temporal evidence graph of changed objects, people, and site areas; the graph indexes what is passed to a vision-language model. The model characterises what it is shown and judges it against the site’s norms, which range from explicit through relational to implicit. Developed at one site and evaluated at another, with reportable and non-reportable situations staged, it reaches an F1 of 0.70 at a precision of 0.66. A camera-only alternative nearly matches its recall but at an alert precision of 0.08; withdrawing the relational evidence costs 0.17 of F1. What a design must supply differs across the three norm categories.
... Read more

Machines that work alongside humans have progressed from mechanical devices to AI-driven robots that can perceive, reason, and act in environments shared with people. Multimodal large language models (MLLMs) now enable robots to interpret open-ended instructions, plan multi-step actions, and generate natural language explanations. However, technical capability alone does not guarantee effective collaboration, because humans and AI-driven robots perceive, reason, and respond through different information-processing mechanisms, and when these processes are misaligned, interaction can fail regardless of the robot’s capability.

This thesis investigated where alignment and misalignment occur between humans and AI-driven robots, and how human-robot interaction should be designed for complementary collaboration. This thesis adopts the vocabulary of Wickens’ Multiple Resource Theory, which distinguishes processing stages (perception, cognition, responding), processing modalities (visual and auditory), and processing codes (spatial, verbal). Four empirical studies targeted different processing stages and progressed in platform complexity and realism.

Across the four studies, a combined 2,292 participants generated evidence that misalignment can occur at every information-processing stage. Three overarching findings are extracted. First, outcome-level results alone are insufficient for detecting misalignment: high correlations in Chapter 2 masked different search mechanisms, and verbal fluency in Chapter 4 masked shallow reasoning. Second, as the AI system becomes more automated, the human role changes from direct operator to supervisor, from commanding every movement (Chapter 3), through evaluating automated perception and strategy (Chapter 4), to diagnosing complete LLM-generated plans (Chapter 5). Third, the quality of collaboration depends on how cognition is distributed, not only on the absolute capability of either partner: voice control freed spatial resources while gestures overloaded them (Chapter 3), and students learned to redistribute cognitive work based on model capability and task difficulty (Chapter 5).

This thesis also has limitations. The studies focused heavily on the visual modality; haptic and auditory modalities were not explicitly investigated. Each study concerned a specific domain and a relatively homogeneous participant population. Whether the documented forms of misalignment transfer to other human-robot-interaction domains, more complex environments, and more diverse populations requires further investigation. Nonetheless, the findings support a set of design principles: use modality switching to offload overloaded cognitive resources, expose AI uncertainty rather than only AI outcomes, provide calibration training before deployment, and design function allocation as dynamic instead of static. ... Read more

Affordances, or action possibilities, have been explored to enable robotic manipulation with everyday objects, however the effect of an agent's embodiment has not received much attention. Here we investigate how embodiment changes affordances between a human and robot. We present a method to automatically generate affordance pseudo-labels from a robotic manipulator for the task of grounding (localising) affordances on an object, as there is no such existing dataset. We then propose a general model for embodiment-conditioned affordance grounding, and explore three ways to condition on the embodiment. Our model learns to perform an affine transformation on image embeddings based on the effect of embodiment on the affordance. We evaluate all three variants of our model and compare them to a variant without embodiment conditioning and a state-of-the-art affordance grounding method. The results show that our best performing model decreases affordance prediction error by 25% when compared to the variant without embodiment conditioning and by 68% when compared to the state-of-the-art method. Through our results we demonstrate that embodiment matters when perceiving affordances. ... Read more

This study investigates how font size, text-drawing style, and drop shadow affect the visibility, legibility, and comprehension of overlaid text. The purpose of this study is to better understand the role of these design factors and to offer practical guidelines for presenting information effectively on see-through displays. Twenty-five participants completed a visual search task, where they were asked to locate a target word presented under varying conditions on a complex background. A full factorial design was employed, incorporating four font sizes (0.10°, 0.15°, 0.20°, and 0.60°), six text-drawing styles (plain green text, plain white text, white text on blue billboards with 30%, 50%, 75% or 100% opacity), and the presence or absence of a drop shadow. Performance was evaluated across conditions in terms of noticeability (as a measure of visibility), processing time (as a measure of legibility), and word identification accuracy to determine significant differences. Applying a drop shadow improved legibility in plain text, while billboards lowered the upper performance threshold for font size. Although billboard conditions outperformed plain text conditions, varying billboard opacity had no significant effect on processing time, word identification, or noticeability. Overall, the findings suggest that font size, text styling, and background complexity interact to influence text visibility and legibility. ... Read more

Current visuomotor manipulators jointly train their perception-planning-action components simultaneously using an end-to-end framework to avoid hand-engineering components. Despite this, methods for human-to-robot object handover tasks require a perception component that segments the hand from the object, which can introduce error propagation. For this reason, this study investigates the applicability of an end-to-end framework that eliminates the need for hand-object segmentation in a simulated human-to-robot object handover task using HandoverSim.

To address this, a behavior cloning agent is used to convert camera input into RGB-D voxel space and output discretized 6-DoF manipulation to directly discover features for the handover task. This study introduces a framework that combines the behavior cloning agent with the HandoverSim, which allows experimenting with various training configurations. These configurations consist of experiments with: 1) expert demonstration data; 2) optimal camera setup; 3) handover objects; and 4) voxel-based RGB augmentation techniques.

The trained model is evaluated on its generalization to diverse handover conditions in the HandoverSim Benchmark. The results demonstrate that the behavior cloning agent can learn features for the handover task without requiring a perception component. The model learns the grasp-object relation whilst minimizing contact with the hand. Despite this, performance is limited by sparse training data and grasping accuracy. ... Read more

This master thesis explores the possibilities of using a quadruped robot for manipulating a milking cup, using only body movments. This shows the capabilities of a quadruped for manipulating an object, without the need of higher degree of freedom manipulator arms. ... Read more

While Vision-Language-Action (VLA) models have shown remarkable success in short-horizon robotic manipulation, their performance degrades significantly over long-horizon tasks due to compounding errors. These failures often stem not from a lack of motor skills, but from the inability of policies to track their own progress and recover from minor mistakes. In this work, we address this challenge by proposing a hierarchical framework that integrates the high-level reasoning of a Large Language Model (LLM) with the low-level visuomotor control of the NVIDIA Isaac GR00T N1.5 VLA. Our system uses the LLM to decompose a long-horizon command into a sequence of verifiable sub-tasks. Critically, we introduce a perception-in-the-loop verification module that confirms, through the use of a VLM (GPT-4o) the success of each sub-task before proceeding to the next, preventing error cascade. Through 30 physical trials per condition on a UR5e robot, we demonstrate that our framework increases the long-horizon task success rate from 43.3% to 63.3% compared to GR00T's baseline end-to-end long horizon task. Our results show that by externalizing semantic state tracking and enabling error awareness, we can significantly enhance the reliability of powerful VLAs for complex, multi-step tasks. ... Read more

With mobile robotics being applied for more and more complex applications, their autonomy should be preserved. While a lot of research is performed into the direction of failure prediction for autonomous processes or systems, the field of mobile robots has received less attention. Proactive failure prediction for mobile robots is a useful tool to prevent unwanted downtime and undesired damages. This work attempts to fill this research gap by showing the applicability of anomaly detection methods for failure prediction in the field of mobile robots. Specifically, we employ an unsupervised Variational Autoencoder to predict failures in the operational data from the Discovery Collector, a manure cleaning robot developed by Lely Industries. We elaborately showcase the feature engineering steps which yield the best performance, provide the performance of three general datasets, and state promising next steps for root cause classification which is enabled by accurate failure prediction. All in all, our work shows that the use of feature offsets, calculated from desired values compared to actual values, enhances the model performance tremendously. The provided datasets showcase F1-scores ranging from 0.64-0.76, showing the proposed solution is able to solve the failure prediction problem in the field of mobile robots, while highlighting the encountered limitations for future improvement.
... Read more

Large Language Models (LLMs) possess significant semantic knowledge about the world, making them valuable for high level control for robots through Vision-Language-Action (VLA) models. These models integrate an LLM to deduce semantic knowledge from a robot’s vision and natural language inputs, facilitating real-world actions. Despite their potential, VLA models are a relatively new research area, with applications mostly limited to simulations or household tasks and insufficient validation in broader contexts. This study aims to develop a LLM-Enhanced Robotic Affordance and Control System (LERACS) for robotic manipulation in applied cases such as the management and maintenance of electrical grid infrastructure. LERACS is designed to visually ground manipulable objects and decompose tasks based on user instructions within a human-robot interaction chat interface, using ChatGPT. A system validation and an AI user experiment were conducted to evaluate its effectiveness in interpreting and performing actions based on pre-made and synthetically generated user instructions. These assessed LERACS’ performance across various settings and instructions. Results indicate high success rates in environmental interpretation and task execution, with robust labeling accuracy, especially in complex settings. Feedback from the AI user experiment highlighted LERACS’ adaptability, identified areas for improvement, and demonstrated its practical utility across diverse settings and task complexities. ... Read more

Despite significant advancements in computer vision models, their ability to generalize to novel object-attribute combinations remains limited. In Compositional Zero-Shot Learning (CZSL), the goal is to recognize all possible attribute-object combinations while training on only a subset of these compositions. We demonstrate that Vision-Language Models (VLMs) are suitable for CZSL due to their ability to understand both text and image modalities, and we show the importance of cross-modality fusion. Existing methods for CZSL mainly focus on image classification. In this thesis, we enhance CZSL in object detection without forgetting prior learned knowledge. We use Grounding DINO, a VLM for object detection, and create auxiliary tokens for the words present in the training dataset, training only these to mitigate forgetting. This technique, referred to as Compositional Soft Prompting (CSP), was initially used for image classification, and we incorporate it into Grounding DINO to leverage it for object detection. We show that for CZSL, it is beneficial to anticipate the orientation of embeddings in the embedding space, promoting independence between attributes and objects. Additionally, by anticipating possible compositions that could exist based on the attributes and objects present in the training data, we assign soft labels for partial correctness using a method we call Compositional Smoothing. This approach guides the model to learn what the compositions are composed of rather than learning the compositions themselves. We refer to the combined approach of these methods as Compositional Anticipation. Our approach achieves a 70.5% improvement over CSP on the harmonic mean (HM) between seen and unseen compositions on the CLEVR dataset. Furthermore, we introduce Contrastive Prompt Tuning to incrementally address model confusion between similar compositions. We demonstrate the effectiveness of this method, achieving an increase of 14.5% in HM across the pretrain, increment, and unseen sets. Collectively, these methods provide a framework for learning various compositions with limited data, as well as improving the performance of underperforming compositions when additional data becomes available. ... Read more

With the advancement of Artificial Intelligence leading to increasingly human-like outputs, assessing a machine’s ability to exhibit human-like intelligence has become more essential than ever. This study aims to investigate how human-like chess players perceive four conditions: one human opponent and three different types of algorithms. One of these algorithms, Maia, has been trained on human data and aims to play the most human-like move. In a custom-designed experiment similar to a Turing test, chess players faced off against Maia, Stockfish and a human without knowing their opponent’s nature. After each game, the chess player assessed how human-like the moves of the opponent were and estimated whether they played against an engine or a human opponent. During the game, participants were asked to think aloud about their next move and react towards the moves of the opponent. Additionally, the gaze of the player was captured with the SR EyeLink Portable Duo at 1000Hz, with the goal of finding differences within the player’s gaze while participants tried to discover the nature of their opponent. Results from the experiment revealed that, based on responses to a subjective questionnaire, the perceived humanness of Maia is statistically similar to a human and different from the other two chess engines. From the analysis of the voice recordings, categories of sentences were identified that could suggest recognition of the opponent, specifically: "expected", "unexpected", "human-like" and "engine-like". From the eye-tracking results, the average fixation duration and pupil diameter changes following the opponent’s move were compared for each condition, but showed no statistical differences between conditions. In summary, Maia was perceived more human-like compared with other chess engines. However, differences in underlying cognitive processes on how the human perceived this difference in a Turing Test experiment were not identified. ... Read more

The increasing presence of robots calls for a more seamless and information-rich communication method between humans and robots. This paper explores how natural user interface (NUI) modalities, particularly speech and gesture controls, can be used through augmented reality (AR) to operate robots. The increasing presence of robots calls for proper evaluation methods of how to use AR for operating mobile robots. The study uses the Microsoft HoloLens and the robot, named Spot, from Boston Dynamics as primary technologies. The research consists of a user study consisting of 218 participants, one of the largest participant pools for this field to date. The experiment consists of walking the robot over a trajectory with discrete steps, with the perspective of following the robot or standing on a predetermined stationary point. To support the control of the robot, visual information and feedback are included in the HoloLens. Speech control showed the best time performance of the experiment, regardless of the perspective condition. Conversely, errors made during the trials were the majority for the speech condition, due to the waiting time of the speech recognition that caused participants to repeat the commands. The walking condition gave participants the impression that control commands were more intuitively mapped to the robot's motion. Overall, the participants preferred the speech control method while walking with the robot, and the least preferred method was using gestures in a stationary perspective. Even though the speech was the preferred control method and perspective-taking was preferred by participants, this was only for the experiment and task discussed in this paper. Both control methods have different characteristics that make them favorable to be used for specific tasks. Speech and gestures can be used for different tasks when operating a robot with Augmented Reality glasses; preference will depend on the task at hand and the control method design. ... Read more

This study investigates the influence of synthetic data on the accuracy of 6D pose estimation in RGB images compared to RGB-Depth image-based methods. Additionally, it aims to examine how this performance varies across different types of small chess pieces during a picking task with a robotic arm. The methodology involves 3D scanning the chess pieces and generating a dataset of 61,910 synthetic images with diverse domain randomization. Using this synthetic dataset, six unique 6D pose estimation models were trained for each type of chess piece. The models were validated using a real-world 6D pose estimation dataset, and the obtained results were compared with those from the estimations on a synthetic dataset. It was observed that synthetic data can be used for bridging the visual simulation to reality gap. However there is superior performance on synthetic data compared to real-world data. This implies that results obtained in a synthetic environment cannot be directly projected to real-world scenarios. Also a noticeable decrease in accuracy in both object detection and 6D pose estimation was observed with respect to camera distance for real-world images, primarily linked to the reduced size of chess pieces in the images. Notably, each chess piece exhibited improvement after depth refinement, with accuracy closely linked to the performance of the depth camera. A picking experiment was also conducted, revealing that models relying solely on RGB data achieved a positive grasping rate of approximately 52%, while RGB-Depth-based methods reached around 70%. The findings underscore the potential for successfully picking chess pieces with and without depth refinement, emphasizing the feasibility of bridging the visual simulation to reality gap. Additionally, the study suggests several avenues for future research, including comparing synthetic data with real-world data, further exploring the training process, and introducing domain randomization in the picking experiment, such as background changes or different distractors. Furthermore, it suggests investigating diverse approaches to improve depth accuracy. ... Read more

As the global population continues to increase, so does its dependance on food, which has led to the development of more efficient food production methods. Greenhouse horticulture and genetically modified seeds have increased crop yield and decreased water and energy consumption. However, delicate greenhouse-grown vegetables continue to necessitate manual labour. The purpose of this thesis is to design and evaluate a vision system capable of detecting the spatial structure of greenhouse vegetation in order to automate plant repositioning, pruning, and leaf removal. The system employs a robotic arm and a single camera to generate custom plant features, match them, and generate a 3D model of the plant’s structure. The thesis discusses the system’s requirements, design, and evaluation, including the segmentation of plant outlines using deep learning networks, the construction of a 2D plant structure using constrained Delaunay triangulation, and triangulation-based depth calculation. Analysis of three-dimensional precision reveals a correlation between camera positional errors and the algorithm’s depth estimation. The findings support the viability of using a vision system to automate greenhouse duties if certain modifications are made. Contributing to ongoing efforts to increase greenhouse agriculture’s efficiency and reduce manual labour. ... Read more

In the engineering industry, all structural parts have to be designed as efficient and lightweight as possible. Traditionally, the design process has been carried out through manual design iterations, which can be time-consuming and require significant engineering expertise. Over the last decades however, several computational design techniques like Topology Optimization and Generative Design have been developed to support engineers in the structural part design process. Even though these techniques can have a positive influence on the design process, they both also have their downsides. Topology Optimization only gives a single result that is often a local optimum, influenced by boundary conditions and numerical settings. Commercial Generative Design tools explore multiple design options in a single run using AI algorithms, but need cloud-based systems to carry out their demanding simulations which still take several hours per run. It is however expected that a combination of the two, a Topology Optimization based Generative Design approach in the form of an auxiliary tool, has potential to improve the early stages of the design process even more. With such a design approach, multiple design solutions are explored quickly to study the effect of boundary conditions or numerical settings. This can help designers by giving direction and insight in trade-offs between multiple objectives, early on in the design process when design decisions still have the highest impact.

The goal for this research was therefore to research the effect of such a Topology Optimization based Generative Design approach on the design performance and experience. In order to do so, a robust and user-friendly TOP-GD tool was created. In this tool, multiple design solutions are explored quickly by implementing a batch-run setup that varies several chosen parameters, without needing to manually run several optimizations consecutively. Calculations are done with a simple TO script using coarse geometries, and without taking into account manufacturing methods yet. This asks for less demanding, detailed and complicated calculations than AI-based Generative Design tools currently offer, while at the same time moving from a single TO result to generating a range of candidate solutions. A lot of effort was put in the user-friendliness of the TOP-GD tool, enabling an easy workflow for the setup of design problems and a clear presentation of the results by means of a simple GUI.

The use of the TOP-GD tool in the design process was evaluated in an experiment, where it was compared with a more simple TO tool and a basic manual design approach using just pen and paper. This was done by giving the participants of the experiments three simple design assignments, that they had to carry out using each of the design approaches one by one. Evaluation of the approaches was done by comparing the design performance, and assessing the design experience with a survey and using Eye-tracking techniques.

The results of this experiment did not show enough evidence to conclude that the different design approaches had an effect on the design performance for the simple assignments executed during the experiment. However, the results of the survey show a clear positive impact of both the TO tools on the design experience, compared to manually designing. Furthermore, the TOP-GD tool has the largest positive impact on the design experience and its use in the design process is considered a big improvement, especially in quickly exploring new design directions and creating overview. This confirms the expectation that a Topology Optimization based Generative Design approach has a positive effect on the early stages of the design process. The differences found with Eye-tracking between the TO tools support this, although a more extensive experiment should be done to convincingly confirm this conclusion. ... Read more