Autonomous exploration by drones in unknown environments has traditionally focused on maximizing spatial coverage without semantic understanding. This thesis presents a framework that integrates vision-language models (VLMs) with adaptive path planning to enable anomaly-aware exploration and inspection. The system employs a three-phase approach: frontier-based exploration, continuous VLM-based anomaly detection, and inspection of detected anomalies. Comparative experiments demonstrated that YOLO+CLIP with negative embeddings achieved the highest F1 score of 0.7218 on the SegmentifyMeIfYouCan benchmark. Experiments showed that dedicated inspection yielded improvements over solely exploration observations. However, system-level evaluation across nine experimental runs revealed that the inspection phase took up most of the mission time (85.9% average), with varying anomaly detection consistency across anomaly instances. False positive analysis identified VLM error as the primary limitation (52% of false positives), followed by simulation artifacts (37%) and semantic ambiguity (11%). The framework successfully demonstrated the feasibility of coupling VLM-based anomaly detection with adaptive planning, though precision limitations and large inspection inefficiencies show opportunities for future work. ... Read more

Autonomous motion planning requires the ability to safely reason about learned trajectory predictors, particularly in settings where an agent can influence other agents' behavior. These learned predictors are essential for anticipating the future states of uncontrollable agents, whose decision-making process can be difficult to model analytically. Thus, uncertainty quantification of these predictors is crucial for ensuring safe planning and control. In this work, we introduce a framework for interactive motion planning in unknown dynamic environments with probabilistic safety assurances. We adapt a model predictive controller (MPC) to distribution shifts in learned trajectory predictors when other agents react to the ego agent's plan. Our approach leverages tools from conformal prediction (CP) to detect when the other agent's behavior deviates from the training distribution and employs robust CP to quantify the uncertainty in trajectory predictions during these agent interactions. We propose a method for estimating interaction-induced distribution shifts during runtime and the Huber quantile for enhanced outlier detection. Using a KL divergence ambiguity set that upper bounds the distribution shift, our method constructs prediction regions with probabilistic assurances in the presence of distribution shifts caused by interactions with the ego agent. We evaluate our framework in interactive scenarios involving navigation around autonomous vehicles in the BITS simulator, demonstrating enhanced safety and reduced conservatism. ... Read more

The accurate prediction of object softness is crucial in many fields, from agriculture to medical care. Vision-based tactile sensors, which capture high-resolution images of contact interactions, have shown great potential in determining this material property. Many existing approaches, particularly those using end-to-end models, suffer from a 'black-box' problem where it is difficult to understand which features the models use to make their predictions. This lack of transparency makes it challenging to determine and correct errors. To overcome this, this paper shows a data-driven method that can decode information from acquired tactile images to extract pressure distributions and assign softness levels to different objects. A novel approach is explored, integrating a Convolutional Neural Network (CNN) to predict the pressure distribution and a Long Short-Term Memory (LSTM) network to assess material softness. It is demonstrated that the CNN model effectively learns necessary features from the tactile images, enabling precise pressure distribution predictions. Concurrently, the LSTM model analyzes temporal sequences of tactile data, accurately predicting material softness and differentiating ripe from overripe fruits. By utilizing the spatiotemporal pressure distribution, this method improves on existing methods by enabling the efficient use of tactile data and providing additional information that can be used to further enhance the model. This paper can be used as a stepping stone to a more complex system in which robotic control can be implemented based on the sensed material properties, allowing for better control loop mechanisms and expanding the applications of tactile sensing technologies. ... Read more

This paper presents a novel Graph Optimal Transport (Graph OT) framework for analyzing and aligning plant structures across different growth stages and transformations. Our method extends existing graph matching techniques by incorporating domain-specific botanical features and employing a multi-scale matching strategy that captures both local and global structural characteristics. The framework combines multiple feature representations, including node descriptors, spectral embeddings, Node2Vec embeddings, and relative positions, to construct an augmented cost matrix for optimal transport based matching. We evaluated our approach on a dataset of 50 distinct plant structures under various transformations, including rotation, deformation, and partial matching scenarios.

The results indicate that our Graph OT framework significantly outperforms traditional optimal transport (OT) methods, achieving node-matching accuracy scores of 0.75 for rotated,
0.74 for deformed structures, 0.67 for cut structures, and 0.71 for structures with skipped nodes. Our approach demonstrates particular robustness in handling complex transformations. This method provides a powerful tool for botany applications such as crop management, growth modeling, and automated pruning systems. ... Read more

Legged animals possess extraordinary agility with which they can gracefully traverse a wide range of environments, from running through grasslands to jumping across cliffs and climbing nearly vertical walls. Inspired by this, in this work, we use Deep Reinforcement Learning to give legged robots the ability to perform a diverse set of highly explosive and agile jumps. Unlike other works, our approach is not constrained to imitating a reference trajectory. We instead use curriculum-based learning to progressively learn more challenging tasks, starting from a vertical high jump and then generalising to forward and diagonal jumps. In the final curriculum stage, the robot learns to leap over barrier-like obstacles or to land on them, conditioned on the desired jumping distance and the object's dimensions. We show that such an approach can produce a wide range of robust and precise motions, which we thoroughly and successfully validated in several indoor and outdoor real-world experiments on the Unitree Go1 robot. In our real-world experiments, we show a forward jump of 90cm, exceeding previous records for similar robots reported in the literature. Additionally, we investigate the effects of incorporating bio-inspired parallel elastic actuators to improve the jumping performance further. This resulted in smoother motions, much softer landings with lower joint velocities and larger jumps. Finally, we present and analyse the limitations of our method and introduce exciting directions for future work to address them.
... Read more

This thesis focuses on developing a model that effectively captures and generalizes the four quadrant behaviour of propellers, which is crucial for understanding and optimizing propulsion systems in marine vessels. Accurate prediction of four quadrant behaviour offers significant benefits to the industry, including reducing fuel consumption, mapping hull growth accurately, and identifying efficiency losses due to propeller cavitation. Unlike existing approaches that mainly concentrate on modelling the physical interactions between the water and the propeller blade, this thesis investigates different machine learning methods and finds that the ensemble method yields the best model for predicting four-quadrant behaviour. The method is tested on different the different data sets available. The data set whose propeller type is the same as the propellers used in the fleet is used to generate predictions for propellers currently in use in the fleet. ... Read more

While robots execute many tasks where physical interaction with the environment is required, it is still challenging to control a robot that deliberately makes contact at a non-zero velocity, especially with multiple contact points that are impacted simultaneously.
When there is a mismatch between planned and actual impact time, the robot typically does not respond as desired.
In this paper, we demonstrate that an Impact-Aware Learning from Demonstration (IA-LfD) framework, that is based on Reference Spreading, can be developed and validated by physical experiments on real robots.
The proposed IA-LfD framework is based on the following key aspects:
(a) generating suitable ante-impact and post-impact tracking references from demonstrations;
(b) development and validation of an impact detection mechanism to identify the contact transition, typically consisting of multiple impacts.
The validation of the approach shows in particular the advantage of using an intermediate phase controller in reducing peak contact forces and oscillations during the dynamic contact transition, when compared to baseline approaches not using this controller.
In addition, the validation highlights the role played by active/physical contact damping during the contact transition to improve execution performance. ... Read more

Grocery e-commerce has been rapidly increasing in recent years, posing a new challenge for retailers as groceries, unlike other goods, have a limited shelf life. Thus, customers expect their orders to arrive quickly and undamaged. Currently, most processes between a customer placing an order and the delivery are performed manually in a warehouse making it labor-intensive and time consuming. The use of autonomous robots in these environments can help improve operational efficiency and productivity while at the same time reducing labor costs and accidents.

One specific process important for maintaining the desired quality of the customer's ordered items and fast delivery is item packing. For an autonomous robot to pack items independently, it must learn and predict possible placement positions that are both geometrically feasible and semantically plausible so that items maintain their desired quality. Retail-related environments are heterogeneous environments containing many different items. Thus, instead of the robot knowing what the item is, it is more beneficial to know how an item can interact with a scene. This entails that robots must learn interactions between items and the scene, which are called affordances, to place items in plausible positions. These so-called object-object affordances describe how an object can interact with another object.

In this thesis, object-object affordances are learned to predict where to place items inside a box in an item packing task. With the use of a simulator called SAPIEN, item packing is simulated, and large-scale interaction data is generated. A model is then trained to predict a placement position by giving as input a complete pointcloud of the item to be placed and a partial pointcloud of the scene. An item packing pipeline is then built that can pack items using the trained model. Several item packing experiments are performed to pack single items and a list of items. The results show that the model successfully learns the semantic relationships between objects resulting in packing items in stable and plausible positions. Several other experiments are performed to evaluate whether the model is generalizable to novel items and real pointcloud data. Results show that the model successfully predicts where novel items should be placed. The model can also adapt to real pointcloud data of a box and predict where items should be placed in a real box containing real items. ... Read more

Active inference is a novel brain theory based on the free energy principle, stating that every organism, in order to stay alive, minimizes a certain free energy. This theory is being translated into robot control, hoping to mimic the capabilities of the brain. Research in this field of robotics is still quite young, and active inference has yet to mature into a reliable control theory. Implementations of active inference are scarce, and hierarchical implementations, containing multiple layers of active inference, are almost non-existent. This work presents the first implementation of hierarchical active inference performing online control on a robot in continuous state-space. The robot in question is a skid-steering mobile robot, on which two different active inference controllers are implemented and compared. The first controller is a non-hierarchical active inference controller giving wheel speed commands to the default PID controller of the robot manufacturer based on desired body velocities. The second controller is a novel implementation of a hierarchical active inference controller, that besides the function of the first controller, also replaces the low-level PID control with active inference, resulting in two-layers of active inference ultimately taking desired body velocities as inputs and translating it into open loop voltage commands for the motors. Both controllers are tuned to give roughly the same step responses in order to get a clearer view on the differences between non-hierarchical and hierarchical active inference. During the tuning process of the hierarchical controller, it became clear that no amount of tuning could get the (causal) states to convergence sufficiently fast. This problem was identified to be caused by the message passing in between the layers of active inference, where the prediction error from below negatively impacts convergence. This impact scales together with the tuning parameters, and therefore is not easily fixed by tuning alone. A potential solution is given in the form a an extra parameter in front of the upwards prediction error, giving the ability the independently tune the impact of this error on the behaviour of the system. This solution is used to fully tune the hierarchical controller, after which the performance is compared to the non-hierarchical controller, using a series of experiments. A total of 8 experiments are done where both controllers are tested performing two driving actions in two situations, namely cornering and pivoting on the ground and with the wheels suspended. The non-hierachical controller, as a baseline, performed satisfactory, but the hierarchical controller does not run smoothly on the ground, potentially struggling with the stick-slip of the wheel-ground interactions. From this it seems that using active inference for the low-level, as compared to something like PID, is less suitable. However, that conclusion is too premature for this work as the available encoder feedback frequency for the active inference is way lower than what the MCU-level PID has access to, making the comparison less fair. The main contribution of this work lies in the insight in the implementation structure of hierachical active inference for continuous state-space control, which can be further built upon, using the lessons learned during implementation. ... Read more

The demand for faster production times and higher precisions in the industrial automation is ever-increasing. Resonance modes caused by flexural elements in these machines are limiting the maximum bandwidth. Because of this, high-precision motion systems in industrial machines are limited in the maximum operating speed and precision. To improve the performance of these machines, an active vibration control (AVC) system is needed. In present scientific literature, all AVC systems consist of an under-actuated or perfect-actuated setup. However, the damping performance of these systems could be increased by implementing an over-actuated setup. In an over-actuated setup, multiple actuators are used to control one resonance mode. In this way, control inputs for suppressing modes are provided at more efficient locations, which increases the amount of damping. In this thesis, an over-actuation and over-sensing strategy for active damping is proposed. In this new method, a large number of piezoelectric sensors and actuators are used to control the first four vibration modes of a cantilever beam. The damping performance is evaluated in an experimental setup. Finally, the performance of the new topology of sensors and actuators is compared to the state-of-the-art active damping method that uses a perfect-actuation strategy. The improvement of the new method compared to the state-of-the-art method is shown both in time and frequency domain. ... Read more

Human-robot interaction is a growing field that aims to research and develop communication channels between humans and robots to enhance comfort, safety, and productivity in healthcare, the household, and the industry. Researchers have considered ergonomy-related metrics to compose these channels for physical human-robot collaborative scenarios. We refer to these communication channels as arbitration methods. Several of these metrics, such as human arm manipulability and muscle fatigue, have taken their turns in the literature to set the base for arbitration methods reaching promising results. Human arm force manipulability represents the transmission between joint torques in the joint space and end-point force in the task space depending on the configuration of the joint angles. Muscle fatigue keeps track of the muscle activation and builds up depending on the muscle activation level and previous fatigue value. The first one has predictive value. The other has a reactive value.\par
Nevertheless, no work in the literature explores the power of combining both metrics into an arbitration method. Here we develop a multi-metric arbitration method that combines human arm force manipulability and muscle fatigue as input for a finite state machine (FSM) that translates the human multi-metric state to robot control level over a collaborative task. Although some modifications may be worth trying and evaluating to reach generalizability in physical human-robot collaborative tasks, the system reached satisfactory results. Moreover, as future steps, we should conduct human-factors research to compare the effect of the system on task performance. ... Read more

Deformable objects manipulation (DOM) is largely considered an open problem in robotics. The complexity stems from the high degrees of freedom and nonlinear nature of the object configurations. In this thesis, we consider placing and flattening tasks for cloth-like objects. We propose a practical framework to place a cloth on a surface based on visual perception and human demonstrations. We present a novel feature, Wrinkle cOntRaction Direction (WORD), which extracts a stretching direction to flatten clothes from image and depth data. Furthermore, we integrate WORD and demonstrations into Gaussian Processes to learn a cloth placing policy. Simulation and robot experiment results are used to validate the performance of WORD and the proposed learning framework in this study. The results show that WORD efficiently captures wrinkles on the contact part of the cloth in the simulation as well as the real robot experiment. Besides, the proposed learning framework performs successful results in cloth placing and flattening.A video of the experiments and execution of the tasks is available at https://youtu.be/iV2mAPqL7mA. ... Read more

Teleoperated semi-autonomous care robots aim to alleviate work pressure from care workers. Unlike many traditional stiff position-controlled robots, the care robot is operating in a shared environment with humans that is often unpredictable and unknown. Especially when dealing with tasks that involve contact with the environment, modulation of compliance is a key component to successfully execute autonomous manipulation tasks. Using Learning from Demonstrating (LfD) techniques, the teleoperator must have a system that allows intuitive demonstration of compliance to the robot. Current state-of-the-art systems are either not teleoperated, only allow limited modulation of the stiffness matrix or, are too complex and cumbersome for practical applications. This research tries to overcome these limitations and proposes a teleoperated stiffness commanding method that allows complete modulation of stiffness matrix in 3 Degrees of Freedom (DoF). The system uses the same haptic device (Geomagic Touch) as used for controlling robot manipulator, hence does not require specialized equipment. Through wiggling the stylus of the haptic device, stiffness is commanded to the robot and directly fed back to the operator through haptic and visual feedback. The system is illustrated in a simulated task where a task appropriate stiffness profile is demonstrated along a kinematic trajectory. Additionally, the performance and acceptance of the system is evaluated through a simulated user study. It shows how varying the commanded DoF, orientation, and size of the stiffness commands significantly influences the performance through the eigenvectors and eigenvalues of the stiffness matrix. ... Read more