Contact-rich robotic teleoperation requires continuous impedance regulation for safe physical interaction, but manually configuring multi-dimensional stiffness parameters places a substantial burden on operators. To address this, this paper
presents a zero-shot, context-aware hierarchical LLM-VLM reasoning framework that estimates task-dependent Cartesian stiffness parameters directly from natural-language instructions and visual scene context. The system integrates a Vision-Language Model (VLM) and a dual Large Language Model (LLM) pipeline
to generate six-degree-of-freedom stiffness profiles, alongside a human-in-the-loop web interface for safe oversight. Evaluation through a controlled user study (N = 30) demonstrated that AI assistance reduced median task completion time (15 s vs. 32 s) and parameter adjustments (3 vs. 10), while decreasing cognitive workload (p < 0.001).A real-world deployment on a PAL TIAGo Pro mobile manipulator validated the framework’s ability to modulate stiffness proportionally to physical demands across four cleaning tasks Scrubbing Dried Coffee (6.64 N) > Erasing Pencil Marks (5.54 N) > Wiping Spilled Water (2.47
N) > Sweeping Loose Dust (1.97 N) without requiring task specific training data. These findings demonstrate that semantic reasoning can effectively bridge high-level intent and low-level robot compliance. ... Read more

Contact-rich teleoperation must balance accurate execution of operator commands with the interaction forces generated at contact. Fixed-stiffness impedance control imposes one compromise across changing contact conditions, while existing variable-stiffness approaches often trade off operator burden, model dependence, and the authority assigned to learned components. This work presents ViSTAR (\textbf{Vi}sion-guided \textbf{St}iffness \textbf{A}daptation \textbf{T}hrough \textbf{R}esidual Learning), a shared-control framework in which the operator commands task-space motion and an image-only policy predicts a bounded residual around a nominal translational stiffness. The residual is trained from offline contact-supervised labels and executed through a torque-level operational-space impedance controller. ViSTAR is evaluated in MuJoCo with a haptic teleoperation interface and a simulated Franka Panda on three insertion geometries. In controlled human-proxy paired screening, ViSTAR increased aggregate success from 109/120 to 116/120. In live operator trials from one operator, ViSTAR increased success from 28/30 to 30/30, increased successful low-force outcomes from 0/30 to 6/30, reduced paired force by a median 108.72 N, and shortened completion time by a median 6.82 s. The results suggest that visually conditioned residual stiffness adaptation can improve the success-force trade-off in simulated contact-rich teleoperation, with benefits that remain task- and geometry-dependent. ... Read more

Autonomous extraterrestrial modular construction using mobile manipulators requires solving complex Task and Motion Planning (TAMP) problems under strict structural stability and geometric constraints. The solution space of this problem scales super-exponentially with the number of building blocks, leading to state-space explosion causing classical task planners to be intractable and finding optimal solutions to be near impossible. It is made worse by the fact that the robot can manipulate each block from multiple positions, causing the solution space to be even larger. To address this gap a domain-dependent geometrically aware TAMP framework with stability-guidance and anticipatory heuristic is proposed, tailored for the sequential assembly of multi-surface polyhedral blocks in a monotone Pick-Place-Move (PPM) domain such that it is scalable within the domain and can find high-quality solutions with low navigation costs.

Three main contributions are presented: (1) a stability-guided construction task planner composed of a reasoning framework and an extended support relation graph for stability verification; (2) an Action Cost Propagation (ACP) heuristic that anticipates future navigation obstructions caused by present actions, and propagates their costs to affected actions, optimizing the picking order to minimize overall robot navigation effort; and (3) a complete PPM planner framework with a Gazebo simulation that provides descriptions of polygons via surface-normal transforms and robot base positions.

The stability-guided planner is validated on simple structures with multiple supports, and the ACP heuristic is validated in a full TAMP framework with an active navigation-manipulation loop, and is simulated using a KUKA omniRob mobile manipulator in Gazebo and RViz from the third contribution. The results demonstrate that the integrated framework successfully circumvents combinatorial explosion, reliably filters out structurally unstable states, and generates collision-free, energy-efficient trajectories scalable for autonomous construction using modular building blocks. ... Read more

Coordinated multi-robot networks are essential for planetary exploration. However, achieving fully autonomous operation in extreme extraterrestrial environments remains challenging due to stringent safety, stability, and reliability requirements. This paper presents a semi-autonomous supervisory teleoperation framework that combines high-level human oversight with decentralized multi-agent coordination. A remote operator specifies mission objectives and intermediate via-regions, while the robotic network autonomously performs real-time, collision-free navigation. To address GNSS-denied conditions, Ultra-Wideband (UWB) technology is employed for relative localization, requiring structured formation maintenance for hardware calibration. The proposed control architecture integrates Dynamic Movement Primitives (DMPs) for trajectory tracking with Affine Formation Control (AFC) for formation preservation, supplemented by adaptive collision avoidance mechanisms, and adaptive temporal scaling. Experimental validation demonstrates sub-meter tracking and formation accuracy, while extensive Monte Carlo simulations confirm robust destination reachability, formation maintenance, and network connectivity in cluttered environments. ... Read more

Musculoskeletal injuries are among the leading causes of pain, disability, and loss of independence worldwide. They affect millions of people, with prevalence rising steeply with age. One of the most common musculoskeletal injuries is tears to the shoulder rotator cuff. As these muscle-tendon tissues are anatomically constricted in a very narrow space between the shoulder bones, they are frequently subject to trauma or wear. Treatment of these injuries is both medically and socially pressing: they impair daily activities, limit the ability to work and engage in sports, and generate high personal and healthcare costs. Rehabilitation is essential to recovery, but it is often lengthy and labor-intensive for both physiotherapists (PTs) and patients. Moreover, it is prone to setbacks such as re-injury, since PTs lack quantitative tools to monitor the evolution of complex musculoskeletal structures during therapy.
In this context, the adoption of robotic devices offers opportunities to support manual manipulation of patients and provide sophisticated sensors to monitor them. Yet, despite advances in robot design and control, current systems remain unaware of the patient’s underlying biomechanics, and therefore cannot monitor or prevent harmful loading of healing tissues.
This thesis addresses such critical lack of knowledge by embedding state-of-the-art musculoskeletal models into the control of rehabilitation robots. Through the development of novel algorithms, it enables real-time estimation of deep muscle activity and tendon strain in the shoulder during physical human-robot interaction. By spanning from improved biomechanical simulations to their integration in robotic therapy execution, this work significantly advances the current state of the art to form a cohesive framework for biomechanics-aware robotic physiotherapy. ... Read more

With a growing elderly population, shoulder injuries are becoming more common, and part of the recovery plan is to go to physiotherapy. However, to alleviate the demand for physiotherapists, robots could help with shoulder rehabilitation. To do this safely and enjoyably, the robot will need to prevent re-injury caused by fatigue while keeping the patient interested and motivated to continue with their therapy. In this study, a method for managing fatigue of the two most commonly injured shoulder muscles, the supraspinatus and infraspinatus, in a game is proposed and tested. To validate the developed method, a human factors experiment was conducted. The fatigue-adaptive game was compared to a baseline in which participants controlled fatigue themselves. The participants played three cases for each version of the game. To minimize the risk of over-fatiguing during physiotherapy and not crossing the line of being too fatigued. Therefore, we measured the overshoot of fatigue in both versions of the game. The mean of the overshoot is compared with a Welch's t-test with Bonferroni correction for each fatigue case. The results show a significant difference for some of the fatigue cases, where the controller is either significantly better or there is no significant difference in the overshoot. The fatigue-adaptive game shows consistency across the cases, whereas the baseline does not. Therefore, the fatigue-adaptive game can compete with a person in managing fatigue, while being easier to learn and automatically identifying and removing risky shoulder positions where fatigue changes rapidly. The fatigue-adaptive game can also be played with an industrial robot arm and still demonstrates the capability to manage the fatigue of the two most commonly injured muscles. ... Read more

Wire Arc Additive Manufacturing (WAAM) is a promising technique for large-scale metal 3D printing, offering cost-effective and material-efficient alternatives to traditional methods. A core challenge in WAAM lies in trajectory planning—specifically, determining optimal tool orientations that maintain print quality while ensuring smooth and feasible robot motion. Due to the rotational symmetry of welding nozzles, 6-axis robotic arms exhibit yaw redundancy, creating an unbounded inverse kinematics (IK) solution space. This significantly increases computational complexity when planning across millions of print targets. This thesis presents a novel trajectory optimization framework that addresses this challenge by combining Ant Colony Optimization (ACO) with geometric pruning and multiobjective cost functions. The framework systematically reduces the tool yaw search space using reachability analysis and collision detection with external axes. ACO then identifies joint configurations that optimize motion smoothness, yaw consistency, movability, and safety. To further improve efficiency, we introduce subsampling strategies that exploit the continuity of WAAM layer structures, allowing representative yaw configurations to be computed from a reduced set of targets. The system is evaluated on geometrically diverse parts—including the Stanford Bunny and Hilbert Wall—to demonstrate generalizability. Results show improved trajectory smoothness and path feasibility compared to MX3D’s baseline Dijkstra-based weld optimizer with fixed yaw. This work advances the state of WAAM motion planning and provides a scalable foundation for adaptive tool path generation in robotic additive manufacturing. ... Read more

This thesis explores how deliberate modifications to reward function design in the reinforcement learning can induce skill mutations in robotic reinforcement learning, specifically within a precision pouring task. Using a simulated Franka Emika Panda robot in NVIDIA Isaac Lab, we evaluate 25 distinct reward configurations composed of weighted terms for effort, accuracy, and velocity. The resulting policies exhibit a wide range of behaviors—from fast and efficient pours to novel skills such as rim cleaning, mixing, and watering—demonstrating that small adjustments in reward structure can yield significant variations in learned strategies. Our analysis demonstrates that even small changes in reward structure can lead to significant shifts in policy behavior, facilitating both task-optimal and creative, potentially transferable strategies. To validate this concept, we implement it using the Proximal Policy Optimization (PPO) algorithm, showing that reward design alone—without altering the learning architecture—can drive meaningful skill diversification. This approach offers promising directions for adaptive control, transfer learning, and multi-objective optimization in robotic systems. ... Read more

Personalised rehabilitation is essential for restoring shoulder function following injury or surgery, particularly due to the joint’s anatomical complexity and variability across patients. While robotic systems offer consistent and intensive therapy, they often neglect internal biomechanical stress, such as tendon strain, potentially resulting in movements that appear safe externally but exceed physiological limits.

Recent approaches have integrated strain maps into robotic rehabilitation, but remain limited to patient-led motion. This presents a major limitation during early-stage rehabilitation, when patients cannot initiate movement themselves. To address this, we present a method that enables robot-led execution of therapist-defined shoulder movements, while adapting these movements to individual patient anatomies using musculoskeletal strain data.

Our system uses Dynamic Movement Primitives (DMPs) to encode a demonstrated shoulder trajectory, then adapts the motion based on strain maps generated in OpenSim. Regions of elevated tendon strain are modelled as ellipsoidal high-strain zones. These zones exert repulsive forces during trajectory execution, and an adaptive time scaling mechanism slows motion near unsafe areas to ensure smoothness and safety.

We validated the system using two input trajectories: one manually drawn and one recorded through kinaesthetic teaching with a KUKA LBR iiwa-7 robot. Both were adapted across six virtual patient models with varying tendon insertion points. In all cases, the adapted trajectories avoided high-strain zones and remained dynamically feasible. While some shape deviations occurred in constrained anatomies, the system maintained smooth and plausible motion across patient variants. Compared to a baseline, our method produced smoother trajectories with reduced peak accelerations. Unlike prior systems that rely on patient-led adaptation, our approach enables fully robot-driven motion generalization, making it particularly suitable for early-stage rehabilitation where patients cannot safely or actively control movement.

While this study focuses on a single muscle and 2D motion, the framework establishes a foundation for future extensions to 3D movement and multi-muscle safety adaptation. These results highlight the potential of strain-aware robot-led therapy to deliver safe and personalised rehabilitation, especially in early recovery stages when patients cannot actively participate. ... Read more

Three-year-old toddlers can effortlessly guide a toy train along a wooden track, whereas this slide-in-the-groove position tracking task requires a skilled operator using a teleoperated robot arm due to the lack of direct contact and force feedback. Although an autonomous robot can perform this task in a fixed setup, telerobotics is crucial for unknown environments where human control is essential, as humans provide the adaptability needed to handle unpredictable conditions. The introduction of torque-controlled motors and haptic devices has enhanced teleoperation by improving telepresence and immersion. Operators can perceive interaction forces through the primary position control input via the haptic device, while a secondary control input allows them to adjust the robot arm's impedance. This ability, known as teleimpedance, allows operators to control the robot’s physical interaction based on environmental context. A toddler naturally remains relaxed in the plane perpendicular to the train's forward direction, where gravity and the groove sides provide stability, preventing derailment and wheel damage. At the same time, they maintain firmness along the track for smooth forward movement. Tele-impedance enables the operator to achieve a similar balance. It allows adaptation of an optimal balance of low and high impedance in different axes of Cartesian space to match task demands. Current impedance control interfaces rely on complex muscle activity measurements, requiring long calibration procedures to map the operator’s arm stiffness to the robot arm. Other interfaces use hand-controlled input devices that must be operated in addition to the haptic device, reducing the operator's cognitive bandwidth for the position tracking task. Existing interfaces typically provide only partial stiffness control or introduce visual distractions. In contrast, we propose a novel visio-verbal interface that leverages gaze and speech, natural modes of interaction, to enable hands-free semi-autonomous control of translational stiffness in all three dimensions while maintaining visual attention on the position tracking task. The interface’s vision-language model (VLM) determines the three-dimensional robot endpoint stiffness by combining the operator's verbal intent with gaze estimates from a mobile eye tracker. We demonstrate a proof of concept for this approach. The hardware includes Tobii Pro Glasses 2 eye trackers, a Force Dimension sigma7 haptic position input interface, and a KUKA LBR iiwa collaborative robot arm equipped with a custom-built endpoint camera mount for the Realsense D455 camera and a 3D-printed peg to evaluate the interface in a 3D-printed U-shaped slot for a slide-in-the-groove task similar to guiding a toy train. ... Read more

This paper proposes a multimodal controller that interactively leverages pose and velocity control for teleoperation, designed to address workspace limitations by dynamic workspace reindexing while taking into account operator ergonomics. Dynamic workspace reindexing offers a solution to the limitations of existing approaches, such as scaling and clutching. To ensure operator ergonomics, The Rapid Upper Limb Assessment (RULA) method is used to define an Ergonomic Workspace (EW) within which the operator must remain to maintain an ergonomic posture. Within the boundaries of the EW, non-scaled pose control is used to control the follower, offering intuitive interaction with the remote environment while maintaining good operator ergonomics. Outside the EW boundaries, velocity control is applied, where the velocity of the follower is based on the force exerted by the operator on the leader haptic device, allowing the operator to dynamically reindex the follower workspace. This control mode facilitates coarse positioning of the follower between targets. A proof-of-concept demonstration shows that the proposed controller succesfully addresses workspace limitations by dynamically reindexing the follower's workspace towards target objects. Furthermore, it is shown that the controller consistently maintains good operator ergonomics during interaction with the remote environment, thereby making it a suitable option for prolonged teleoperation tasks. ... Read more

This study examines how fixed robot personalities (patient, impatient, leader, follower) influence co-learning in human-robot teams by answering the research question: How do different robot personalities influence co-learning. To do this, we implemented a reinforcement learning framework for a handover task where a robot and human participant co-learn to solve a task. The robot has personalities encoded along two axes: patient/impatient (via motion speed and stiffness) and leader/follower (via exploration rates and reward structures in phased Q-learning).

Through a within-subject design, we analyze policy metrics and human perceptions. While task success rates remain stable, strategy and internal policy metrics vary significantly. This underpins the key finding: robot personality does not affect task performance since humans can adapt to overcome subtle differences in robot personality. However, robot personality significantly affects how the collaboration is performed as human-robot teams adopt different strategies for different robot personalities.

Results demonstrate that robot personality is salient for differences in physical behaviour yet is unperceivable for modifications of internal parameters like exploration rate/decay and reward function for short interactions. This work bridges a critical gap in understanding how static robot traits shape collaborative adaptation, even when overt performance metrics remain unchanged. ... Read more

This work demonstrates successful adaptation of a stochastic planning framework to increase movement accuracy and speed in human-robot co-manipulation during assembly tasks. The approach provides an alternative to data-driven methods, enabling fast adaptation to new tasks in increasingly important high-mix, low-volume industrial settings. The approach enables more efficient and intuitive collaboration by exploiting the strengths of both the human and the robot. During assembly, a robot can augment accuracy by constraining movement to a known handover plane, while the human controls in-plane positioning using perception and task knowledge. By constraining variability in the direction holding the highest collision risk, this accuracy augmentation is hypothesized to enable humans to move faster while accepting higher variability in lower-consequence in-plane dimensions. The planning framework developed in this work generates human-like trajectories for 7-DoF manipulators. It incorporates a final state constraint to suppress endpoint variability for accuracy augmentation and a Fitts' law-based temporal constraint predicting duration from predicted task difficulty. Recursive parameter estimation adapts this temporal constraint to the hypothesized speed increase resulting from accuracy augmentation. Experimental validation provides evidence for this hypothesis and demonstrates successful adaptation. After convergence, the planner predicted movements 46\% faster at constant task difficulty compared to without the planner. These results demonstrate that robots should anticipate and adapt to changed human behavior resulting from their assistance, enabling more efficient and intuitive co-manipulation. ... Read more

Offshore geotechnical exploration presents unique challenges due to harsh environments, limited sensor data, and the need for high-fidelity soil characterization. This has led to the deployment of automated solutions, such as subsea platforms, that support exploration activities. These robotic systems are often coupled with intelligent systems capable of performing tasks like drilling autonomously. This work focuses on developing an intelligent advisory system for the Blue Dragon subsea platform to enable autonomous drilling operations. To address the challenges presented by this unique application, the research explores multi-task learning to enhance timeseries forecasting models that predict optimal drilling parameters including thrust, torque, and rotation speed.
Four model architectures were evaluated: a baseline forecasting model, a forecasting model enriched with a separately trained lithology classifier, a jointly trained classifier-forecaster system, and an unsupervised embedding extractor approach. These models were tested on both synthetic vehicle dynamics data and real-world drilling data from Blue Dragon® operations. While the synthetic data demonstrated theoretical soundness of the approaches, evaluation on the drilling dataset revealed significant limitations due to insufficient data quality and quantity, resulting in severe overfitting that prevented conclusive model validation.
The results indicate that although lithology-informed forecasting shows theoretical promise, current data constraints prevent robust generalization to operational conditions. Future work should prioritize acquiring production-grade datasets and developing refined data preprocessing techniques to better isolate drilling-relevant signals. This research establishes foundational methodology for safe, efficient, and adaptive automation in offshore drilling, contributing to the broader objective of autonomous geotechnical exploration. ... Read more

The state-of-the-art teleimpedance command interfaces used to command the robot stiffness configuration are either too complex to set up, such as those that use physiological signals and other tracking methods or cannot configure the stiffness appropriately for 3d environments.
To mitigate these issues, a novel teleimpedance interface is proposed.
The proposed interface can independently control the stiffness configuration's shape, orientation, and size with single-hand operations while allowing the operator to use that hand to command the robot's position.
The teleimpedance interface is attached to the operator's hand and uses two scroll wheels, a joystick, and a force sensor to configure the robot's stiffness and has two different modes of operation.
Compared to the state-of-the-art methods, the main advantage of the proposed teleimpedance command interface is that it does not require additional hardware with force feedback or complex setup calibrations while allowing for control of the robot's 3D stiffness configuration with single-handed operation.
An experiment with human subjects was performed to demonstrate the proposed interface's acceptance and functionality.
To demonstrate the teleimpedance command interface's ability to adjust 3D stiffness configurations a teleoperation was performed, utilizing a Kuka robotic arm and a Force Dimension Sigma7 position input interface.
The teleimpedance interface functioned as intended during teleoperation in a 3D environment to configure and adjust the 3D stiffness configuration for the task in real-time.
The results from the human subject trials indicate that the participants can successfully operate the interface to complete the alignment tasks in both modes for 3D stiffness configurations. ... Read more

This thesis presents the design and evaluation of a comprehensive system for developing voice-based interfaces to support users in supermarkets. These interfaces enable customers to convey their needs across both generic and specific queries. While current state-of-the-art systems like GPTs by OpenAI are easily accessible and adaptable, featuring low-code deployment with options for functional integration, they still face challenges such as increased response times and limitations in strategic control for tailored use-case and cost optimisation. Motivated by the goal of crafting inclusive, personalised, and efficient conversational agents, this study advances on three fronts: 1) a comparative analysis of four popular off-the-shelf speech recognition technologies to identify the most accurate model for different genders (male/female) and languages (English/Dutch); 2) an assessment of the effects of personalised recommendations versus generic responses, using a blindfolded, counterbalanced within-subject experiment; and 3) the development and evaluation of a novel multi-LLM supermarket chatbot framework, comparing its performance with a specialized GPT model powered by the GPT-4 Turbo, using the Artificial Social Agent Questionnaire (ASAQ) in a counterbalanced within-subjects experiment and qualitative participant feedback. Our find-ings reveal that OpenAI’s Whisper leads in speech recognition accuracy across genders and languages, users significantly prefer personalised chatbots over the non-personalised counterparts and that our proposed multi-LLM chatbot architecture outperformed the benchmarked GPT model across all 13 measured criteria, including statistically significant improvements in four key areas: performance, user satisfaction, user-agent partnership, and self-image enhancement. The thesis concludes by presenting a simple method for supermarket robot navigation by mapping the final chatbot response to correct shelf numbers towards which the robot can plan sequential visits. This later enables effective use of low-level perception, motion planning, and control capabilities for product retrieval and collection. We hope this work encourages more efforts into using multiple, specialised smaller models instead of always relying on a single powerful model. ... Read more

This paper proposes a novel framework that combines both planning and learning-based trajectory generation methods to handle complex robotic assembly tasks. The framework utilizes MoveIt! for planning large-scale reaching motions and Dynamic Movement Primitives (DMPs) for precise grasping and placing movements, with both methods integrated into a single system controlled by a behavior tree. An impedance controller is employed to ensure smooth and safe execution of the generated trajectories, particularly in scenarios that involve human interaction.

The proposed framework was evaluated within the context of the European Space Agency-funded Rhizome project, which focuses on off-earth habitat construction. The project involves assembling habitats using custom-designed Voronoi-shaped building blocks, which were also utilized in experiments to test the framework. The results showed that combining planning for large-reaching motions with DMPs for detailed movements effectively addressed the limitations of each individual method, delivering a flexible and robust solution to the challenges of robotic assembly. ... Read more

In this work, we propose a method of processing patient input on discomfort level during robot shoulder physiotherapy into discomfort maps. These maps represent the patient's discomfort distribution throughout the range of motion of the shoulder, interpretable by both physiotherapists and robots. This method consists of three parts: the patient can input discomfort with a linear push-button; a collaborative robot arm is used to track the motion of the patient's shoulder; and audiovisual feedback of inputted discomfort is given to the patient and the therapist.
The method was validated in human factors experiments simulating shoulder physiotherapy sessions, where the subject is tasked with recreating a reference discomfort map through an auditory reference signal that emulates this discomfort. Here the robot also acts as the physiotherapist, moving the subject's shoulder. The signal is a beeping sound, whose rate scales with the discomfort intensity at the measured pose in the reference discomfort map.
We performed experiments with a total of 10 participants, demonstrating the viability of our method during patient-robot interaction. The results we collected also highlighted the presence of a time delay between the discomfort signal and the user input, and its effect on discomfort maps. ... Read more

While Artificial Intelligence (AI) is geared towards automating tasks like writing and designing, the challenge persists in finding adequate human resources for tasks such as handling luggage in and out of airplanes or harvesting produce in greenhouses. Nonetheless, the demand to tailor robotic abilities to diverse scenarios, ranging from agriculture to household chores, necessitates a general-purpose morphology for the robot, such as a dexterous arm, along with sufficient sensory capabilities and intelligence to swiftly adjust to new situations.
Despite the prevalence of click-baiting videos shared online, current robot technologies have yet to address this requirement adequately. The primary obstacle hindering robot manipulators from effectively performing daily chores, aiding in supermarkets, and harvesting fruits from fields is the insufficient data available to construct a robust model of the world. Typically, autonomously exploring their surroundings and determining optimal strategies is considered unsafe and impractical.
A more effective approach to imparting knowledge to robots involves human supervision. Ideally, this entails interactive supervision where robots can seek clarification when uncertain about a situation, and humans can intervene when the robot’s actions are incorrect or fail to meet the required performance. Moreover, when receiving instructions or asking for them, the robot should quantify the confidence in the interpretation of the corrections. This thesis makes significant contributions to the field of interactive robot learning by introducing various uncertainty-aware methods. These methods facilitate enhancements in data efficiency during learning and safety during execution.
Before delving into the main contributions, Chapter 2 introduces the reader to the topic of Interactive Imitation Learning (IIL) and the different modalities that can be used to give feedback, from evaluative to corrective, underlying the importance of uncertainty quantification on the robot belief. For this reason, Chapter 3, introduces the foundations of the main function approximator used in this thesis, i.e. Gaussian Process (GP), to learn behaviors while quantifying uncertainties. The chapter highlights how a GP is trained given the evidence of the data and the corrections and how predictions of the mean and the variance of the actions are obtained. Particular attention is given to how GP models can be used for efficient updating and aggregation of online data and how to analytically estimate the uncertainty rate of change.
The proposed function approximator is first applied in Chapter 4. The presented machine learning framework allows the robot to learn complex manipulation tasks from interactive demonstrations. Essentially, the user needs to show a kinesthetic demonstration to the robot, i.e. dragging the robot around in a fully compliant modality to transfer their knowledge on a desired skill, e.g. cleaning a table or inserting a plug in a socket. The experiments highlight how the quantification and the rejection of uncertainties can be used to bring the robot always close to high-confidence regions. Moreover, the GP online model update is used to aggregate the corrections received from the user to reshape the learned attractor and the stiffness field. This ensures that the proper force is executed in the correct direction for instance when cleaning a table.
To extend the learning of a skill to the whole robot pose and gripper, Chapter 5 studies how to address this with GP and with the least amount of demonstrations and corrections. Moreover, the experiments focus on teaching human-like skills to robots by exploiting the possibility of giving incremental corrections. In particular, novice users, are asked to perform the picking task of objects in one fluid motion by teaching the complete pose and gripper behavior. The execution of the skill without any supervision is usually too slow or knocks the object down before closing the gripper. Nevertheless, after providing feedback, novice users were able to incrementally shape the robot’s velocity to perform the picking at non-zero velocity, without knocking the object and correcting for any delay in gripper dynamics.
However, learning skills only relying on the current robot’s Cartesian position can be a limitation since it cannot encode skills that entail overlapping, e.g. when approaching a goal and then moving back on the same trajectory. This motivates Chapter 6 which formulates a new trajectory encoding to teach single or bimanual manipulation skills while being safe around humans with constrained velocity and force actuation. The user study also investigates the effectiveness of giving kinesthetic corrections, i.e. by simply touching the robot, and validating this in teaching bimanual skills. Teaching two manipulators at the same time or correcting them using teleoperation devices can become overwhelming. Hence, the method explores adjusting movements interactively through kinesthetic perturbations rather than re-teaching skills entirely from scratch due to imprecise attempts.
Despite the successful applications of the proposed methods in single and bimanual motion skills, during task learning, the robot must not only master the motor aspect but also be attentive to the context, such as the object’s location or shape. This motivates Chapter 7, which emphasizes the generalization of acquired motor skills across various contexts. The proposed approach hinges on GP theory to acquire a non-linear transformation map from the demonstrated task space to the execution space while preserving and propagating uncertainties. Through experiments involving tasks such as pick-and-place operations, dressing human arms, and cleaning surfaces, it is demonstrated how the robot can generalize the execution by transforming the attractor, orientation, and stiffness policy to numerous new scenario configurations even with just a single demonstration of the skill.
In Chapter 8, the concept of task parametrization and uncertainty awareness is expanded to over-parameterizing the context, such as by tracking more objects than required. The proposed algorithm would prompt user attention when encountering ambiguity, like when multiple detected objects could be the goal of the skill. Decision ambiguity can be resolved by various feedback modalities, such as pushing the robot, moving it, or providing reward/punishment. A user study also highlighted the preference of novice users for not giving conventional kinesthetic demonstrations but only intervening when necessary. ... Read more

Face recognition using lidar presents challenges arising from high dimensionality and data sparsity, especially at longer distances. This paper proposes a novel approach for face recognition via automotive lidar. The approach leverages a combination of deep learning and point cloud processing techniques. After identification of the facial point clouds, an alpha-shaped convex hull is employed for regional linearization, resulting in the creation of a depth image. This depth image is then fed to a convolutional neural network architecture, BasicNet, specifically trained for face recognition. The approach is evaluated on a dataset comprising 52 individuals acquired using two lidar sensors with different point densities. The individuals walked at distances ranging from 5 to 18 meters from the sensors. The approach achieves interesting results on this challenging dataset, thereby challenging the notion that lidar sensors are privacy-preserving. ... Read more