Rhizome 1.0 and 2.0 are European Space Agency (ESA) co-funded projects that have been implemented with a team from the Architecture, Mechanical Engineering, and Aerospace Engineering Faculties, TU Delft, and various industrial partners. The focus is on the development of a Martian habitat using 3D-printed components and in situ resource utilization. This paper presents a new multi-modal method developed for the collaborative assembly of building components with the support of Computer Vision (CV) and Human-Robot Interaction (HRI) using compliant robotic collaborative manipulators. The building components are Voronoi-based and are fabricated using Design-to-Robotic-Production and -Assembly (D2RP&A). During the collaborative assembly, the robot uses CV to detect the fabricated component and generate autonomous actions to perform the pick-and-place movement. Between the autonomous robot actions, HRI is used by the human to physically guide the robot when grasping and positioning. To evaluate the proposed method, lab experiments were conducted using robotically milled mock-up components from Styrofoam, which were assembled with a collaborative robotic arm. The results indicate that robots can assist humans during the assembly process to implement tasks that are beyond their physical abilities, while robots benefit from human cognitive capabilities during more complex actions.

For successful goal-directed human-robot interaction, the robot should adapt to the intentions and actions of the collaborating human. This can be supported by musculoskeletal or data-driven human models, where the former are limited to lower-level functioning such as ergonomics, and the latter have limited generalizability or data efficiency. What is missing, is the inclusion of human motor control models that can provide generalizable human behavior estimates and integrate into robot planning methods. We use well-studied models from human motor control based on the speed-accuracy and cost-benefit trade-offs to plan collaborative robot motions. In these models, the human trajectory minimizes an objective function, a formulation we adapt to numerical trajectory optimization. This can then be extended with constraints and new variables to realize collaborative motion planning and goal estimation. We deploy this model, as well as a multi-component movement strategy, in physical collaboration with uncertain goal-reaching and synchronized motion tasks, showing the ability of the approach to produce human-like trajectories over a range of conditions.

This paper presents the design and evaluation of a comprehensive system to develop voice-based interfaces to support users in supermarkets. These interfaces enable shoppers to convey their needs through both generic and specific queries. Although customisable state-of-the-art systems like GPTs from OpenAI are easily accessible and adaptable, featuring low-code deployment with options for functional integration, they still face challenges such as increased response times and limitations in strategic control for tailored use cases and cost optimization. Motivated by the goal of crafting equitable and efficient conversational agents with a touch of personalisation, this study advances on two fronts: 1) a comparative analysis of four popular off-the-shelf speech recognition technologies to identify the most accurate model for different genders (male/female) and languages (English/Dutch) and 2) the development and evaluation of a novel multi-LLM supermarket chatbot framework, comparing its performance with a specialized GPT model powered by the GPT-4 Turbo, using the Artificial Social Agent Questionnaire (ASAQ) and qualitative participant feedback. Our findings reveal that OpenAI’s Whisper leads in speech recognition accuracy between genders and languages and that our proposed multi-LLM chatbot architecture, which outperformed the benchmarked GPT model in performance, user satisfaction, user-agent partnership, and self-image enhancement, achieved statistical significance in these four key areas out of the 13 evaluated aspects that all showed improvements. The paper concludes with a simple method for supermarket robot navigation by mapping the final chatbot response to the correct shelf numbers to which the robot can plan sequential visits. Later, this enables the effective use of low-level perception, motion planning, and control capabilities for product retrieval and collection. We hope that this work encourages more efforts to use multiple specialized smaller models instead of always relying on a single powerful model.

In this paper, we present a design and evaluation of a novel finger-operated teleimpedance interface used to command stiffness ellipsoids to the remote robot. The proposed interface provides a practical alternative to the state-of-the-art teleimpedance interfaces based on physiological signals that can be impractical in daily use. On the other hand, as opposed to existing practical interfaces that lack in terms of controlled degrees of freedom, the proposed interface enables control of 3D aspects of the ellipsoid. The remote robot stiffness ellipsoid is controlled with a single hand using the thumb, index, and middle fingers to operate two scroll wheels, a joystick, and a force sensor. These combinations of inputs can be mapped to control different aspects of the stiffness ellipsoid, i.e., orientation and shape/size. To investigate different modes of input mapping, we perform a human factors experiment to evaluate the performance and user acceptance of the proposed interface modes. The results of the experiments indicate that the participants can successfully operate the interface to complete 3D stiffness configuration alignment tasks in different modes. To further demonstrate the functionality of the proposed teleimpedance interface, we performed an additional experiment utilising a Force Dimension Sigma7 haptic device to control the motion of a KUKA LBR iiwa robotic arm while performing a complex physical interaction task.

In this work, we propose a method of capturing the patient’s discomfort during robotic shoulder physiotherapy, creating "discomfort maps". These maps depict the personalized distribution of discomfort that each patient perceived across their shoulder range of motion, facilitating both robotic devices and human therapists to account for patient-specific characteristics during the therapeutic process. Our system enables a patient to communicate and map discomfort in space and time during movement via a handheld push-button device, while interacting with a robotic physical therapy device capable of moving the patient and estimating their pose. We validated our method through human factors experiments simulating shoulder physiotherapy sessions with 10 healthy participants. To avoid the risk of injury to the participants and to allow for ground truth map information, we emulate perceived discomfort via an auditory signal. Our experimental apparatus enabled participants to reconstruct synthetic discomfort maps, demonstrating the feasibility of automatically capturing and storing patient discomfort during robotic physiotherapy.

In the original version of the book, on page xi, one of the authors listed for Chapter 2 is “R. Schnmehl”, which should be “R. Schmehl”. On page 21, the same correction needs to be made twice, in the listed authors at the top of the page and also in the footnotes. “Schnmehl” should become “Schmehl” in both the cases. This has now been rectified and the author’s name has been corrected. The correction to this book have been updated with the changes.

In order for off-Earth top surface structures built from regolith to protect astronauts from radiation, they need to be several metres thick. In a feasibility study, funded by the European Space Agency, Technical University Delft (TUD aka TU Delft) explored the possibility of building in empty lava tubes to create rhizomatic subsurface habitats. With this approach natural protection from radiation is achieved as well as thermal insulation because the temperature is more stable underground. It involves a swarm of autonomous mobile robots that survey the areas and mine for materials such as regolith in order to create cement-based concrete reproducible on Mars through in-situ resource utilisation (ISRU). The concrete is 3D printed by means of additive Design-to-Robotic-Production (D2RP) methods developed at TUD for on-Earth applications with the 3D printing system of industrial partner, Vertico. The printed components are assembled using a Human--Robot Interaction (HRI) supported approach. The 3D printed and HRI-supported assembled structures are structurally optimised porous material systems with increased insulation properties. In order to regulate the indoor pressurised environment a Life Support System (LSS) is integrated, which in this study is only conceptually developed. The habitat and the D2RP production system are powered by an automated kite power system and solar panels developed at TUD. The long-term goal is to develop an autarkic, automated and HRI-supported D2RP system for building autarkic habitats from locally obtained materials.

Many daily tasks exhibit a periodic nature, necessitating that robots possess the ability to execute them either alone or in collaboration with humans. A widely used approach to encode and learn such periodic patterns from human demonstrations is through periodic Dynamic Movement Primitives (DMPs). Periodic DMPs encode cyclic data independently across multiple dimensions of multi-degree of freedom systems. This method is effective for simple data, like Cartesian or joint position trajectories. However, it cannot account for various geometric constraints imposed by more complex data, such as orientation and stiffness. To bridge this gap, we propose a novel periodic DMP formulation that enables the encoding of periodic orientation trajectories and varying stiffness matrices while considering their geometric constraints. Our geometry-aware approach exploits the properties of the Riemannian manifold and Lie group to directly encode such periodic data while respecting its inherent geometric constraints. We initially employed simulation to validate the technical aspects of the proposed method thoroughly. Subsequently, we conducted experiments with two different real-world robots performing daily tasks involving periodic changes in orientation and/or stiffness, i.e., operating a drilling machine using a rotary handle and facilitating collaborative human–robot sawing.

In recent years, providing additional visual feedback about the interaction forces has been found to offer benefits to haptic-assisted teleoperation. However, there is limited insight into the effects of the design of force feedback-related visual cues and the type of visual display on the performance of teleoperation of robotic arms executing industrial tasks. In this study, we provide new insights into this interaction by extending these findings to the haptic assistance teleoperation of a simulated robotic arm in a virtual environment, in which the haptic assistance is comprised of a set of virtual fixtures. We design a novel method for providing visual cues about the interaction forces to complement the haptic assistance and augment visual feedback in virtual reality with a head-mounted display. We evaluate the visual cues method and head-mounted display method through human factors experiments in a teleoperated dross removal use case. The results show that both methods are beneficial for task performance, each of them having stronger points in different aspects of the operation. The visual cues method was found to significantly improve safety in terms of peak collision force, whereas the head-mounted display additionally improves the performance significantly. Furthermore, positive scores of the subjective analysis indicate an increased user acceptance of both methods. This work provides a new study on the importance of visual feedback related to (interaction) forces and spatial information for haptic assistance and provides two methods to take advantage of its potential benefits in the teleoperation of robotic arms.

During the learning of a new sensorimotor task, individuals are usually provided with instructional stimuli and relevant information about the target task. The inclusion of haptic devices in the study of this kind of learning has greatly helped in the understanding of how an individual can improve or acquire new skills. However, the way in which the information and stimuli are delivered has not been extensively explored. We have designed a challenging task with nonintuitive visuomotor perturbation that allows us to apply and compare different motor strategies to study the teaching process and to avoid the interference of previous knowledge present in the naïve subjects. Three subject groups participated in our experiment, where the learning by repetition without assistance, learning by repetition with assistance, and task Segmentation Learning techniques were performed with a haptic robot. Our results show that all the groups were able to successfully complete the task and that the subjects’ performance during training and evaluation was not affected by modifying the teaching strategy. Nevertheless, our results indicate that the presented task design is useful for the study of sensorimotor teaching and that the presented metrics are suitable for exploring the evolution of the accuracy and precision during learning.

This paper presents a method for semiautonomous teleimpedance where the control is shared between the human operator and the robot. The human commands the position of the teleoperated robotic arm end-effector while the robot autonomously adjusts the impedance depending on the object with which the end-effector interacts. We developed a vision system that calculates the appropriate robot stiffness based on the detected object geometry and material and object’s relation to the environment. This system uses an RGB-D camera near the robot’s end-effector to capture different perspectives of the scene. To validate the proposed method, we conducted experiments on a teleoperation system where a Force Dimension Sigma7 haptic device was used to operate a KUKA LBR iiwa robotics arm. At the same time, the Intel RealSense D455 depth camera provided the visual input. We examined two practical tasks: engaging with bolts on a plate and polishing a stripe.

Teleoperation is a crucial technology enabling human operators to control robots remotely to perform tasks in hazardous and difficult-to-access environments. Tasks in such environments often involve complex physical interactions with tools and objects of various softness. To this end, teleimpedance enables the operators to adjust the robot impedance in real-time to simplify such interactions. While the existing teleimpedance approaches provide several interfaces to command the robot impedance, there are no interfaces to visualize both the commanded impedance and that of the objects to be interacted with. This paper presents a novel interface to provide visual feedback on the impedance of remote robots and objects. To do so, we use virtual stiffness ellipsoids and different modes that display the individual impedance of the robot and objects as well as combined post-contact impedance. The key advantage of visual feedback on the impedance compared to force feedback is that the operator can see the interaction characteristics before the contact occurs. This enables the operator to act proactively before contact rather than just reactively after the contact. This paper also proposes a new intuitive way to command the robot impedance using mixed reality, interacting with these ellipsoids and modifying them as needed. To demonstrate the key functionalities of the developed interface, we performed proof-of-concept experiments on teleoperated tasks.

Performing bimanual tasks with dual robotic setups can drastically increase the impact on industrial and daily life applications. However, performing a bimanual task brings many challenges, such as synchronization and coordination of the single-arm policies. This article proposes the safe, interactive movement primitives learning (SIMPLe) algorithm, to teach and correct single or dual arm impedance policies directly from human kinesthetic demonstrations. Moreover, it proposes a novel graph encoding of the policy based on Gaussian process regression where the single-arm motion is guaranteed to converge close to the trajectory and then toward the demonstrated goal. Regulation of the robot stiffness according to the epistemic uncertainty of the policy allows for easily reshaping the motion with human feedback and/or adapting to external perturbations. We tested the SIMPLe algorithm on a real dual-arm setup where the teacher gave separate single-arm demonstrations and then successfully synchronized them only using kinesthetic feedback or where the original bimanual demonstration was locally reshaped to pick a box at a different height.

Despite a large body of research on robot learning, it has not yet been thoroughly studied how collaborating humans and robots learn reciprocally. In such situations, both humans and robots continuously learn about each other and the task through interaction. This letter addresses the research question: "How can human-robot co-learning be facilitated in physically embodied collaborative tasks?". First, we derived five requirements for successful human-robot co-learning from literature: shared goal, synchrony, interdependence, adaptability, and transparency. Based on these requirements, we designed a collaborative human-robot handover task and a robot Q-learning method. In an evaluation with six human participants co-learning was indeed found to emerge in the hand-over task. Particularly, for three of the human-robot dyads, our designed setup proved to facilitate co-learning in a way that met all five requirements. The task and robot learning method presented in this letter demonstrate how human-robot co-learning can be enabled in physically embodied tasks.

Real-world applications of Artificial Intelligence (AI) in architecture have been explored more recently at Technical University (TU) Delft by integrating AI in Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) methods. These embed robotics into building processes and buildings by linking computational design with robotic construction and/ or operation of building components and buildings. This paper presents two case studies in which AI-supported D2RA is implemented in a multidisciplinary approach that requires the integration of research domains such as architecture, robotics, computer and material science.

Robotic tasks which involve uncertainty – due to variation in goal, environment configuration, or confidence in task model – may require human input to instruct or adapt the robot. In tasks with physical contact, several existing methods for adapting robot trajectory or impedance according to individual uncertainties have been proposed, e.g., realizing intention detection or uncertainty-aware learning from demonstration. However, isolated methods cannot address the wide range of uncertainties jointly present in many tasks. To improve generality, this paper proposes a model predictive control (MPC) framework which plans both trajectory and impedance online, can consider discrete and continuous uncertainties, includes safety constraints, and can be efficiently applied to a new task. This framework can consider uncertainty from: contact constraint variation, uncertainty in human goals, or task disturbances. An uncertainty-aware task model is learned from a few (≤3) demonstrations using Gaussian Processes. This task model is used in a nonlinear MPC problem to optimize robot trajectory and impedance according to belief in discrete human goals, human kinematics, safety constraints, contact stability, and frequency-domain disturbance rejection. This MPC formulation is introduced, analyzed with respect to convexity, and validated in co-manipulation with multiple goals, a collaborative polishing task, and a collaborative assembly task.

Daily household tasks involve manipulation in cluttered and unpredictable environments and service robots require complex skills and adaptability to perform such tasks. To this end, we developed a teleoperated online learning approach with a novel skill refinement method, where the operator can make refinements to the initially trained skill by a haptic device. After a refined trajectory is formed, it is used to update a probabilistic trajectory model conditioned to the environment state. Therefore, the initial model can be adapted when unknown variations occur and the method is able to deal with different object positions and initial robot poses. This enables human operators to remotely correct or teach complex robotic manipulation skills. Such an approach can help to alleviate shortages of caretakers in elderly care and reduce travel time between homes of different elderly to reprogram the service robots whenever they get stuck. We performed a human factors experiment on 18 participants teaching a service robot how to empty a dishwasher, which is a common daily household task performed by caregivers. We compared the developed method against three other methods. The results show that the proposed method performs better in terms of how much time it takes to successfully adapt a model and in terms of the perceived workload.

Biological systems, including human beings, have the innate ability to perform complex tasks in a versatile and agile manner. Researchers in sensorimotor control have aimed to comprehend and formally define this innate characteristic. The idea, supported by several experimental findings, that biological systems are able to combine and adapt basic units of motion into complex tasks finally leads to the formulation of the motor primitives’ theory. In this respect, Dynamic Movement Primitives (DMPs) represent an elegant mathematical formulation of the motor primitives as stable dynamical systems and are well suited to generate motor commands for artificial systems like robots. In the last decades, DMPs have inspired researchers in different robotic fields including imitation and reinforcement learning, optimal control, physical interaction, and human–robot co-working, resulting in a considerable amount of published papers. The goal of this tutorial survey is two-fold. On one side, we present the existing DMP formulations in rigorous mathematical terms and discuss the advantages and limitations of each approach as well as practical implementation details. In the tutorial vein, we also search for existing implementations of presented approaches and release several others. On the other side, we provide a systematic and comprehensive review of existing literature and categorize state-of-the-art work on DMP. The paper concludes with a discussion on the limitations of DMPs and an outline of possible research directions.

Skill propagation among robots without human involvement can be crucial in quickly spreading new physical skills to many robots. In this respect, it is a good alternative to pure reinforcement learning, which can be time-consuming, or learning from human demonstration, which requires human involvement. In the latter case, there may not be enough humans to quickly spread skills to many robots. However, propagation among robots without direct human supervision can result in robotic skills mutating from the original source. This can be beneficial when better skills might emerge or when a new skill is obtained to be used for other similar tasks. However, it can also be dangerous in terms of task execution safety. This letter studies the mutation of a robotic skill when it is propagated from one robot to another during a physically collaborative task. We chose the collaborative sawing task as a study case since it involves complex two-agent physical interaction/coordination and because its periodic nature can facilitate repetitive learning. The study employs periodic Dynamic Movement Primitives and Locally Weight Regression to encode and learn the motion and impedance required to execute the task. To explore what influences mutation, we varied several control and environment conditions such as the maximum stiffness, robot base position, friction coefficient of the sawed object, and movement period. The results showed that the skill varied over propagation steps and we identified several key aspects of mutation such as movement length, movement offset, and trajectory shape. Based on the results we identified possible benefits (skill mutations useful for different settings or different tasks, and energy efficiency) and dangers (high forces and skill mutations becoming useless for the original task) of the mutation.