The paper presents a visio-verbal teleimpedance interface for commanding 3D stiffness ellipsoids to the remote robot with a combination of the operator’s gaze and verbal interaction. The gaze is detected by an eye-tracker, allowing the system to understand the context in terms of what the operator is currently looking at in the scene. Along with verbal interaction, a Vision-Language Model (VLM) processes this information, enabling the operator to communicate their intended action or provide corrections. Based on these inputs, the interface can then generate appropriate stiffness matrices for different physical interaction actions. To validate the proposed visio-verbal teleimpedance interface, we conducted a series of experiments on a setup including a Force Dimension Sigma.7 haptic device to control the motion of the remote Kuka LBR iiwa robotic arm. The human operator’s gaze is tracked by Tobii Pro Glasses 2, while human verbal commands are processed by a VLM using GPT-4o. The first experiment explored the optimal prompt configuration for the interface. The second and third experiments demonstrated different functionalities of the interface on a slide-in-the-groove task

Constructing a Martian habitat presents significant challenges due to extreme temperature variations and a low-density and -pressure atmosphere. To address these challenges a habitat constructed from prefabricated, interlocking Voronoi-based components that are assembled by human-robot collaboration has been explored in the Rhizome projects at TU Delft. In this paper, we propose a combined robot motion planning and learning method that can optimize human involvement in assembly tasks in on-site construction. The proposed hybrid approach exploits motion planning to create motion trajectories for aspects of the task where robot autonomy is capable of solving the problem on its own using sensors and intelligence. When the task becomes too difficult for existing planning capabilities, the human can step in and teach motion trajectories via kinaesthetic demonstration using Dynamic Movement Primitives (DMPs). The trajectories are then executed on the low level by an impedance controller to handle the physical interaction with the environment during the assembly. The decision-making process is managed by a behavior tree.

For successful goal-directed human-robot interaction, the robot should adapt to the intentions and actions of the collaborating human. This can be supported by musculoskeletal or data-driven human models, where the former are limited to lower-level functioning such as ergonomics, and the latter have limited generalizability or data efficiency. What is missing, is the inclusion of human motor control models that can provide generalizable human behavior estimates and integrate into robot planning methods. We use well-studied models from human motor control based on the speed-accuracy and cost-benefit trade-offs to plan collaborative robot motions. In these models, the human trajectory minimizes an objective function, a formulation we adapt to numerical trajectory optimization. This can then be extended with constraints and new variables to realize collaborative motion planning and goal estimation. We deploy this model, as well as a multi-component movement strategy, in physical collaboration with uncertain goal-reaching and synchronized motion tasks, showing the ability of the approach to produce human-like trajectories over a range of conditions.

This paper presents the design and evaluation of a comprehensive system to develop voice-based interfaces to support users in supermarkets. These interfaces enable shoppers to convey their needs through both generic and specific queries. Although customisable state-of-the-art systems like GPTs from OpenAI are easily accessible and adaptable, featuring low-code deployment with options for functional integration, they still face challenges such as increased response times and limitations in strategic control for tailored use cases and cost optimization. Motivated by the goal of crafting equitable and efficient conversational agents with a touch of personalisation, this study advances on two fronts: 1) a comparative analysis of four popular off-the-shelf speech recognition technologies to identify the most accurate model for different genders (male/female) and languages (English/Dutch) and 2) the development and evaluation of a novel multi-LLM supermarket chatbot framework, comparing its performance with a specialized GPT model powered by the GPT-4 Turbo, using the Artificial Social Agent Questionnaire (ASAQ) and qualitative participant feedback. Our findings reveal that OpenAI’s Whisper leads in speech recognition accuracy between genders and languages and that our proposed multi-LLM chatbot architecture, which outperformed the benchmarked GPT model in performance, user satisfaction, user-agent partnership, and self-image enhancement, achieved statistical significance in these four key areas out of the 13 evaluated aspects that all showed improvements. The paper concludes with a simple method for supermarket robot navigation by mapping the final chatbot response to the correct shelf numbers to which the robot can plan sequential visits. Later, this enables the effective use of low-level perception, motion planning, and control capabilities for product retrieval and collection. We hope that this work encourages more efforts to use multiple specialized smaller models instead of always relying on a single powerful model.

This article proposes a novel method of coverage path planning for the purpose of scanning an unstructured environment autonomously. The method uses the morphological skeleton of a prior 2D navigation map via SLAM to generate a sequence of points of interest (POIs). This sequence is then ordered to create an optimal path based on the robot’s current position. To control the high-level operation, a finite state machine (FSM) is used to switch between two modes: navigating toward a POI using Nav2 and scanning the local surroundings. We validate the method in a leveled, indoor, obstacle-free, non-convex environment, evaluating time efficiency and reachability over five trials. The map reader and path planner can quickly process maps of widths and heights ranging between [196,225] pixels and [185,231] pixels in 2.52ms and 1.7ms, respectively. Their computation time increases with 22.0ns/pixel and 8.17 μs/pixel, respectively. The robot managed to reach 86.5% of all waypoints across the five runs. The proposed method suffers from drift occurring in the 2D navigation map.

Combining biomechanical modeling with robotic physiotherapy is a promising direction to provide real-time insights during the rehabilitation of patients with musculoskeletal injuries, such as rotator-cuff tears. One aspect is to prevent re-injuries caused by high strain in the injured tissues while allowing patients to perform the required rehabilitation exercises. In this paper, we propose a novel shared control method for robots to limit unsafe patient movements, through physical guidance based on a strain-space representation of the human rotator cuff. The method provides motion corrections through two complementary predictive modules. The first module exerts a lower degree of intervention and is analogous to rumble strips or speed bumps for cars on the road. In this case, an impedance controller induces variable damping to slow down the patient's movement when a danger zone is approached. The second module produces a higher degree of intervention and is analogous to lane-assist in cars. In this case, the robot plans an optimal deflection trajectory and temporarily takes over control of the movement to avoid an unsafe situation. We performed experiments with a healthy participant acting as a patient and evaluated the effect of different human-robot interaction modalities on the resulting human movement in terms of avoidance of high-strain areas of the rotator-cuff tendons and contact forces exchanged.

In this letter, we investigate whether classical function allocation—the principle of assigning tasks to either a human or a machine—holds for physical Human-Robot Collaboration, which is important for providing insights for Industry 5.0 to guide how to best augment rather than replace workers. This study empirically tests the applicability of Fitts' List within physical Human-Robot Collaboration, by conducting a user study (N=26, within-subject design) to evaluate four distinct allocations of position/force control between human and robot in an abstract blending task. We hypothesize that the function in which humans control the position achieves better performance and receives higher user ratings. When allocating position control to the human and force control to the robot, compared to the opposite case, we observed a significant improvement in preventing overblending. This was also perceived better in terms of physical demand and overall system acceptance, while participants experienced greater autonomy, more engagement and less frustration. An interesting insight was that the supervisory role (when the robot controls both position and force) was rated second best in terms of subjective acceptance. Another surprising insight was that if position control was delegated to the robot, the participants perceived much lower autonomy than when the force control was delegated to the robot. These findings empirically support applying Fitts' principles to static function allocation for physical collaboration, while also revealing important nuanced user experience trade-offs, particularly regarding perceived autonomy when delegating position control.

This paper presents a multi-modal dynamic workspace re-indexing method for addressing operator ergonomics and workspace limitations. The proposed method has two interactive modes: pose-to-pose mode, which is active when the operator is within an ergonomic workspace of comfortable arm postures, and ergonomic workspace drift mode, which activates after the operator makes an excursion beyond the boundaries of the ergonomic workspace when trying to reach more distant targets with the remote robot. In the ergonomic workspace drift mode, the operator temporarily stays slightly outside these boundaries, while the offset between the local and remote workspace drifts with a velocity proportional to the excursion distance. This dynamically re-indexes the remote workspace toward the distant target, and the operator can remain in a comfortable posture while the remote robot moves toward the intended target where the task is. To construct the ergonomic workspace, we employed the Rapid Upper Limb Assessment method. To validate the proposed method, we conducted experiments on a teleoperation setup involving a Force Dimension Sigma7 haptic device controlling a Kuka LBR iiwa robotic arm. The results show that the proposed controller successfully addresses workspace limitations by dynamically reindexing the follower's workspace towards target objects, while maintaining good operator ergonomics.

Rhizome 1.0 and 2.0 are European Space Agency (ESA) co-funded projects that have been implemented with a team from the Architecture, Mechanical Engineering, and Aerospace Engineering Faculties, TU Delft, and various industrial partners. The focus is on the development of a Martian habitat using 3D-printed components and in situ resource utilization. This paper presents a new multi-modal method developed for the collaborative assembly of building components with the support of Computer Vision (CV) and Human-Robot Interaction (HRI) using compliant robotic collaborative manipulators. The building components are Voronoi-based and are fabricated using Design-to-Robotic-Production and -Assembly (D2RP&A). During the collaborative assembly, the robot uses CV to detect the fabricated component and generate autonomous actions to perform the pick-and-place movement. Between the autonomous robot actions, HRI is used by the human to physically guide the robot when grasping and positioning. To evaluate the proposed method, lab experiments were conducted using robotically milled mock-up components from Styrofoam, which were assembled with a collaborative robotic arm. The results indicate that robots can assist humans during the assembly process to implement tasks that are beyond their physical abilities, while robots benefit from human cognitive capabilities during more complex actions.

In this paper, we present a design and evaluation of a novel finger-operated teleimpedance interface used to command stiffness ellipsoids to the remote robot. The proposed interface provides a practical alternative to the state-of-the-art teleimpedance interfaces based on physiological signals that can be impractical in daily use. On the other hand, as opposed to existing practical interfaces that lack in terms of controlled degrees of freedom, the proposed interface enables control of 3D aspects of the ellipsoid. The remote robot stiffness ellipsoid is controlled with a single hand using the thumb, index, and middle fingers to operate two scroll wheels, a joystick, and a force sensor. These combinations of inputs can be mapped to control different aspects of the stiffness ellipsoid, i.e., orientation and shape/size. To investigate different modes of input mapping, we perform a human factors experiment to evaluate the performance and user acceptance of the proposed interface modes. The results of the experiments indicate that the participants can successfully operate the interface to complete 3D stiffness configuration alignment tasks in different modes. To further demonstrate the functionality of the proposed teleimpedance interface, we performed an additional experiment utilising a Force Dimension Sigma7 haptic device to control the motion of a KUKA LBR iiwa robotic arm while performing a complex physical interaction task.

During the learning of a new sensorimotor task, individuals are usually provided with instructional stimuli and relevant information about the target task. The inclusion of haptic devices in the study of this kind of learning has greatly helped in the understanding of how an individual can improve or acquire new skills. However, the way in which the information and stimuli are delivered has not been extensively explored. We have designed a challenging task with nonintuitive visuomotor perturbation that allows us to apply and compare different motor strategies to study the teaching process and to avoid the interference of previous knowledge present in the naïve subjects. Three subject groups participated in our experiment, where the learning by repetition without assistance, learning by repetition with assistance, and task Segmentation Learning techniques were performed with a haptic robot. Our results show that all the groups were able to successfully complete the task and that the subjects’ performance during training and evaluation was not affected by modifying the teaching strategy. Nevertheless, our results indicate that the presented task design is useful for the study of sensorimotor teaching and that the presented metrics are suitable for exploring the evolution of the accuracy and precision during learning.

In order for off-Earth top surface structures built from regolith to protect astronauts from radiation, they need to be several metres thick. In a feasibility study, funded by the European Space Agency, Technical University Delft (TUD aka TU Delft) explored the possibility of building in empty lava tubes to create rhizomatic subsurface habitats. With this approach natural protection from radiation is achieved as well as thermal insulation because the temperature is more stable underground. It involves a swarm of autonomous mobile robots that survey the areas and mine for materials such as regolith in order to create cement-based concrete reproducible on Mars through in-situ resource utilisation (ISRU). The concrete is 3D printed by means of additive Design-to-Robotic-Production (D2RP) methods developed at TUD for on-Earth applications with the 3D printing system of industrial partner, Vertico. The printed components are assembled using a Human--Robot Interaction (HRI) supported approach. The 3D printed and HRI-supported assembled structures are structurally optimised porous material systems with increased insulation properties. In order to regulate the indoor pressurised environment a Life Support System (LSS) is integrated, which in this study is only conceptually developed. The habitat and the D2RP production system are powered by an automated kite power system and solar panels developed at TUD. The long-term goal is to develop an autarkic, automated and HRI-supported D2RP system for building autarkic habitats from locally obtained materials.

Real-world applications of Artificial Intelligence (AI) in architecture have been explored more recently at Technical University (TU) Delft by integrating AI in Design-to-Robotic-Production-Assembly and -Operation (D2RPA&O) methods. These embed robotics into building processes and buildings by linking computational design with robotic construction and/ or operation of building components and buildings. This paper presents two case studies in which AI-supported D2RA is implemented in a multidisciplinary approach that requires the integration of research domains such as architecture, robotics, computer and material science.

Performing bimanual tasks with dual robotic setups can drastically increase the impact on industrial and daily life applications. However, performing a bimanual task brings many challenges, such as synchronization and coordination of the single-arm policies. This article proposes the safe, interactive movement primitives learning (SIMPLe) algorithm, to teach and correct single or dual arm impedance policies directly from human kinesthetic demonstrations. Moreover, it proposes a novel graph encoding of the policy based on Gaussian process regression where the single-arm motion is guaranteed to converge close to the trajectory and then toward the demonstrated goal. Regulation of the robot stiffness according to the epistemic uncertainty of the policy allows for easily reshaping the motion with human feedback and/or adapting to external perturbations. We tested the SIMPLe algorithm on a real dual-arm setup where the teacher gave separate single-arm demonstrations and then successfully synchronized them only using kinesthetic feedback or where the original bimanual demonstration was locally reshaped to pick a box at a different height.

Despite a large body of research on robot learning, it has not yet been thoroughly studied how collaborating humans and robots learn reciprocally. In such situations, both humans and robots continuously learn about each other and the task through interaction. This letter addresses the research question: "How can human-robot co-learning be facilitated in physically embodied collaborative tasks?". First, we derived five requirements for successful human-robot co-learning from literature: shared goal, synchrony, interdependence, adaptability, and transparency. Based on these requirements, we designed a collaborative human-robot handover task and a robot Q-learning method. In an evaluation with six human participants co-learning was indeed found to emerge in the hand-over task. Particularly, for three of the human-robot dyads, our designed setup proved to facilitate co-learning in a way that met all five requirements. The task and robot learning method presented in this letter demonstrate how human-robot co-learning can be enabled in physically embodied tasks.

This paper presents a method for semiautonomous teleimpedance where the control is shared between the human operator and the robot. The human commands the position of the teleoperated robotic arm end-effector while the robot autonomously adjusts the impedance depending on the object with which the end-effector interacts. We developed a vision system that calculates the appropriate robot stiffness based on the detected object geometry and material and object’s relation to the environment. This system uses an RGB-D camera near the robot’s end-effector to capture different perspectives of the scene. To validate the proposed method, we conducted experiments on a teleoperation system where a Force Dimension Sigma7 haptic device was used to operate a KUKA LBR iiwa robotics arm. At the same time, the Intel RealSense D455 depth camera provided the visual input. We examined two practical tasks: engaging with bolts on a plate and polishing a stripe.

Teleoperation is a crucial technology enabling human operators to control robots remotely to perform tasks in hazardous and difficult-to-access environments. Tasks in such environments often involve complex physical interactions with tools and objects of various softness. To this end, teleimpedance enables the operators to adjust the robot impedance in real-time to simplify such interactions. While the existing teleimpedance approaches provide several interfaces to command the robot impedance, there are no interfaces to visualize both the commanded impedance and that of the objects to be interacted with. This paper presents a novel interface to provide visual feedback on the impedance of remote robots and objects. To do so, we use virtual stiffness ellipsoids and different modes that display the individual impedance of the robot and objects as well as combined post-contact impedance. The key advantage of visual feedback on the impedance compared to force feedback is that the operator can see the interaction characteristics before the contact occurs. This enables the operator to act proactively before contact rather than just reactively after the contact. This paper also proposes a new intuitive way to command the robot impedance using mixed reality, interacting with these ellipsoids and modifying them as needed. To demonstrate the key functionalities of the developed interface, we performed proof-of-concept experiments on teleoperated tasks.

Many daily tasks exhibit a periodic nature, necessitating that robots possess the ability to execute them either alone or in collaboration with humans. A widely used approach to encode and learn such periodic patterns from human demonstrations is through periodic Dynamic Movement Primitives (DMPs). Periodic DMPs encode cyclic data independently across multiple dimensions of multi-degree of freedom systems. This method is effective for simple data, like Cartesian or joint position trajectories. However, it cannot account for various geometric constraints imposed by more complex data, such as orientation and stiffness. To bridge this gap, we propose a novel periodic DMP formulation that enables the encoding of periodic orientation trajectories and varying stiffness matrices while considering their geometric constraints. Our geometry-aware approach exploits the properties of the Riemannian manifold and Lie group to directly encode such periodic data while respecting its inherent geometric constraints. We initially employed simulation to validate the technical aspects of the proposed method thoroughly. Subsequently, we conducted experiments with two different real-world robots performing daily tasks involving periodic changes in orientation and/or stiffness, i.e., operating a drilling machine using a rotary handle and facilitating collaborative human–robot sawing.

In this work, we propose a method of capturing the patient’s discomfort during robotic shoulder physiotherapy, creating "discomfort maps". These maps depict the personalized distribution of discomfort that each patient perceived across their shoulder range of motion, facilitating both robotic devices and human therapists to account for patient-specific characteristics during the therapeutic process. Our system enables a patient to communicate and map discomfort in space and time during movement via a handheld push-button device, while interacting with a robotic physical therapy device capable of moving the patient and estimating their pose. We validated our method through human factors experiments simulating shoulder physiotherapy sessions with 10 healthy participants. To avoid the risk of injury to the participants and to allow for ground truth map information, we emulate perceived discomfort via an auditory signal. Our experimental apparatus enabled participants to reconstruct synthetic discomfort maps, demonstrating the feasibility of automatically capturing and storing patient discomfort during robotic physiotherapy.