Mixed Reality For Preserving Craftmanship Experiences in Human Robot Interaction

Abstract

This project investigates how Mixed Reality (MR) can enhance the experience of Human-Robot Interaction (HRI) for crafting tasks, aiming to preserve the positive aspects of user experience and feeling of craftsmanship.
Initiated with an interest in the growth of MR and robotic automation, the research examines how MR might bridge the gap between user satisfaction and the increasing robot implementation in automating tasks execution. While automation streamlines repetitive or labour-intensive tasks, it often removes aspects that contribute to user engagement and job satisfaction. MR, as an immersive and interactive tool, offers a promising solution to enabling users to control and collaborate with robots in a more intuitive and meaningful way.
The project is built around the research question: “How can a unilateral vision-based control system, implemented through a Mixed Reality headset, enable telemanipulation of a robotic arm for crafting purposes, without disrupting the user’s experience and performance?”
The study defines MR’s unique potential for HRI by leveraging vision-based hand-tracking to control a robotic arm, specifically in a crafting task. A prototype was developed to explore the interaction dynamics, using, due to technical constraints, virtual reality to simulate the MR interaction and the robot The primary goal was to design an interaction system that offers full control of spatial navigation and force application, allowing the direct and real-time adjustments essential to a crafting task.
The iterative design process involved testing multiple prototypes with users, identifying challenges and the experience in relation to craftsmanship. Key issues identified included difficulty in manipulating the robot precisely, due to the absence of haptic feedback, limited visual depth awareness, and a reliance on visual-only feedback, which, while informative, have a low robustness and clutters the user interface. These findings lead to the design of three revised prototypes with various constraint levels, one with full user autonomy, one with low constraints and a one highly constrained, to test the impact on user experience and task effectiveness and find the right balance between user control and ease of manipulation.
The results indicate that the low-constraint system provided the most positive user experience, finding the right balance between user control and guidance while maintaining the essential qualities of craftsmanship. Although MR offers a viable framework for enhancing user experience in HRI, the research highlights limitations in replicating the experience of real-world crafting. The study concludes by recommending that future MR applications should leverage MR’s unique qualities, rather than simply mimicking traditional crafting qualities, to foster novel, user-centred interactions.