Natural User Interface in Augmented Reality to Control Spot

A Large Scale User Study on Speech and Gesture Control of Robots With The Microsoft HoloLens

Master thesis (2024)

Authors

J. van der Linden Mechanical Engineering

Contributors

Y.B. Eisma Human-Robot Interaction - Mechanical, Maritime and Materials Engineering (supervisor 1)
Faculty
Mechanical Engineering
Copyright: © 2024 Jesse van der Linden
More Info
expand_more

Published Date

26-02-2024

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Mechanical Engineering

Abstract

The increasing presence of robots calls for a more seamless and information-rich communication method between humans and robots. This paper explores how natural user interface (NUI) modalities, particularly speech and gesture controls, can be used through augmented reality (AR) to operate robots. The increasing presence of robots calls for proper evaluation methods of how to use AR for operating mobile robots. The study uses the Microsoft HoloLens and the robot, named Spot, from Boston Dynamics as primary technologies. The research consists of a user study consisting of 218 participants, one of the largest participant pools for this field to date. The experiment consists of walking the robot over a trajectory with discrete steps, with the perspective of following the robot or standing on a predetermined stationary point. To support the control of the robot, visual information and feedback are included in the HoloLens. Speech control showed the best time performance of the experiment, regardless of the perspective condition. Conversely, errors made during the trials were the majority for the speech condition, due to the waiting time of the speech recognition that caused participants to repeat the commands. The walking condition gave participants the impression that control commands were more intuitively mapped to the robot's motion. Overall, the participants preferred the speech control method while walking with the robot, and the least preferred method was using gestures in a stationary perspective. Even though the speech was the preferred control method and perspective-taking was preferred by participants, this was only for the experiment and task discussed in this paper. Both control methods have different characteristics that make them favorable to be used for specific tasks. Speech and gestures can be used for different tasks when operating a robot with Augmented Reality glasses; preference will depend on the task at hand and the control method design.