Natural User Interface in Augmented Reality to Control Spot: A Large Scale User Study on Speech and Gesture Control of Robots With The Microsoft HoloLens

The increasing presence of robots calls for a more seamless and information-rich communication method between humans and robots. This paper explores how natural user interface (NUI) modalities, particularly speech and gesture controls, can be used through augmented reality (AR) to operate robots. The increasing presence of robots calls for...

A TinyML system for gesture detection using 3D pre-processed data

Visible light sensing is a field of research that creates new possibilities for human-computer interaction. This research shows the viability of designing a system for detecting hand gestures using a cost-effective detection circuit employing 3 light-sensitive photodiodes. The way this research shows viability is by developing a machine-learning...

Towards a low-cost air-written character recognition system: Designing an embedded machine learning system to recognise the first 10 letters of the Latin alphabet

This study introduces a novel system that leverages three photodiodes and ambient light to identify air-written characters on a resource-constrained device. Through experimentation, suitable methods of data preprocessing, machine learning and model compression were selected to recognize the first 10 characters of the Latin alphabet. The final...

µLightDigit: A TinyML System for Contactless Digit Recognition using Ambient Light

This thesis describes the design and implementation of µLightDigit, which is the second iteration of the LightDigit project: a contactless air-writing system based on ambient light detection with embedded deep learning using only simple photodiodes. The system is able to classify digits 0–9 written in the air by detecting the dynamic hand shadow...

Towards Natural Language Understanding using Multimodal Deep Learning

This thesis describes how multimodal sensor data from a 3D sensor and microphone array can be processed with deep neural networks such that its fusion, the trained neural network, is a) more robust to noise, b) outperforms unimodal recognition and c) enhances unimodal recognition in absence of multimodal data. We built a framework for a complete...

Real-Time Gesture Recognition with a 2D camera

There has been a vast improvement in Human-Computer Interaction over the last decade. Yet there are only a very few systems with natural interfaces such as with speech and gestures. This thesis here addresses the topic of gesture recognition using a 2D camera and how they can be used as natural interfaces to control applications. The gesture...