Natural User Interface in Augmented Reality to Control Spot: A Large Scale User Study on Speech and Gesture Control of Robots With The Microsoft HoloLens

The increasing presence of robots calls for a more seamless and information-rich communication method between humans and robots. This paper explores how natural user interface (NUI) modalities, particularly speech and gesture controls, can be used through augmented reality (AR) to operate robots. The increasing presence of robots calls for...

WaveTune: Millimeter Wave Radar based Hand Gesture Recognition for Musical Applications

In the ever-evolving field of music technology, new solutions continue to emerge that enhance musical expression and creativity. This thesis introduces WaveTune, a novel lightweight hand gesture recognition system that enables real-time control of musical composition and performance through natural hand motions.<br/><br/>WaveTune utilizes...

Low-Power Gesture Recognition Using Convolutional Neural Networks and Ambient Lighting

This paper presents a study focused on developing an efficient signal processing pipeline and identifying suitable machine learning models for real-time gesture recognition using a testbed consisting of an Arduino Nano 33 BLE and three OPT101 photodiodes. Our research aims to address the challenges of limited computational power whilst...

A TinyML system for gesture detection using 3D pre-processed data

Visible light sensing is a field of research that creates new possibilities for human-computer interaction. This research shows the viability of designing a system for detecting hand gestures using a cost-effective detection circuit employing 3 light-sensitive photodiodes. The way this research shows viability is by developing a machine-learning...

Towards a low-cost air-written character recognition system: Designing an embedded machine learning system to recognise the first 10 letters of the Latin alphabet

This study introduces a novel system that leverages three photodiodes and ambient light to identify air-written characters on a resource-constrained device. Through experimentation, suitable methods of data preprocessing, machine learning and model compression were selected to recognize the first 10 characters of the Latin alphabet. The final...

µLightDigit: A TinyML System for Contactless Digit Recognition using Ambient Light

This thesis describes the design and implementation of µLightDigit, which is the second iteration of the LightDigit project: a contactless air-writing system based on ambient light detection with embedded deep learning using only simple photodiodes. The system is able to classify digits 0–9 written in the air by detecting the dynamic hand shadow...

Recognising Gestures Using Ambient Light and Convolutional Neural Networks: Adapting Convolutional Neural Networks for Gesture Recognition on Resource-constrained Microcontrollers

This paper presents how a convolutional neural network can be constructed in order to recognise gestures using photodiodes and ambient light. A number of candidates are presented and evaluated, with the most performant being adopted for in-depth analysis. This network is then compressed in order to be ran on an Arduino Nano 33 BLE...

Designing an adaptable and low-cost system for gesture recognition using visible light

This paper describes the design of an adaptable, low-cost, and energy efficient gesture detection system. The system leverages the ambient light available in the environment to perform Visible Light Sensing (VLS) using powerful Convolutional Neural Networks. The focus lies on designing a system that is capable of robust gesture detection in any...

Hand Gesture Recognition on Arduino Using Recurrent Neural Networks and Ambient Light

Touching physical buttons to interact with public electronic devices has raised some concerns regrading disease transmission following the COVID-19 pandemic. The use of hand gestures as a touchless replacement sounds appealing, but comes with the challenge of recognizing which gesture is being performed by the user, with only the processing...

Constructing A Dataset For Gesture Recognition Using Ambient Light

Public technology has been shown to have a strong dependence on physical touch, which increases the transmission of diseases. Gesture recognition helps to reduce this transmission, as the dependence on physical touch is removed. Furthermore, the use of visible light for gesture recognition would reduce the power consumption of public technology,...

Designing a Software Receiver for Gesture Recognition with Ambient Light

There is a growing need for touch-free interaction with public utilities such as coffeemakers and vending machines that will help prevent the spread of diseases such as COVID-19. One solution is the integration of embedded gesture recognition systems relying on ambient light. However, existing work so far is found to be inefficient in terms of...

Traffic Gesture Classification for Intelligent Vehicles

Self-driving vehicles have shown rapid development in recent years and continue to move towards full autonomy. For high or full automation, self-driving vehicles will have to be able to address and solve a broad range of situations, one of which is interaction with traffic agents. For correct and save maneuvering through these situations,...

Towards Natural Language Understanding using Multimodal Deep Learning

This thesis describes how multimodal sensor data from a 3D sensor and microphone array can be processed with deep neural networks such that its fusion, the trained neural network, is a) more robust to noise, b) outperforms unimodal recognition and c) enhances unimodal recognition in absence of multimodal data. We built a framework for a complete...

Micro-Activity Recognition using Wearables for Human Augmentation

Wearable devices have paved way for several context-aware applications in the field of health-care and sports to improve the well-being of users and their performance for human augmentation. During rehabilitation patients need accurate feedback that can empower and improve the speed of recovery. On the other hand competitive athletes need a...

Handshake Recognition applied to Wireless Data Exchange in Smartbands

The way people meet each other is usually face to face. Meanwhile, the way people maintain their contacts is mostly through social media. This results in a gap of translating a handshake into a digital connection. Shake-On is a start-up that has come up with an answer by means of a smart bracelet. Their aim is to wirelessly exchange contact...

Real-Time Gesture Recognition with a 2D camera

There has been a vast improvement in Human-Computer Interaction over the last decade. Yet there are only a very few systems with natural interfaces such as with speech and gestures. This thesis here addresses the topic of gesture recognition using a 2D camera and how they can be used as natural interfaces to control applications. The gesture...