µLightDigit: A TinyML System for Contactless Digit Recognition using Ambient Light

This thesis describes the design and implementation of µLightDigit, which is the second iteration of the LightDigit project: a contactless air-writing system based on ambient light detection with embedded deep learning using only simple photodiodes. The system is able to classify digits 0–9 written in the air by detecting the dynamic hand shadow...

Debugging Machine Learning on Time-series Data

Machine learning models are so-called a "black box," which means people can not easily observe the relationship between the output and input or explain the reason for such results. In recent years, much work has been done on interpretable machine-learning, such as Shapley values, counterfactual explanations, partial dependence plots, or saliency...

Recognising Gestures Using Ambient Light and Convolutional Neural Networks: Adapting Convolutional Neural Networks for Gesture Recognition on Resource-constrained Microcontrollers

This paper presents how a convolutional neural network can be constructed in order to recognise gestures using photodiodes and ambient light. A number of candidates are presented and evaluated, with the most performant being adopted for in-depth analysis. This network is then compressed in order to be ran on an Arduino Nano 33 BLE...

Designing a Software Receiver for Gesture Recognition with Ambient Light

There is a growing need for touch-free interaction with public utilities such as coffeemakers and vending machines that will help prevent the spread of diseases such as COVID-19. One solution is the integration of embedded gesture recognition systems relying on ambient light. However, existing work so far is found to be inefficient in terms of...

Designing an adaptable and low-cost system for gesture recognition using visible light

This paper describes the design of an adaptable, low-cost, and energy efficient gesture detection system. The system leverages the ambient light available in the environment to perform Visible Light Sensing (VLS) using powerful Convolutional Neural Networks. The focus lies on designing a system that is capable of robust gesture detection in any...

Hand Gesture Recognition on Arduino Using Recurrent Neural Networks and Ambient Light

Touching physical buttons to interact with public electronic devices has raised some concerns regrading disease transmission following the COVID-19 pandemic. The use of hand gestures as a touchless replacement sounds appealing, but comes with the challenge of recognizing which gesture is being performed by the user, with only the processing...

Constructing A Dataset For Gesture Recognition Using Ambient Light

Public technology has been shown to have a strong dependence on physical touch, which increases the transmission of diseases. Gesture recognition helps to reduce this transmission, as the dependence on physical touch is removed. Furthermore, the use of visible light for gesture recognition would reduce the power consumption of public technology,...

Integrated Communication and Sensing with RGB LEDs

Visible Light Communication (VLC) has gained popularity due to its inherent security as a complementary technology to Radio Frequency (RF) in the last decades to solve the "spectrum crunch'' problem. Meanwhile, The latest IEEE 802.11ah standard, also called WiFi HaLow, offers the range, throughput, and low power consumption that is extremely...

LightDigit: Embedded Deep Learning Empowered Fingertip Air-Writing with Ambient Light

This thesis introduces LightDigit, a recognition system for fingertip air-writing digits that enables low-cost, privacy-preserving, and in air interaction with pub- lic devices such as touchscreens. The current LightDigit prototype can input digits to public devices with a simple array of photodiodes and with ubiquitous ambient light in our...