A TinyML system for gesture detection using 3D pre-processed data
S.A.J. van den Broek (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Qing Wang – Mentor (TU Delft - Embedded Systems)
Mingkun Yang – Mentor (TU Delft - Embedded Systems)
Ran Zhu – Mentor (TU Delft - Embedded Systems)
R Venkatesha Venkatesha Prasad – Graduation committee member (TU Delft - Networked Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Visible light sensing is a field of research that creates new possibilities for human-computer interaction. This research shows the viability of designing a system for detecting hand gestures using a cost-effective detection circuit employing 3 light-sensitive photodiodes. The way this research shows viability is by developing a machine-learning model that works on 3D-structured sensor data that is able to distinguish 10 different gestures and deploying the model on a standalone Arduino Nano 33 BLE microcontroller controlling the system. Using a combination of Convolutional Neural Networks and Recurrent Neural Networks it is possible to deploy a model called ConvLSTM-128 that achieves an accuracy of 70% on a dataset of limited size. This research acknowledges that the achieved accuracy is not suitable for real-world use, but concludes by outlining steps that could help future research in increasing the accuracy. Furthermore, an analysis of the 10 gestures shows that in order to improve accuracy, the way some gestures are performed might need alteration. Finally, a model size of around 140Kb and an inference time of 660ms show that this model is compact and fast enough to be deployed in real-world applications.