Through-Screen Visible Light Sensing Empowered by Embedded Deep Learning

Liu, Hao; Ye, Hanting; Yang, J.; Wang, Q.

doi:10.1145/3485730.3493454

Through-Screen Visible Light Sensing Empowered by Embedded Deep Learning

Title

Through-Screen Visible Light Sensing Empowered by Embedded Deep Learning

Author

Liu, Hao (Student TU Delft)
Ye, Hanting (TU Delft Embedded Systems)
Yang, J. (TU Delft Web Information Systems)
Wang, Q. (TU Delft Embedded Systems)

Date

2021

Abstract

Motivated by the trend of realizing full screens on devices such as smartphones, in this work we propose through-screen sensing with visible light for the application of fingertip air-writing. The system can recognize handwritten digits with under-screen photodiodes as the receiver. The key idea is to recognize the weak light reflected by the finger when the finger writes the digits on top of a screen. The proposed air-writing system has immunity to scene changes because it has a fixed screen light source. However, the screen is a double-edged sword as both a signal source and a noise source. We propose a data preprocessing method to reduce the interference of the screen as a noise source. We design an embedded deep learning model, a customized model ConvRNN, to model the spatial and temporal patterns in the dynamic and weak reflected signal for air-writing digits recognition. The evaluation results show that our through-screen fingertip air-writing system with visible light can achieve accuracy up to 91%. Results further show that the size of the customized ConvRNN model can be reduced by 94% with less than a 10% drop in performance.

Subject

embedded AI
embedded deep learning
Through-screen sensing

To reference this document use:

http://resolver.tudelft.nl/uuid:7544cd5e-5474-4b64-87ac-c73c504e4d41

DOI

https://doi.org/10.1145/3485730.3493454

Publisher

Association for Computing Machinery (ACM)

ISBN

9781450390972

Source

SenSys 2021 - Proceedings of the 2021 19th ACM Conference on Embedded Networked Sensor Systems