Through-Screen Finger Localization and Tracking using Reflected Light

Abstract

Visible light positioning systems conventionally fix the light sources on a ceiling and let a receiver move through the scene. We invert this geometry by tracking a hovering finger above a transparent segmented OLED display placed above four photodiodes. From the four signals influenced by the reflected light from the finger we track its position. The motivating application is pre-touch sensing on mobile devices, where anticipating the user's next touch during the hover to touch window lets the system pre-load content. The central question is whether four under-screen photodiodes can localize and track a hovering finger in real time using only the microcontroller already driving the screen, avoiding the deep neural networks that prior through-screen sensing required. We collected a reflected light dataset of 199 finger captures across a 10×4 calibration grid and evaluated localization on a 5×2 cell grid. After subtracting a temporally interpolated no finger baseline, we build an 18-dimensional feature vector and classify the cell with a two-stage logistic-regression head that predicts column and row independently. This reaches 77.2% cell accuracy under a random split and 66.8% under a leave-one-calibration-dot-out protocol. The second one is more representative of deployment because every recording of the tested position is withheld from training. The complete pipeline runs on an Arduino Due that also drives the screen with sub-millisecond inference. We conclude that through-screen reflected light carries enough spatial information for cell-level finger localization without deep learning, on the same embedded hardware that runs the display.