ScreenSense: Utilizing Communication Signals for Dynamic Finger Tracking for On-Screen Antennas

Abstract

Future 6G smartphones are proposed to embed transparent on-screen antenna arrays that use communication signals for passive finger tracking. Our research proposes two novel localisation methods that exploit the finger's electromagnetic backscattering response. Using model-generated time-series data, we simulate the spatiotemporal backscattering of a finger hovering above a transparent planar array at sub-terahertz frequencies. We compare a classical matched filter and subspace methods against our proposed approaches: a CNN-adapted matched filter (MF-CNN) and a multi-tone CNN position regressor (MT-CNN), alongside a near-field subspace baseline. The learned methods achieve sub-millimeter accuracy and remain robust\newline across variations in signal-to-noise ratio, array size, dielectric properties, and hover height, with MT-CNN offering the best trade-off between accuracy and latency.