Demo Abstract: Catch My Eye

Gaze-Based Activity Recognition in an Augmented Reality Art Gallery

Conference paper (2022)

Authors

Tim Scargill Duke University
G. Lan Embedded Systems - EEMCS
Maria Gorlatova Duke University
Research Group
Embedded Systems (EEMCS) (TU Delft)
Copyright: © 2022 Tim Scargill, G. Lan, Maria Gorlatova
DOI
help_outline
https://doi.org/10.1109/IPSN54338.2022.00052
Eye tracking Augmented reality Edge computing Activity recognition Augmented reality art gallery
More Info
expand_more

Published Date

2022

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Software Technology

Research Group

Embedded Systems

Abstract

The personalization of augmented reality (AR) experiences based on environmental and user context is key to unlocking their full potential. The recent addition of eye tracking to AR headsets provides a convenient method for detecting user context, but complex analysis of raw gaze data is required to detect where a user's attention and thoughts truly lie. In this demo we present Catch My Eye, the first system to incorporate deep neural network (DNN)-based activity recognition from user gaze into a realistic mobile AR app. We develop an edge computing-based architecture to offload context computation from resource-constrained AR devices, and present a working example of content adaptation based on user context, for the scenario of a virtual art gallery. It shows that user activities can be accurately recognized and employed with sufficiently low latency for practical AR applications.