Real-time Face and Landmark Localization for Eyeblink-Response Detection

Abstract

Pavlovian eyeblink conditioning is a powerful experiment used in the field of neuroscience to measure multiple aspects of how we learn in our daily life. To track the movement of the eyelid during an experiment, researchers traditionally made use of potentiometers or electromyography (EMG). More recently, the use of computer vision and image processing alleviated the need for these techniques, but currently employed methods require human intervention and are not fast enough to enable real-time processing. We selected a combination of face and landmark-detection algorithms in order to fully automate eyelid tracking, and accelerated them to make the first step towards an on-line implementation. Various different algorithms for face detection and landmark detection (eyelid detection) are analyzed and evaluated. Based on this analysis, two algorithms are identified as most suitable for our use case: the Histogram of Oriented Gradients (HOG) algorithm for face detection and Ensemble of Regression Trees (ERT) algorithm for landmark detection. These two algorithms are accelerated on GPU and CPU, achieving speedups of 1753× and 11.49×, respectively. A combination of these algorithms is successfully implemented for a real neuroscientific use-case: eyeblink response detection, achieving an overall application runtime of 0.533 ms per frame, which is 1067× faster than the sequential implementation. Furthermore, this accelerated implementation was used to generate a database of 1440 eyeblink responses during the conditioning experiment. We made use of multiple machine-learning techniques to analyze this database and concluded that there is a correlation between the asynchronicity of the eyelids and the eyeblink-response performance during the conditioning experiment.