Automatic Detection of Mind-Wandering using Facial Expressions

Bachelor thesis (2022)

Authors

R. Kargul Electrical Engineering, Mathematics and Computer Science

Contributors

B.J.W. Dudzik Pattern Recognition and Bioinformatics - EEMCS (mentor)
X. Zhang Pattern Recognition and Bioinformatics - EEMCS (mentor)
H.S. Hung Pattern Recognition and Bioinformatics - EEMCS (mentor)
P.K. Murukannaiah Interactive Intelligence - EEMCS (graduation committee member)
Faculty
Electrical Engineering, Mathematics and Computer Science, Electrical Engineering, Mathematics and Computer Science
Copyright: © 2022 Radek Kargul
More Info
expand_more

Published Date

22-06-2022

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Abstract

Spending time in front of screens has become an inescapable activity, which might be interrupted by unrelated external causes. While automatic approaches to identify mind-wandering (MW) have already been investigated, past research was done with self-reports or physiological data. This work explores automated detection utilizing solely facial expressions from Mementos data, which comes in the form of webcam recordings, where participants react to music videos. The recordings are annotated with labels indicating perceived MW. Video responses are turned into time series by first extracting facial characteristics, which are encoded with Facial Action Coding System (FACS). Temporal information is represented with 170 temporal features. Classification is conducted with support vector machines (SVM) through a data-level approach and an algorithm-level approach, first by synthesizing data and second by adding class weights to SVM. Both approaches are evaluated with metric scores insensitive to imbalanced data. On average,
results show that detection performs marginally better than by chance. However, the evaluation metric values vary across multiple classification runs, thus the prospect of using the Mementos dataset for automatic MW detection based on only facial expressions is not promising.