Camera-based face and infrared temperature sensing of learner's affective state in the remote learning context using machine learning

Camera-based face and infrared temperature sensing of learner's affective state in the remote learning context using machine learning

Den Toonder, Jurriaan (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Software Technology)

Lee, Y. (mentor)
Specht, M.M. (graduation committee)
Migut, M.A. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2021-07-01

Remote learning, learning from home using online available materials, is becoming increasingly more common. This paper focuses on reading activities during remote learning. An important part of learning is keeping sustained attention on the learning materials, as a shift from sustained attention to internal thought or mind-wandering oftentimes decreases the learning performance. Yet this shift is very common during daily activities. It is therefore important for a remote E-learning system to be aware of these attention shifts and allow it to intervene when necessary. Prior research indicates that facial emotion detection and participant's body temperature are generally missing as data modalities. Especially facial emotion recognition may contribute to this area of research, as facial emotions may indicate a learner's affective state during the learning process, and detection of this can be done using a simple consumer laptop web camera. Therefore this paper aims to investigate the feasibility of using a standard laptop web camera to detect facial emotions and a cheap Infrared Temperature Sensor (IR-sensor) to detect the loss of sustained attention. Two experiments were conducted to gather attentive and inattentive data. From this data, features were extracted which were used in multiple machine learning models. The created Machine Learning (ML) models worked well on synthetic test and validation data, but they performed poorly in practice. Our main hypothesis for this is overfitting of the ML models on the data, as due to the Coronavirus Disease (COVID-19) no more than three participants partook in the user studies. Overall, the conclusion is that both emotional facial detection and participant body temperature show great potential to detect sustained attention, but further research needs to be done with a larger group of participants to confirm this.

sustained attention
multimodal learning analytics
machine learning

http://resolver.tudelft.nl/uuid:3d68fb3a-9ea1-4916-9192-23d2fbb996c3

https://github.com/MultimodalLearningAnalytics Software and datasets

Student theses

bachelor thesis

© 2021 Jurriaan Den Toonder