Camera-based face and infrared temperature sensing of learner's affective state in the remote learning context using machine learning

Abstract

Remote learning, learning from home using online available materials, is becoming increasingly more common. This paper focuses on reading activities during remote learning. An important part of learning is keeping sustained attention on the learning materials, as a shift from sustained attention to internal thought or mind-wandering oftentimes decreases the learning performance. Yet this shift is very common during daily activities. It is therefore important for a remote E-learning system to be aware of these attention shifts and allow it to intervene when necessary. Prior research indicates that facial emotion detection and participant's body temperature are generally missing as data modalities. Especially facial emotion recognition may contribute to this area of research, as facial emotions may indicate a learner's affective state during the learning process, and detection of this can be done using a simple consumer laptop web camera. Therefore this paper aims to investigate the feasibility of using a standard laptop web camera to detect facial emotions and a cheap Infrared Temperature Sensor (IR-sensor) to detect the loss of sustained attention. Two experiments were conducted to gather attentive and inattentive data. From this data, features were extracted which were used in multiple machine learning models. The created Machine Learning (ML) models worked well on synthetic test and validation data, but they performed poorly in practice. Our main hypothesis for this is overfitting of the ML models on the data, as due to the Coronavirus Disease (COVID-19) no more than three participants partook in the user studies. Overall, the conclusion is that both emotional facial detection and participant body temperature show great potential to detect sustained attention, but further research needs to be done with a larger group of participants to confirm this.