Technology-enhanced learning systems, specifically multimodal learning technologies, use sensors to collect data from multiple modalities to provide personalized learning support beyond traditional learning settings. However, many studies surrounding such multimodal learning systems mostly focus on technical aspects concerning data collection and exploitation and therefore overlook theoretical and instructional design aspects such as feedback design in multimodal settings. This paper explores multimodal learning systems as a critical part of technology-enhanced learning used for capturing and analyzing the learning process to exploit the collected multimodal data to generate feedback in multimodal settings. By investigating various studies, we aim to reveal the roles of multimodality in technology-enhanced learning across various learning domains. Our scoping review outlines the conceptual landscape of multimodal learning systems, identifies potential gaps, and provides new perspectives on adaptive multimodal system design: intertwining learning data for meaningful insights into learning, designing effective feedback, and implementing them in diverse learning domains.

Beginner table-tennis players require constant real-time feedback while learning the funda-mental techniques. However, due to various constraints such as the mentor’s inability to be around all the time, expensive sensors and equipment for sports training, beginners are unable to get the immediate real-time feedback they need during training. Sensors have been widely used to train beginners and novices for various skills development, including psychomotor skills. Sensors enable the collection of multimodal data which can be utilised with machine learning to classify training mistakes, give feedback, and further improve the learning outcomes. In this paper, we introduce the Table Tennis Tutor (T3), a multi-sensor system consisting of a smartphone device with its built-in sensors for collecting motion data and a Microsoft Kinect for tracking body position. We focused on the forehand stroke mistake detection. We collected a dataset recording an experienced table tennis player performing 260 short forehand strokes (correct) and mimicking 250 long forehand strokes (mistake). We analysed and annotated the multimodal data for training a recurrent neural network that classifies correct and incorrect strokes. To investigate the accuracy level of the afore-mentioned sensors, three combinations were validated in this study: smartphone sensors only, the Kinect only, and both devices combined. The results of the study show that smartphone sensors alone perform sub-par than the Kinect, but similar with better precision together with the Kinect. To further strengthen T3’s potential for training, an expert interview session was held virtually with a table tennis coach to investigate the coach’s perception of having a real-time feedback system to assist beginners during training sessions. The outcome of the interview shows positive expectations and provided more inputs that can be beneficial for the future implementations of the T3.

Learning fine psychomotor skills such as handwriting is a tedious endeavour which requires close supervision of the teacher to master. However, the increasing number of students in classes means less time a teacher can allocate for each student. This adversely affects the development of handwriting in students. Sensor-based technologies can help address this problem, as they are capable of providing feedback to the student whilst the teacher is not present during the student's writing. While there are multiple sensor-based applications to date for handwriting practice, such applications provide feedback in only for simple tracing over practice tasks. In this paper, we present a conceptual methodology using AI and sensors, for providing feedback in non-tracking tasks that do not have a single correct solution and allow larger variations.

Traditional online communications tools used in education are limited in terms of fostering naturalistic or life-like interaction. Such limited interactions in classrooms can negatively impact learning. Holograms are promising tools that show potential to overcome such limitations by affording more life-like interactions in virtual classrooms. In this paper, we introduce the prototype built within the context of the project,”HoloLearn”, which is currently ongoing and aims to foster lifelike interactions between teachers and students. Furthermore, we discuss the limitations of the current prototype and also the steps that need to be undertaken in the future.

Gamification aims at addressing problems in various fields such as the high dropout rates, the lack of engagement, isolation, or the lack of personalisation faced by Massive Open Online Courses (MOOC). Even though gamification is widely applied, not only in MOOCs, only few cases are meaningfully designed and empirically tested. The Gamification Design Process (GaDeP) aims to cover this gap. This article first briefly introduces GaDeP, presents the concept of meaningful gamification, and derives how it motivates the need for the Gamifire platform (as a scalable and platform-independent reference infrastructure for MOOC). Secondly, it defines the requirements for platformindependent gamification and describes the development of the Gamifire infrastructure. Thirdly we describe how Gamifire was successfully applied in four different cases. Finally, the applicability of GaDeP beyond MOOC is presented by reporting on a case study where GaDeP has been successfully applied by four student research and development projects. From both, the Gamifire cases and the GaDeP cases we derive the key contribution of this article: insights in the strengths and weaknesses of the Gamifire infrastructure as well as lessons learned about the applicability and limitations of the GaDeP framework. The paper ends detailing our future works and planned development activities.

Sensors can monitor physical attributes and record multimodal data in order to provide feedback. The application calligraphy trainer, exploits these affordances in the context of handwriting learning. It records the expert’s handwriting performance to compute an expert model. The application then uses the expert model to provide guidance and feedback to the learners. However, new learners can be overwhelmed by the feedback as handwriting learning is a tedious task. This paper presents the pilot study done with the calligraphy trainer to evaluate the mental effort induced by various types of feedback provided by the application. Ten participants, five in the control group and five in the treatment group, who were Ph.D. students in the technology-enhanced learning domain, took part in the study. The participants used the application to learn three characters from the Devanagari script. The results show higher mental effort in the treatment group when all types of feedback are provided simultaneously. The mental efforts for individual feedback were similar to the control group. In conclusion, the feedback provided by the calligraphy trainer does not impose high mental effort and, therefore, the design considerations of the calligraphy trainer can be insightful for multimodal feedback designers.

Experts are imperative for training apprentices, but learning from experts is difficult. Experts often struggle to explicate and/or verbalize their knowledge or simply overlook important details due to internalization of their skills, which may make it more difficult for apprentices to learn from experts. In addition, the shortage of experts to support apprentices in one-to-one settings during trainings limits the development of apprentices. In this review, we investigate how augmented reality and sensor technology can be used to capture expert performance in such a way that the captured performance can be used to train apprentices without increasing the workload on experts. To this end, we have analysed 78 studies that have implemented augmented reality and sensor technology for training purposes. We explored how sensors have been used to capture expert performance with the intention of supporting apprentice training. Furthermore, we classified the instructional methods used by the studies according to the 4C/ID framework to understand how augmented reality and sensor technology have been used to support training. The results of this review show that augmented reality and sensor technology have the potential to capture expert performance for training purposes. The results also outline a methodological approach to how sensors and augmented reality learning environments can be designed for training using recorded expert performance.

The WEKIT.one prototype is a platform for immersive procedural training with wearable sensors and Augmented Reality. Focusing on capture and re-enactment of human expertise, this work looks at the unique affordances of suitable hard- and software technologies. The practical challenges of interpreting expertise, using suitable sensors for its capture and specifying the means to describe and display to the novice are of central significance here. We link affordances with hardware devices, discussing their alternatives, including Microsoft Hololens, Thalmic Labs MYO, Alex Posture sensor, MyndPlay EEG headband, and a heart rate sensor. Following the selection of sensors, we describe integration and communication requirements for the prototype. We close with thoughts on the wider possibilities for implementation and next steps.