Technology-enhanced learning systems, specifically multimodal learning technologies, use sensors to collect data from multiple modalities to provide personalized learning support beyond traditional learning settings. However, many studies surrounding such multimodal learning systems mostly focus on technical aspects concerning data collection and exploitation and therefore overlook theoretical and instructional design aspects such as feedback design in multimodal settings. This paper explores multimodal learning systems as a critical part of technology-enhanced learning used for capturing and analyzing the learning process to exploit the collected multimodal data to generate feedback in multimodal settings. By investigating various studies, we aim to reveal the roles of multimodality in technology-enhanced learning across various learning domains. Our scoping review outlines the conceptual landscape of multimodal learning systems, identifies potential gaps, and provides new perspectives on adaptive multimodal system design: intertwining learning data for meaningful insights into learning, designing effective feedback, and implementing them in diverse learning domains.

This paper presents an approach that helps distinguish expert and novice performance easily by observing the sensor data without having to understand nor apply models to the sensor signal. The method consists of plotting the sensor data and identifying irregularities. We corroborate, with the help of sensors, that expert performances are smoother, contain fewer irregularities, and have consistently uniform patterns than novice performances. In this paper, we present six different cases pointing out this assertion, namely bachata and salsa dances, tennis swings, football penalty kicks, badminton, and running.

While digital education technologies have improved to make educational resources more available, the modes of interaction they implement remain largely unnatural for the learner. Modern sensor-enabled computer systems allow extending human-computer interfaces for multimodal communication. Advances in Artificial Intelligence allow interpreting the data collected from multimodal and multi-sensor devices. These insights can be used to support deliberate practice with personalised feedback and adaptation through Multimodal Learning Experiences (MLX). This chapter elaborates on the approaches, architectures, and methodologies in five different use cases that use multimodal learning analytics applications for deliberate practice.

It takes considerable time, experience, and direct assistance from teachers to become a skilled writer. Handwriting fluency is one of the predictors of writing quality among students. However, students do not receive enough teacher supervision as a beginner to develop handwriting fluency in a proper manner. The ``Calligraphy tutor'' presented in this paper, is an application developed to assist teachers to help students learn proper handwriting fluency skills. Calligraphy tutor is designed to support deliberate practice of handwriting, in which teachers play the central role. To reduce workload of teachers, Calligraphy tutor automates repetitive actions such as providing mundane real-time feedback, while also collecting performance data from students, allowing students to practice without the presence of a teacher. The collected performance data is used by teachers to further personalise students' training.

Acquisition and internalisation of many fundamental skills rely on repeated authentic practice and teachers providing support during practice. Despite this well accepted norms in skills acquisition, much of our assumptions about learning skills, mostly from a cognitive perspective, remain nebulous. Besides splitting hairs to classify skills acquisition into a paradigm, much of findings of related research from educational science and psychology have struggled to transfer into the domain of skills acquisition. Instead, in this paper, we propose to shift our view of skills acquisition from a cognitive approach to an embodied one with the help of multimodal technologies and provide a use-case which combines deliberate practice framework, embodied cognition principles, and multimodal learning.

Learning fine psychomotor skills such as handwriting is a tedious endeavour which requires close supervision of the teacher to master. However, the increasing number of students in classes means less time a teacher can allocate for each student. This adversely affects the development of handwriting in students. Sensor-based technologies can help address this problem, as they are capable of providing feedback to the student whilst the teacher is not present during the student's writing. While there are multiple sensor-based applications to date for handwriting practice, such applications provide feedback in only for simple tracing over practice tasks. In this paper, we present a conceptual methodology using AI and sensors, for providing feedback in non-tracking tasks that do not have a single correct solution and allow larger variations.

Traditional online communications tools used in education are limited in terms of fostering naturalistic or life-like interaction. Such limited interactions in classrooms can negatively impact learning. Holograms are promising tools that show potential to overcome such limitations by affording more life-like interactions in virtual classrooms. In this paper, we introduce the prototype built within the context of the project,”HoloLearn”, which is currently ongoing and aims to foster lifelike interactions between teachers and students. Furthermore, we discuss the limitations of the current prototype and also the steps that need to be undertaken in the future.

Beginner table-tennis players require constant real-time feedback while learning the funda-mental techniques. However, due to various constraints such as the mentor’s inability to be around all the time, expensive sensors and equipment for sports training, beginners are unable to get the immediate real-time feedback they need during training. Sensors have been widely used to train beginners and novices for various skills development, including psychomotor skills. Sensors enable the collection of multimodal data which can be utilised with machine learning to classify training mistakes, give feedback, and further improve the learning outcomes. In this paper, we introduce the Table Tennis Tutor (T3), a multi-sensor system consisting of a smartphone device with its built-in sensors for collecting motion data and a Microsoft Kinect for tracking body position. We focused on the forehand stroke mistake detection. We collected a dataset recording an experienced table tennis player performing 260 short forehand strokes (correct) and mimicking 250 long forehand strokes (mistake). We analysed and annotated the multimodal data for training a recurrent neural network that classifies correct and incorrect strokes. To investigate the accuracy level of the afore-mentioned sensors, three combinations were validated in this study: smartphone sensors only, the Kinect only, and both devices combined. The results of the study show that smartphone sensors alone perform sub-par than the Kinect, but similar with better precision together with the Kinect. To further strengthen T3’s potential for training, an expert interview session was held virtually with a table tennis coach to investigate the coach’s perception of having a real-time feedback system to assist beginners during training sessions. The outcome of the interview shows positive expectations and provided more inputs that can be beneficial for the future implementations of the T3.

Gamification aims at addressing problems in various fields such as the high dropout rates, the lack of engagement, isolation, or the lack of personalisation faced by Massive Open Online Courses (MOOC). Even though gamification is widely applied, not only in MOOCs, only few cases are meaningfully designed and empirically tested. The Gamification Design Process (GaDeP) aims to cover this gap. This article first briefly introduces GaDeP, presents the concept of meaningful gamification, and derives how it motivates the need for the Gamifire platform (as a scalable and platform-independent reference infrastructure for MOOC). Secondly, it defines the requirements for platformindependent gamification and describes the development of the Gamifire infrastructure. Thirdly we describe how Gamifire was successfully applied in four different cases. Finally, the applicability of GaDeP beyond MOOC is presented by reporting on a case study where GaDeP has been successfully applied by four student research and development projects. From both, the Gamifire cases and the GaDeP cases we derive the key contribution of this article: insights in the strengths and weaknesses of the Gamifire infrastructure as well as lessons learned about the applicability and limitations of the GaDeP framework. The paper ends detailing our future works and planned development activities.

Sensors can monitor physical attributes and record multimodal data in order to provide feedback. The application calligraphy trainer, exploits these affordances in the context of handwriting learning. It records the expert’s handwriting performance to compute an expert model. The application then uses the expert model to provide guidance and feedback to the learners. However, new learners can be overwhelmed by the feedback as handwriting learning is a tedious task. This paper presents the pilot study done with the calligraphy trainer to evaluate the mental effort induced by various types of feedback provided by the application. Ten participants, five in the control group and five in the treatment group, who were Ph.D. students in the technology-enhanced learning domain, took part in the study. The participants used the application to learn three characters from the Devanagari script. The results show higher mental effort in the treatment group when all types of feedback are provided simultaneously. The mental efforts for individual feedback were similar to the control group. In conclusion, the feedback provided by the calligraphy trainer does not impose high mental effort and, therefore, the design considerations of the calligraphy trainer can be insightful for multimodal feedback designers.

Experts are imperative for supporting expertise development in apprentices but learning from them is difficult. In many cases, there are shortages of experts to train apprentices. To address this issue, we use wearable sensors and augmented reality to record expert performance for supporting the training of apprentices. In this context, we present the conceptual framework which outlines different instructional design methodologies for training various attributes of a task. These instructional design methodologies are characterized by their dependencies on expert performance and experts as model for training. In addition, they exploit the affordances of modern wearable sensors and augmented reality. The framework also outlines a training workflow based on the 4C/ID model, a pedagogic model for complex learning, which ensures that all aspects of conventional training are considered. The paper concludes with application guidelines and examples along with reflection of the authors.

Body-worn sensors can be used to capture, analyze, and replay human performance for training purposes. The key challenge to any such approach is to establish validity that the captured expert experience is actually suitable for training. In this paper, to evaluate this, we apply a questionnaire-based expert assessment and a complementary trainee knowledge assessment to study the approach adopted and the models generated with the WEKIT solution, a hardware and software application that complements Augmented Reality glasses with wearable sensor-actuator experience. This solution was developed using the ID4AR framework which as also developed within the WEKIT project. ID4AR framework is a domain agnostic framework which can be used to design augmented reality and sensor based applications for training. The study presented triangulates validity across three independent test-beds in the professional domains of aircraft maintenance, medical imaging, and astronaut training, with 61 experts completing the expert survey and 337 students completing the trainee knowledge test. Results show that the captured expert models were positively received in all three domains and the identified level of acceptance suggests that the solution is capable of capturing models for training purposes at large.

The chapter highlights the role of sensors for supporting seamless learning experiences. In the first part, the relation between sensor tracking of learning activities and research around real-time feedback in educational situations is introduced. The authors present an overview of the kinds of sensor data that have been used for educational purposes in the literature. Secondly, the authors introduce the link between sensor data and educational interventions, and especially the role of building expert models from real-world expert tracking. The third part of the paper illustrates how educational AR applications have used sensor data for different forms of learning support. The authors present 15 design patterns that have been implemented in different educational AR applications that build on our analysis of sensor tracking. For future AR applications, the authors propose that the use of sensors for building expert performance models is essential for a variety of educational interventions.

Augmented reality and sensor technologies have been analysed extensively in several domains including education and training. Although, varieties of use cases and applications exist, these studies were conducted in controlled laboratory environments. This paper reports on the first user study of augmented reality prototype developed to support students to learn from trainers in professional domains using augmented reality and sensors. The prototype records the performance of trainers in the first phase to support students by making it available during practice in the second phase. The performance data is made available to both the students and trainers in the third phase for reflection. A total of 142 participants which included trainers and students from three professional domains, namely 1) aircraft maintenance 2) medical imaging and 3) astronaut training, evaluated the prototype. The trainers used the prototype to record their performance while the students used the prototype to learn from the recorded performance. Participants from the three professional domains evaluated the usability of the prototype by means of a questionnaire. Randomly selected participants were also interviewed to collect their opinions and suggestion for further usability improvement. Furthermore, they also evaluated the implementation of the instructional design methods, which were identified prior in a literature review, with a brief questionnaire. The questionnaire was designed to measure the acceptance of the implementation of instructional design methods and to evaluate its adherence to the authors definition. The results of this study show that the usability of the prototype is below expected standard acceptable level. The results of the questionnaire on the implementation of the instructional design methods varied show above average acceptance levels by both the trainers and the students in the three professional domains. To conclude, the prototype shows potential to be used in different domains to support expertise development.

Experts are imperative for training apprentices, but learning from experts is difficult. Experts often struggle to explicate and/or verbalize their knowledge or simply overlook important details due to internalization of their skills, which may make it more difficult for apprentices to learn from experts. In addition, the shortage of experts to support apprentices in one-to-one settings during trainings limits the development of apprentices. In this review, we investigate how augmented reality and sensor technology can be used to capture expert performance in such a way that the captured performance can be used to train apprentices without increasing the workload on experts. To this end, we have analysed 78 studies that have implemented augmented reality and sensor technology for training purposes. We explored how sensors have been used to capture expert performance with the intention of supporting apprentice training. Furthermore, we classified the instructional methods used by the studies according to the 4C/ID framework to understand how augmented reality and sensor technology have been used to support training. The results of this review show that augmented reality and sensor technology have the potential to capture expert performance for training purposes. The results also outline a methodological approach to how sensors and augmented reality learning environments can be designed for training using recorded expert performance.