Learners in the process of developing complex skills need a rich mental model of what such skills entail. Textual analytics rubrics (TR) are a widely used instrument to support formative assessment of complex skills, supporting feedback, reflection, and thus mental model development of complex skills. However, the textual nature of a rubric limits the possibility to deliver contextual and dynamic (time-and behaviour-oriented) information. In the Viewbrics online tool, we developed a version supporting the delivery of contextual and dynamic information by adding video-modelling examples with embedded self-explanation prompts to textual analytics rubrics. We called this combination as video-enhanced rubrics (VERs). Our current study investigates whether the Viewbrics online tool supports complex skills development and whether either textual- or video-enhanced rubrics best support complex skills' mastery. The study was a three-group (VERS n = 49, TR n = 54, control n = 50) within-subjects design. Learners' performance of complex skills was measured through expert, peer, and self-assessment using the Viewbrics online tool. A multilevel regression analysis shows learners in the TR and VERS conditions consistently outperforming the control condition to varying degrees across skills. However, no differences have been found between the two experimental conditions. Positive results across different complex skills indicate the Viewbrics online tool can be used to support the development of a wide range of complex skills in secondary education.

High-quality elaborative peer feedback is a blessing for both learners and teachers. However, learners can experience difficulties in giving high-quality feedback on complex skills using textual analytic rubrics. High-quality elaborative feedback can be strengthened by adding video-modeling examples with embedded self-explanation prompts, turning textual analytic rubrics (TR) into so-called 'video-enhanced analytic rubrics' (VER). This study contrasts two experimental conditions (TR, n = 54; VERs, n = 49) with their version of the anonymized online tool (used to collect the given feedback in 'Tips for improvement and Tops identifying strengths'). Peer feedback quality (concreteness and consistency) was evaluated using Natural Language Processing. As expected, the video-enhanced rubrics condition resulted in a higher quantity of words used and a lower amount of naive wording compared to the textual rubric condition. Contrary to our assumptions, it did not lower the amount of non-constructive wording nor improved the amount of behavioral and process-related feedback. Possibly, the transition from providing more feedback to delivering more accurate behavioral and process-related feedback has not yet been made in the time set for the study.

For learners, it can be difficult to imagine how to perform a complex skill based on textual information from solely a text-based analytic rubric. Rubrics lack (1) the contextual information needed to convey real-world attributes (2) the dynamic information (such as gesturing in the complex skill of presenting) (3) the procedural information required to support the automation of constituent skills. We propose to address the text-based rubric’s deficiencies by adding video-modeling examples, self-explanation prompts, an intertwined educational- and instructional narrative, natural segmentation, and a non-verbal script. With the resulting Video Enhanced Rubric, we aim to improve the formative assessment of complex skills by fostering learner’s mental model development, feedback quality, and complex skill mastery. Designing multimedia to support the formative assessment of complex skills using can cause dilemmas for instructional designers. For example, is learner control needed to foster intrinsic motivation or is it deemed to create extraneous cognitive load? Is it wise to use a video modeling example of peer-aged learners when the model does not display perfect performance? We found seven dilemmas around proven complex skill development, motivational design, and multimedia design guidelines. This paper presents a theoretical contribution to instructional design by introducing a framework to address dilemmas regarding such design dilemmas. As a practical contribution, we support educational researchers and practitioners by presenting six practical guidelines for designing a Video Enhanced Rubric. A use case of the Viewbrics-project provides insight into the practical application of the framework within the context of Dutch pre-university education.

Learners can experience difficulties to form a mental model of a complex skill and what its mastery implicates in terms of actual behavior. Text-based analytic rubrics can support training and formative assessment of complex skills. However, they lack (1) contextual information needed to convey real-world attributes (2) sensor-motoric information (such as gesturing in the complex skill of presenting) (3) procedural information needed to support the automation of (sub)skills. Text-based analytic rubrics deficiencies for skill-mastery can be remedied by adding video-modeling examples with embedded self-explanation prompts, turning text-based analytic rubrics (TR) into so-called ‘video-enhanced analytic rubrics’ (VERS). The current study contrasts two experimental conditions (TR, n = 54; VERS, n = 49) for fostering the growth of learners' mental model of complex skills. This was done using the formative assessment supporting Viewbrics online tool and one control condition with a secondary school's current approach for skills-development (without rubrics nor Viewbrics-formative assessment (n = 50)). We measured learners' mental models at three times during their project work by assessing the richness of their constructed concept maps. A multilevel regression analysis showed that learners in the VERS condition form richer mental models when compared to the control condition for most complex skills.

Learners aiming to master a complex skill may benefit from the combination of abstract information found in a text-based analytical rubric and concrete information provided by a video modeling example. In this paper, we address the design dilemmas of combining video modeling examples and rubrics into a Video Enhanced Rubric. We propose a model to address these design dilemma’s and develop our first prototype based on this model. We review the first prototype through a two-stage international expert validation session. In the first stage, 20 experts are asked to design a user interface for the Video Enhanced Rubric. In the second stage, 20 experts are asked to perform an expert appraisal of our first prototype. The preliminary results of the expert validation session are subsequently analyzed using Sauli, Cattaneo and van der Meij’s Framework for Developing Instructional Hypervideo to detect common design suggestions. Following the results of the expert validation, we developed a second prototype of the Video Enhanced Rubric. With the design guidelines of a Video Enhanced Rubric, we aim to improve the formative assessment and mastery of complex skills by fostering learner’s mental model development and the quality (consistency, concreteness) of both given as well as received feedback. On a more general note, we expect the design dilemmas addressed in this paper to inform researchers who aim to apply theoretical multimedia design guidelines to formative assessment practices with rubrics.

This article describes the theory-informed design of the "ELENA goes shopping" mobile game and reports on the evaluation of its effectiveness through a design research approach. The game aimed to foster young children's (aged 4-8) interest in a neighboring (geographically proximate) language and to familiarize them with its sounds, pronunciation and vocabulary. Additionally, it strived to involve adults in young children's language learning activities. To achieve these objectives, the game connects playful learning activities to an accessible, familiar real-world setting (supermarket). The game was developed and evaluated through three iterative design research cycles. First, interdisciplinary experts (n = 8) evaluated the game by means of a questionnaire and focus group discussion. In the second and third cycles, the game's feasibility and usability was evaluated via questionnaires, semi-structured interviews and a language learning outcome test. Results revealed that children (34) and adults (14) alike found the game useful for familiar ization with and motivation to learn another language, and feasible to involve adults. Nevertheless, children could play the game autonomously with minimum adult assistance. A dependent t-test on a repeated vocabulary test revealed adults' and children's perception that the game helped them familiarize with another language to be consistent with test results. A limitation to this study is that the test was administered immediately after game playing. Future studies could explore effects of "real-world" contextualization on early second language learning and vocabulary recall by measuring after longer time spans and compare results versus a non-contextualized game.