Massive Open Online Courses (MOOCs) allow learning to take place anytime and anywhere with little external monitoring by teachers. Characteristically, highly diverse groups of learners enrolled in MOOCs are required to make decisions related to their own learning activities to achieve academic success. Therefore, it is considered important to support self-regulated learning (SRL) strategies and adapt to relevant human factors (e.g., gender, cognitive abilities, prior knowledge). SRL supports have been widely investigated in traditional classroom settings, but little is known about how SRL can be supported in MOOCs. Very few experimental studies have been conducted in MOOCs at present. To fill this gap, this paper presents a systematic review of studies on approaches to support SRL in multiple types of online learning environments and how they address human factors. The 35 studies reviewed show that human factors play an important role in the efficacy of SRL supports. Future studies can use learning analytics to understand learners at a fine-grained level to provide support that best fits individual learners. The objective of the paper is twofold: (a) to inform researchers, designers and teachers about the state of the art of SRL support in online learning environments and MOOCs; (b) to provide suggestions for adaptive self-regulated learning support.

@en

The study of learning is grounded in theories and research. Since learning is complex and not directly observable, it is often inferred by collecting and analysing data based on the things learners do or say. By virtue, theories are developed from the analyses of data collected. With the proliferation of technology, large amounts of data are generated when students learn online. Therefore, researchers not only have data on students’ learning performance, but they also have data on the actions students take to achieve the desired learning outcomes. These data could help researchers to understand how students learn and the conditions needed for successful learning. In turn, the information can be translated to instructional and learning design to support students. The aim of the chapter is to discuss how learning theories and learning analytics are important components of educational research. To achieve this aim, studies employing learning analytics are qualitatively reviewed to examine which theories have been used and how the theories have been investigated. The results of the review show that self-regulated learning, motivation, and social constructivism theories were used in studies employing learning analytics. However, the studies at present are mostly correlational. Therefore, experimental studies are needed to examine how theory-informed practices can be implemented so that students can be better supported in online learning environments. The chapter concludes by proposing an iterative loop for educational research employing learning analytics in which learning theories guide data collection and analyses. To convert data into knowledge, it is important to recognize what we already know and what we want to examine. @en

This paper aims to link student facing Learning Analytics Dashboards (LADs) to the corpus of research on Open Learner Models (OLMs), as both have similar goals. We conducted a systematic review of literature on OLMs and compared the results with a previously conducted review of LADs for learners in terms of (i) data use and modelling, (ii) key publication venues, (iii) authors and articles, (iv) key themes, and (v) system evaluation. We highlight the similarities and differences between the research on LADs and OLMs. Our key contribution is a bridge between these two areas as a foundation for building upon the strengths of each. We report the following key results from the review: in reports of new OLMs, almost 60% are based on a single type of data; 33% use behavioral metrics; 39% support input from the user; 37% have complex models; and just 6% involve multiple applications. Key associated themes include intelligent tutoring systems, learning analytics, and self-regulated learning. Notably, compared with LADs, OLM research is more likely to be interactive (81% of papers compared with 31% for LADs), report evaluations (76% versus 59%), use assessment data (100% versus 37%), provide a comparison standard for students (52% versus 38%), but less likely to use behavioral metrics, or resource use data (33% against 75% for LADs). In OLM work, there was a heightened focus on learner control and access to their own data.

@en

Past research in large-scale learning environments has found one of the most inhibiting factors to learners’ success to be their inability to effectively self-regulate their learning efforts. In traditional small-scale learning environments, personalized feedback (on progress, content, behavior, etc.) has been found to be an effective solution to this issue, but it has not yet widely been evaluated at scale. In this paper we present the Personalized SRL Support System (SRLx), an interactive widget that we designed and open-sourced to improve learners’ self-regulated learning behavior in the Massive Open Online Course platform edX. SRLx enables learners to plan their learning on a weekly basis and view real-time feedback on the realization of those plans. We deployed SRLx in a renewable energies MOOC to more than 2,900 active learners and performed an exploratory analysis on our learners’ SRL behavior.@en

Retrieval practice has been established in the learning sciences as one of the most effective strategies to facilitate robust learning in traditional classroom contexts. The cognitive theory underpinning the "testing effect" states that actively recalling information is more effective than passively revisiting materials for storing information in long-term memory. We document the design, deployment, and evaluation of an Adaptive Retrieval Practice System (ARPS) in a MOOC. This push-based system leverages the testing effect to promote learner engagement and achievement by intelligently delivering quiz questions from prior course units to learners throughout the course. We conducted an experiment in which learners were randomized to receive ARPS in a MOOC to track their performance and behavior compared to a control group. In contrast to prior literature, we find no significant effect of retrieval practice in this MOOC environment. In the treatment condition, passing learners engaged more with ARPS but exhibited similar levels of knowledge retention as non-passing learners.@en

Massive Open Online Courses (MOOCs) aim to educate the world. More often than not, however, MOOCs fall short of this goal — a majority of learners are already highly educated (with a Bachelor degree or more) and come from specific parts of the (developed) world. Learners from developing countries without a higher degree are underrepresented, though desired, in MOOCs. One reason for those learners to drop out of a course can be found in their financial realities and the subsequent limited amount of time they can dedicate to a course besides earning a living. If we could pay learners to take a MOOC, this hurdle would largely disappear. With MOOCS, this leads to the following fundamental challenge: How can learners be paid at scale? Ultimately, we envision a recommendation engine that recommends tasks from online market places such as Upwork or witmart to learners, that are relevant to the course content of the MOOC. In this manner, the learners learn and earn money. To investigate the feasibility of this vision, in this paper we explored to what extent (1) online market places contain tasks relevant to a specific MOOC, and (2) learners are able to solve real-world tasks correctly and with sufficient quality. Finally, based on our experimental design, we were also able to investigate the impact of real-world bonus tasks in a MOOC on the general learner population.@en

Large-scale online learning environments such as MOOCs provide an opportunity to evaluate the efficacy of learning strategies in an informal learning context with a diverse learner population. Here, we evaluate the extent to which retrieval practice — recognized as one of the most effective learning strategies — facilitates long-term knowledge retention (and thus learning outcomes) among MOOC learners using an instructional intervention. We observed no effect on learning outcomes and high levels of treatment non-compliance. In addition, we conducted a series of exploratory studies into long-term recall of knowledge acquired in MOOCs. Surprisingly, both passing and non-passing learners scored similarly on a knowledge post-test, retaining approximately two-thirds of what they learned over the long term.@en

This paper applies theory and methodology from the learning design literature to large-scale learning environments through quantitative modeling of the structure and design of Massive Open Online Courses. For two institutions of higher education, we automate the task of encoding pedagogy and learning design principles for 177 courses (which accounted for for nearly 4 million enrollments). Course materials from these MOOCs are parsed and abstracted into sequences of components, such as videos and problems. Our key contributions are (i) describing the parsing and abstraction of courses for quantitative analyses, (ii) the automated categorization of similar course designs, and (iii) the identification of key structural components that show relationships between categories and learning design principles. We employ two methods to categorize similar course designs---one aimed at clustering courses using transition probabilities and another using trajectory mining. We then proceed with an exploratory analysis of relationships between our categorization and learning outcomes.@en

Taking advantage of the vast history of theoretical and empirical findings in the learning literature we have inherited, this research offers a synthesis of prior findings in the domain of empirically evaluated active learning strategies in digital learning environments. The primary concern of the present study is to evaluate these findings with an eye towards scalable learning. Massive Open Online Courses (MOOCs) have emerged as the new way to reach the masses with educational materials, but so far they have failed to maintain learners' attention over the long term. Even though we now understand how effective active learning principles are for learners, the current landscape of MOOC pedagogy too often allows for passivity — leading to the unsatisfactory performance experienced by many MOOC learners today. As a starting point to this research we took John Hattie's seminal work from 2008 on learning strategies used to facilitate active learning. We considered research published between 2009 and 2017 that presents empirical evaluations of these learning strategies. Through our systematic search we found 126 papers meeting our criteria and categorized them according to Hattie's learning strategies. We found large-scale experiments to be the most challenging environment for experimentation due to their size, heterogeneity of participants, and platform restrictions, and we identified the three most promising strategies for effectively leveraging learning at scale as Cooperative Learning, Simulations & Gaming, and Interactive Multimedia@en

Crowdsourcing has emerged as an effective method of scaling-up tasks previously reserved for a small set of experts. Accordingly, researchers in the large-scale online learning space have begun to employ crowdworkers to conduct research about large-scale, open online learning. We here report results from a crowdsourcing study (N=135) to evaluate the extent to which crowdworkers and MOOC learners behave comparably on lecture viewing and quiz tasks---the most utilized learning activities in MOOCs. This serves to (i) validate the assumption of previous research that crowdworkers are indeed reliable proxies of online learners and (ii) address the potential of employing crowdworkers as a means of online learning environment testing. Overall, we observe mixed results---in certain contexts (quiz performance and video watching behavior) crowdworkers appear to behave comparably to MOOC learners, and in other situations (interactions with in-video quizzes), their behaviors appear to be disparate. We conclude that future research should be cautious if employing crowdworkers to carry out learning tasks, as the two populations do not behave comparably on all learning-related activities.@en

MOOC research is typically limited to evaluations of learner behavior in the context of the learning environment. However, some research has begun to recognize that the impact of MOOCs may extend beyond the confines of the course platform or conclusion of the course time limit. This workshop aims to encourage our community of learning analytics researchers to examine the relationship between performance and engagement within the course and learner behavior and development beyond the course. This workshop intends to build awareness in the community regarding the importance of research measuring multi-platform activity and long-term success after taking a MOOC. We hope to build the community's understanding of what it takes to operationalize MOOC learner success in a novel context by employing data traces across the social web@en

Social comparison theory asserts that we establish our social and personal worth by comparing ourselves to others. In in-person learning environments, social comparison offers students critical feedback on how to behave and be successful. By contrast, online learning environments afford fewer social cues to facilitate social comparison. Can increased availability of such cues promote effective self-regulatory behavior and achievement in Massive Open Online Courses (MOOCs)? We developed a personalized feedback system that facilitates social comparison with previously successful learners based on an interactive visualization of multiple behavioral indicators. Across four randomized controlled trials in MOOCs (overall N = 33, 726), we find: (1) the availability of social comparison cues significantly increases completion rates, (2) this type of feedback benefits highly educated learners, and (3) learners' cultural context plays a significant role in their course engagement and achievement.

@en

Massive Open Online Courses (MOOCs) aim to educate the world, especially learners from developing countries. While MOOCs are certainly available to the masses, they are not yet fully accessible. Although all course content is just clicks away, deeply engaging with a MOOC requires a substantial time commitment, which frequently becomes a barrier to success. To mitigate the time required to learn from a MOOC, we here introduce a design that enables learners to earn money by applying what they learn in the course to real-world marketplace tasks. We present a Paid Task Recommender System (Rec-$ys), which automatically recommends course-relevant tasks to learners as drawn from online freelance platforms. Rec-$ys has been deployed into a data analysis MOOC and is currently under evaluation.@en

In this paper, we demonstrate a first-of-its-kind adaptive intervention in a MOOC utilizing real-time clickstream data and a novel machine learned model of behavior. We detail how we augmented the edX platform with the capabilities necessary to support this type of intervention which required both tracking learners' behaviors in real-time and dynamically adapting content based on each learner's individual clickstream history. Our chosen pilot intervention was in the category of adaptive pathways and courseware and took the form of a navigational suggestion appearing at the bottom of every non-forum content page in the course. We designed our pilot intervention to help students more efficiently navigate their way through a MOOC by predicting the next page they were likely to spend significant time on and allowing them to jump directly to that page. While interventions which attempt to optimize for learner achievement are candidates for this adaptive framework, behavior prediction has the benefit of not requiring causal assumptions to be made in its suggestions. We present a novel extension of a behavioral model that takes into account students' time spent on pages and forecasts the same. Several approaches to representing time using Recurrent Neural Networks are evaluated and compared to baselines without time, including a basic n-gram model. Finally, we discuss design considerations and handling of edge cases for real-time deployment, including considerations for training a machine learned model on a previous offering of a course for use in a subsequent offering where courseware may have changed. This work opens the door to broad experimentation with adaptivity and serves as a first example of delivering a data-driven personalized learning experience in a MOOC.
@en

Massive Open Online Courses (MOOCs) play an ever more central role in open education. However, in contrast to traditional classroom settings, many aspects of learners' behaviour in MOOCs are not well researched. In this work, we focus on modelling learner behaviour in the context of continuous assessments with completion certificates, the most common assessment setup in MOOCs today. Here, learners can obtain a completion certificate once they obtain a required minimal score (typically somewhere between 50-70%) in tests distributed throughout the duration of a MOOC. In this setting, the course material or tests provided after "passing" do not contribute to earning the certificate (which is ungraded), thus potentially affecting learners' behaviour. Therefore, we explore how ``passing'' impacts MOOC learners: do learners alter their behaviour after this point? And if so how? While in traditional classroom-based learning the role of assessment and its influence on learning behaviour has been well-established, we are among the first to provide answers to these questions in the context of MOOCs.@en

Massive Open Online Courses (MOOCs) have gained considerable momentum since their inception in 2011. They are, however, plagued by two issues that threaten their future: learner engagement and learner retention. MOOCs regularly attract tens of thousands of learners, though only a very small percentage complete them successfully. In the traditional classroom setting, it has been established that personality impacts different aspects of learning. It is an open question to what extent this finding translates to MOOCs: do learners' personalities impact their learning & learning behaviour in the MOOC setting? In this paper, we explore this question and analyse the personality profiles and learning traces of hundreds of learners that have taken a EX101x Data Analysis MOOC on the edX platform. We find learners' personality traits to only weakly correlate with learning as captured through the data traces learners leave on edX.

@en

Massive Open Online Course (MOOC) platform designs, such as those of edX and Coursera, aord linear learning sequences by building scaolded knowledge from activity to activity and from week to week. We consider those sequences to be the courses' designed learning paths. But do learners actually adhere to these designed paths, or do they forge their own ways through the MOOCs? What are the implications of either following or not following the designed paths? Existing research has greatly emphasized, and succeeded in, automatically predicting MOOC learner success and learner dropout based on behavior patterns derived from MOOC learners' data traces. However, those predictions do not directly translate into practicable information for course designers & instructors aiming to improve engagement and retention | the two major issues plaguing today's MOOCs. In this work, we present a three-pronged approach to exploring MOOC data for novel learning path insights, thus enabling course instructors & designers to adapt a course's design based on empirical evidence.@en

Massive Open Online Courses (MOOCs) have enabled millions of learners across the globe to increase their levels of expertise in a wide variety of subjects. Research efforts surrounding MOOCs are typically focused on improving the learning experience, as the current retention rates (less than 7% of registered learners complete a MOOC) show a large gap between vision and reality in MOOC learning. Current data-driven approaches to MOOC adaptations rely on data traces learners generate within a MOOC platform such as edX or Coursera. As a MOOC typically lasts between five and eight weeks and with many MOOC learners being rather passive consumers of the learning material, this exclusive use of MOOC platform data traces limits the insights that can be gained from them. The SocialWeb potentially offers a rich source of data to supplement the MOOC platform data traces, as many learners are also likely to be active on one or more Social Web platforms. In this work, we present a first exploratory analysis of the Social Web platforms MOOC learners are active on - we consider more than 320,000 learners that registered for 18 MOOCs on the edX platform and explore their user profiles and activities on StackExchange, GitHub, Twitter and LinkedIn.

@en

The rising number of Massive Open Online Courses (MOOCs) enable people to advance their knowledge and competencies in a wide range of fields. Learning though is only the first step, the transfer of the taught concepts into practice is equally important and often neglected in the investigation of MOOCs. In this paper, we consider the specific case of FP101x (a functional programming MOOC on edX) and the extent to which learners alter their programming behaviour after having taken the course. We are able to link about one third of all FP101x learners to GitHub, the most popular social coding platform to date and contribute a first exploratory analysis of learner behaviour beyond the MOOC platform. A detailed longitudinal analysis of GitHub log traces reveals that (i) more than 8% of engaged learners transfer, and that (ii) most existing transfer learning findings from the classroom setting are indeed applicable in the MOOC setting as well.@en