Massive Open Online Courses (MOOCs), as one of the popular options for people to receive education and learn, are endowed with the mission to educate the world. Typically, there are two types of MOOC platforms: topic-agnostic and topic-specific. Topic-agnostic platforms such as edX and Coursera provide courses covering a wide range of topics, while topic-specific MOOC platforms such as Duolingo and Codeacademy focus on courses in one specific topic. To better support MOOC learners, many works have been proposed to investigate MOOC learning in the past decade. Still, there are many other aspects of MOOC learning to be explored.

In this thesis, we focused on (i) learner modeling and (ii) generation of educational material for both topic-agnostic and topic-specific MOOC platforms.

Taking advantage of the vast history of theoretical and empirical findings in the learning literature we have inherited, this research offers a synthesis of prior findings in the domain of empirically evaluated active learning strategies in digital learning environments. The primary concern of the present study is to evaluate these findings with an eye towards scalable learning. Massive Open Online Courses (MOOCs) have emerged as the new way to reach the masses with educational materials, but so far they have failed to maintain learners' attention over the long term. Even though we now understand how effective active learning principles are for learners, the current landscape of MOOC pedagogy too often allows for passivity — leading to the unsatisfactory performance experienced by many MOOC learners today. As a starting point to this research we took John Hattie's seminal work from 2008 on learning strategies used to facilitate active learning. We considered research published between 2009 and 2017 that presents empirical evaluations of these learning strategies. Through our systematic search we found 126 papers meeting our criteria and categorized them according to Hattie's learning strategies. We found large-scale experiments to be the most challenging environment for experimentation due to their size, heterogeneity of participants, and platform restrictions, and we identified the three most promising strategies for effectively leveraging learning at scale as Cooperative Learning, Simulations & Gaming, and Interactive Multimedia

Massive Open Online Courses (MOOCs) aim to educate the world. More often than not, however, MOOCs fall short of this goal — a majority of learners are already highly educated (with a Bachelor degree or more) and come from specific parts of the (developed) world. Learners from developing countries without a higher degree are underrepresented, though desired, in MOOCs. One reason for those learners to drop out of a course can be found in their financial realities and the subsequent limited amount of time they can dedicate to a course besides earning a living. If we could pay learners to take a MOOC, this hurdle would largely disappear. With MOOCS, this leads to the following fundamental challenge: How can learners be paid at scale? Ultimately, we envision a recommendation engine that recommends tasks from online market places such as Upwork or witmart to learners, that are relevant to the course content of the MOOC. In this manner, the learners learn and earn money. To investigate the feasibility of this vision, in this paper we explored to what extent (1) online market places contain tasks relevant to a specific MOOC, and (2) learners are able to solve real-world tasks correctly and with sufficient quality. Finally, based on our experimental design, we were also able to investigate the impact of real-world bonus tasks in a MOOC on the general learner population.

We present LearningQ, a challenging educational question generation dataset containing over 230K document-question pairs. It includes 7K instructor-designed questions assessing knowledge concepts being taught and 223K learner-generated questions seeking in-depth understanding of the taught concepts. We show that, compared to existing datasets that can be used to generate educational questions, LearningQ (i) covers a wide range of educational topics and (ii) contains long and cognitively demanding documents for which question generation requires reasoning over the relationships between sentences and paragraphs. As a result, a significant percentage of LearningQ questions (~30%) require higher-order cognitive skills to solve (such as applying, analyzing), in contrast to existing question-generation datasets that are designed mostly for the lowest cognitive skill level (i.e. remembering). To understand the effectiveness of existing question generation methods in producing educational questions, we evaluate both rule-based and deep neural network based methods on LearningQ. Extensive experiments show that state-of-the-art methods which perform well on existing datasets cannot generate useful educational questions. This implies that LearningQ is a challenging test bed for the generation of high-quality educational questions and worth further investigation. We open-source the dataset and our codes at https://dataverse.mpi-sws.org/dataverse/icwsm18.

Knowledge tracing serves as a keystone in delivering personalized education. However, few works attempted to model students’ knowledge state in the setting of Second Language Acquisition. The Duolingo Shared Task on Second Language Acquisition Modeling (Settles et al., 2018) provides students’ trace data that we extensively analyze and engineer features from for the task of predicting whether a student will correctly solve a vocabulary exercise. Our analyses of students’ learning traces reveal that factors like exercise format and engagement impact their exercise performance to a large extent. Overall, we extracted 23 different features as input to a Gradient Tree Boosting framework, which resulted in an AUC score of between 0.80 and 0.82 on the official test set.

MOOCs promised to herald a new age of open education.
However, efficient access to MOOC content is still hard, thus unneces-
sarily complicating many use cases like efficient re-use of material, or
tailored access for life-long learning scenarios. One of the reasons for this
lack of accessibility is the shortage of meaningful semantic meta-data de-
scribing MOOC content and the resulting learning experience. In this pa-
per, we explore Concept Focus, a new type of meta-data for describing a
perceptual facet of modern video-based MOOCs, capturing how focused
a learning resource is topic-wise, which is often an indicator of clarity
and understandability. We provide the theoretical foundations of Con-
cept Focus and outline a methodical workflow of how to automatically
compute it for MOOC lectures. Furthermore, we show that the learners’
consumption behavior is correlated with a MOOC lecture’s Concept Focus, thus underlining that this type of meta-data is indeed relevant for user-centric querying, personalizing or even designing the MOOC experience. For showing this, we performed an extensive study with real-life
MOOCs and 12,849 learners over the duration of three months.

Social comparison theory asserts that we establish our social and personal worth by comparing ourselves to others. In in-person learning environments, social comparison offers students critical feedback on how to behave and be successful. By contrast, online learning environments afford fewer social cues to facilitate social comparison. Can increased availability of such cues promote effective self-regulatory behavior and achievement in Massive Open Online Courses (MOOCs)? We developed a personalized feedback system that facilitates social comparison with previously successful learners based on an interactive visualization of multiple behavioral indicators. Across four randomized controlled trials in MOOCs (overall N = 33, 726), we find: (1) the availability of social comparison cues significantly increases completion rates, (2) this type of feedback benefits highly educated learners, and (3) learners' cultural context plays a significant role in their course engagement and achievement.

Massive Open Online Courses (MOOCs) are a promising form of online education. However, the occurrence of academic dishonesty has been threatening MOOC certificates’ effectiveness as a serious tool for recruiters and employers. Recently, a large-scale study on the log traces from more than one hundred MOOCs created by Harvard and MIT has identified a specific cheating strategy viable in MOOCs: Copying Answers using Multiple Existences Online (CAMEO). In essence, learners create several accounts on a MOOC platform, request assessment solutions via some of the accounts, and then submit these “harvested” solutions in their main account to receive credit. In our work, we replicate the CAMEO implementation and apply it to ten edX MOOCs created by the Delft University of Technology. Our results show that in those MOOCs, 1.9% of certificates were likely earned through CAMEO cheating, a number comparable to the fraction of cheating observed in Harvard and MIT MOOCs.

Massive Open Online Courses (MOOCs) aim to educate the world, especially learners from developing countries. While MOOCs are certainly available to the masses, they are not yet fully accessible. Although all course content is just clicks
away, deeply engaging with a MOOC requires a substantial time commitment, which frequently becomes a barrier to success. To mitigate the time required to learn from a MOOC, we here introduce a design that enables learners to earn money by applying what they learn in the course to real-world marketplace tasks. We present a Paid Task Recommender System (Rec-$ys), which automatically recommends course-relevant tasks to learners as drawn from
online freelance platforms. Rec-$ys has been deployed into a data analysis MOOC and is currently under evaluation.

Massive Open Online Courses (MOOCs) play an ever more central role in open education. However, in contrast to traditional classroom settings, many aspects of learners' behaviour in MOOCs are not well researched. In this work, we focus on modelling learner behaviour in the context of continuous assessments with completion certificates, the most common assessment setup in MOOCs today. Here, learners can obtain a completion certificate once they obtain a required minimal score (typically somewhere between 50-70%) in tests distributed throughout the duration of a MOOC. In this setting, the course material or tests provided after "passing" do not contribute to earning the certificate (which is ungraded), thus potentially affecting learners' behaviour. Therefore, we explore how ``passing'' impacts MOOC learners: do learners alter their behaviour after this point? And if so how? While in traditional classroom-based learning the role of assessment and its influence on learning behaviour has been well-established, we are among the first to provide answers to these questions in the context of MOOCs.

MOOC research is typically limited to evaluations of learner behavior in the context of the learning environment. However, some research has begun to recognize that the impact of MOOCs may extend beyond the confines of the course platform or conclusion of the course time limit. This workshop aims to encourage our community of learning analytics researchers to examine the relationship between performance and engagement within the course and learner behavior and development beyond the course. This workshop intends to build awareness in the community regarding the importance of research measuring multi-platform activity and long-term success after taking a MOOC. We hope to build the community's understanding of what it takes to operationalize MOOC learner success in a novel context by employing data traces across the social web

Massive Open Online Courses (MOOCs) have gained considerable momentum since their inception in 2011. They are, however, plagued by two issues that threaten their future: learner engagement and learner retention. MOOCs regularly attract tens of thousands of learners, though only a very small percentage complete them successfully. In the traditional classroom setting, it has been established that personality impacts different aspects of learning. It is an open question to what extent this finding translates to MOOCs: do learners' personalities impact their learning & learning behaviour in the MOOC setting? In this paper, we explore this question and analyse the personality profiles and learning traces of hundreds of learners that have taken a EX101x Data Analysis MOOC on the edX platform. We find learners' personality traits to only weakly correlate with learning as captured through the data traces learners leave on edX.

Massive Open Online Course (MOOC) platform designs, such as those of edX and Coursera, aord linear learning sequences by building scaolded knowledge from activity to activity and from week to week. We consider those sequences to be the courses' designed learning paths. But do learners actually adhere to these designed paths, or do they forge their own ways through the MOOCs? What are the implications of either following or not following the designed paths? Existing research has greatly emphasized, and succeeded in, automatically predicting MOOC learner success and learner dropout based on behavior patterns derived from MOOC learners' data traces. However, those predictions do not directly translate into practicable information for course designers & instructors aiming to improve engagement and retention | the two major issues plaguing today's MOOCs. In this work, we present a three-pronged approach to exploring MOOC data for novel learning path insights, thus enabling course instructors & designers to adapt a course's design based on empirical evidence.

Massive Open Online Courses (MOOCs) have enabled millions of learners across the globe to increase their levels of expertise in a wide variety of subjects. Research efforts surrounding MOOCs are typically focused on improving the learning experience, as the current retention rates (less than 7% of registered learners complete a MOOC) show a large gap between vision and reality in MOOC learning. Current data-driven approaches to MOOC adaptations rely on data traces learners generate within a MOOC platform such as edX or Coursera. As a MOOC typically lasts between five and eight weeks and with many MOOC learners being rather passive consumers of the learning material, this exclusive use of MOOC platform data traces limits the insights that can be gained from them. The SocialWeb potentially offers a rich source of data to supplement the MOOC platform data traces, as many learners are also likely to be active on one or more Social Web platforms. In this work, we present a first exploratory analysis of the Social Web platforms MOOC learners are active on - we consider more than 320,000 learners that registered for 18 MOOCs on the edX platform and explore their user profiles and activities on StackExchange, GitHub, Twitter and LinkedIn.

The rising number of Massive Open Online Courses (MOOCs) enable people to advance their knowledge and competencies in a wide range of fields. Learning though is only the first step, the transfer of the taught concepts into practice is equally important and often neglected in the investigation of MOOCs. In this paper, we consider the specific case of FP101x (a functional programming MOOC on edX) and the extent to which learners alter their programming behaviour after having taken the course. We are able to link about one third of all FP101x learners to GitHub, the most popular social coding platform to date and contribute a first exploratory analysis of learner behaviour beyond the MOOC platform. A detailed longitudinal analysis of GitHub log traces reveals that (i) more than 8% of engaged learners transfer, and that (ii) most existing transfer learning findings from the classroom setting are indeed applicable in the MOOC setting as well.

Massive Open Online Courses (MOOCs) are successful in delivering educational resources to themasses, however, the current retention rates—well below 10%—indicate that they fall short in helping their audience become effective MOOC learners. In this paper, we report two MOOC studies we conducted in order to test the effectiveness of pedagogical strategies found to be beneficial in the traditional classroom setting: retrieval practice (i.e. strengthening course knowledge through actively recalling information) and study planning (elaborating on weekly study plans). In contrast to the classroom-based results, we do not confirm our hypothesis, that small changes to the standard MOOC design can teach MOOC learners valuable self-regulated learning strategies.

Learning analytics for learners has the ability to greatly improve learners' self-regulation. Current learner dashboards are mostly providing learners with an isolated view of their learning behavior, while we believe learners will gain more from a comparison of their own behavior with that of successful peer learners. In this work-in-progress demonstration we describe our design of a Learning Tracker widget that provides MOOC learners with timely and goal-oriented (i.e. towards passing the course) feedback in a manner that encourages reflection and self-regulation. We also present some preliminary ndings which show how exposure to feedback can signicantly increase student success and engagement.