Rene F. Kizilcec | TU Delft Repository

Third Annual Workshop on A/B Testing and Platform-Enabled Learning Research

Conference paper (2022) - Steven Ritter, Neil Heffernan, Joseph Jay Williams, Derek Lomas, Ben Motz, Debshila Basu Mallick, Klinton Bicknell, Danielle McNamara, Rene F. Kizilcec, More Authors...

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing, which is common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE). A number of A/B testing systems focused on educational applications have arisen recently, including UpGrade and E-TRIALS. A/B testing can be part of the puzzle of how to improve educational platforms, and yet challenging issues in education go beyond the generic paradigm. For example, the importance of teachers and instructors to learning means that students are not only connecting with software as individuals, but also as part of a shared classroom experience. Further, learning in topics like mathematics can be highly dependent on prior learning, and thus A or B may not be better overall, but only in interaction with prior knowledge. In response, a set of learning platforms is opening their systems to improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing in educational contexts is different, how learning platforms are opening up new possibilities, and how these empirical approaches can be used to drive powerful gains in student learning. It will also discuss forthcoming opportunities for funding to conduct platform-enabled learning research. ...

The Half-Life of MOOC Knowledge

A Randomized Trial Evaluating the Testing Effect in MOOCs

Conference paper (2018) - Dan Davis, René F. Kizilcec, Claudia Hauff, Geert-Jan Houben

Retrieval practice has been established in the learning sciences as one of the most effective strategies to facilitate robust learning in traditional classroom contexts. The cognitive theory underpinning the "testing effect" states that actively recalling information is more effective than passively revisiting materials for storing information in long-term memory. We document the design, deployment, and evaluation of an Adaptive Retrieval Practice System (ARPS) in a MOOC. This push-based system leverages the testing effect to promote learner engagement and achievement by intelligently delivering quiz questions from prior course units to learners throughout the course. We conducted an experiment in which learners were randomized to receive ARPS in a MOOC to track their performance and behavior compared to a control group. In contrast to prior literature, we find no significant effect of retrieval practice in this MOOC environment. In the treatment condition, passing learners engaged more with ARPS but exhibited similar levels of knowledge retention as non-passing learners. ...

Scaling Effective Learning Strategies

Retrieval Practice and Long-Term Knowledge Retention in MOOCs

Journal article (2018) - Daniel Davis, René F. Kizilcec, Claudia Hauff, Geert-Jan Houben

Large-scale online learning environments such as MOOCs provide an opportunity to evaluate the efficacy of learning strategies in an informal learning context with a diverse learner population. Here, we evaluate the extent to which retrieval practice — recognized as one of the most effective learning strategies — facilitates long-term knowledge retention (and thus learning outcomes) among MOOC learners using an instructional intervention. We observed no effect on learning outcomes and high levels of treatment non-compliance. In addition, we conducted a series of exploratory studies into long-term recall of knowledge acquired in MOOCs. Surprisingly, both passing and non-passing learners scored similarly on a knowledge post-test, retaining approximately two-thirds of what they learned over the long term. ...

Follow the successful crowd

Raising MOOC completion rates through social comparison at scale

Conference paper (2017) - D.J. Davis, Ioana Jivet, René F. Kizilcec, Guanliang Chen, Claudia Hauff, Geert Jan Houben

Social comparison theory asserts that we establish our social and personal worth by comparing ourselves to others. In in-person learning environments, social comparison offers students critical feedback on how to behave and be successful. By contrast, online learning environments afford fewer social cues to facilitate social comparison. Can increased availability of such cues promote effective self-regulatory behavior and achievement in Massive Open Online Courses (MOOCs)? We developed a personalized feedback system that facilitates social comparison with previously successful learners based on an interactive visualization of multiple behavioral indicators. Across four randomized controlled trials in MOOCs (overall N = 33, 726), we find: (1) the availability of social comparison cues significantly increases completion rates, (2) this type of feedback benefits highly educated learners, and (3) learners' cultural context plays a significant role in their course engagement and achievement. ...