Building a workbench to improve sharing and reproducibility of MOOC experiments

Abstract

This thesis improves sharing of code and reproducibility (S&R) in research for massive open online courses (MOOCs). Reproducibility is recreating an experiment by a different researcher. Science in general struggles with repro- ducibility. MOOC experiments often contain useful code that could be used by other researchers, but that code is oftentimes not shared with others. To improve S&R in MOOC research, this thesis first identifies the challenges MOOC sci- entists encounter when trying to share code and when making their experiments reproducible. Then, user interviews are performed to further understand MOOC research and to better understand the challenges identified earlier. A conceptual experimental workflow is designed and implemented in the form of a workbench. The workbench is then evaluated.
The identified challenges based on literature with regards to S&R are: (1) Difficulty in selecting the right tools, (2) lack of a standardized workflow that en- ables reprodubility and (3) manual work required to enable reproducibility with- out a clear incentive to perform that work. From the user interviews, researchers indicate experiencing these same challenges. Based on these challenges, I pro- pose an experimental workflow that focuses on making research reproducible and on sharing code. This workflow is implemented in the form of a work- bench, where researchers can create and manage their MOOC experiments. This workbench allows for sharing code between researchers and is evaluated using a real-world data science task by three actual large-scale learning analytics re- searchers and a visiting volunteer researcher. The evaluation finds that the work- bench needs more work to be suitable for actual MOOC research use and that more researcher education is needed to improve sharing and reproducibility in MOOC research.