Reproducibility and Replicability of Simulation Models

Abstract

Computer simulation is increasingly complex and popular in virtually every domain. But computer models and experiments are rarely reproduced or replicated by independent researchers. With the goal to form a stronger community for (computationally) reproducible and/or replicable simulation models, and to encourage collaboration on the topic, this paper aims to highlight the values and challenges of simulation Reproducibility and Replicability (R&R), and call for more R&R research. It first reviews the terms R&R, and then presents different views and opinions on the topic. It explains the separation of method and result stages in reproducibility assessment, and discusses typical challenges in conducting R&R studies. Given the current complexity and widespread use of diverse simulation models, this paper argues that the R&R of such models must be explicitly integrated into existing modelling workflows. Researchers who engage in these efforts face numerous methodological challenges, many of which remain under-studied.