Reducing Human Error in Online Controlled Experiments

Abstract

Online controlled experimentation (OCE), also called A/B testing, is an often used tool in industry to determine if deploying changes into production is the right decision to make. Running experiments has shown an immense impact to the revenue of companies in industry, however this type of experimentation comes with a lot of pitfalls, of which some that can invalidate the entire experiment. This thesis describes the impact these pitfalls have on the work of experimenters at ING, a global bank, by performing informal interviews with practitioners and performing a survey with 52 participants. Next, building on existing solutions, a set of solutions is proposed to solve these pitfalls. To determine if these solutions solve the problem and will help the experimenter, these solutions are validated in the same survey. This thesis shows that experimenters are well informed about the existence of pitfalls and believe that almost all should be resolved, with the exception of competitor safety, which is believed to not be important. There are many promising solutions to these pitfalls which experimenters rate as helpful. The best rated solution was ”Enforcing the correct experiment duration”. Almost all respondents perceived the solution to (slightly) help the experimenter in performing their experiments. Finally, this thesis creates a roadmap for evaluating these solutions in a real-world scenario.