Data Driven Decisions

Validating and Supporting a Continuous Experimentation Development Environment

Master thesis (2019)

Authors

E.A. Mulders Electrical Engineering, Mathematics and Computer Science

Contributors

G. Gousios Software Engineering - EEMCS (supervisor 1)
A. van Deursen Software Technology (supervisor 2)
A Katsifodimos Web Information Systems - EEMCS (supervisor 2)
Kevin Anderson ING (supervisor 1)
Faculty
Electrical Engineering, Mathematics and Computer Science
Copyright: © 2019 Ernst Mulders
More Info
expand_more

Published Date

04-12-2019

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Abstract

The number of conducted A/B tests is growing throughout companies in software development. Many of these companies develop their own in-house Experimentation Platform to support these experiments. In this thesis we identify factors that influence the trustworthiness and soundness of A/B tests by conducting a literature review. We discuss nineteen influential factors categorised as essentials and pitfalls. Using the data of 268 experiments from ING we verify the trustworthiness of ING’s own Experiment Platform, and conclude that there is room for improvement. Finally, we provide a method for developers and engineers to consider these factors during the experimentation phase by modelling them into a questionnaire containing 67 questions. These questions are grouped into three categories, which are referred to as the three A’s of A/B Test validation: Availability, Analysability and Accuracy. To help administer this questionnaire we introduce the first Open-Source toolkit for this matter: ABvalidator.