Reducing Human Error in Online Controlled Experiments: A case study at ING

Online controlled experimentation (OCE), also called A/B testing, is an often used tool in industry to determine if deploying changes into production is the right decision to make. Running experiments has shown an immense impact to the revenue of companies in industry, however this type of experimentation comes with a lot of pitfalls, of which...

Server-Side Experimentation: Common challenges from 12 organisations in The Netherlands

Background: experimentation is widely adopted within industry. Many large organisations have invested in their own infrastructure to be able to run experiments server-side. Objective: we wanted to investigate why organisations switch to server-side experimen- tation and which challenges they encounter in doing so. Method: a qualitative virtual...

Data Driven Decisions: Validating and Supporting a Continuous Experimentation Development Environment

The number of conducted A/B tests is growing throughout companies in software development. Many of these companies develop their own in-house Experimentation Platform to support these experiments. In this thesis we identify factors that influence the trustworthiness and soundness of A/B tests by conducting a literature review. We discuss...