Print Email Facebook Twitter Automated Sample Ratio Mismatch (SRM) Detection and Analysis Title Automated Sample Ratio Mismatch (SRM) Detection and Analysis Author Vermeer, Lukas (Vista) Anderson, K.S. (TU Delft Software Engineering; Vista) Acebal, Mauricio (Vista) Contributor Staron, M. (editor) Berger, C. (editor) Simmonds, J. (editor) Prikladnicki, R. (editor) Date 2022 Abstract Background: Sample Ratio Mismatch (SRM) checks can help detect data quality issues in online experimentation [3]. Not all experimentation platforms provide these checks as part of their solution. Users of these platforms must therefore manually check for SRM, or rely on additional processes—such as checklists [2]—or automation. Objective: To ensure reliable and early detection of SRM, we wanted to automate the detection and analysis of SRM in experiments running on third-party experimentation platforms. Method: A set of Looker dashboards were built to facilitate self-serve SRM detection and root cause analysis. In addition, we added email and chat based alerting to pro-actively inform experimenters of SRM and guide them towards these dashboards when needed. Results: Several cases of SRM have been detected and experimenters have been warned. Bad decisions based on flawed data were avoided. We provide one such example as an illustration. Conclusions: SRM checks are relatively straightforward to automate and can be useful for data quality monitoring even for companies who rely on third-party experimentation platforms. Pro-active alerting—rather than passive reporting—can reduce time to detection and help non-experts avoid making decisions based on biased data. Subject InfrastructureData QualityTrustworthinessA/B TestingSample Ratio MismatchSRMOnline Controlled Experimentation To reference this document use: http://resolver.tudelft.nl/uuid:31d2806f-2bf3-43c3-9e71-3985fd7368a6 DOI https://doi.org/10.1145/3530019.3534982 Publisher Association for Computing Machinery (ACM), New York ISBN 978-1-4503-9613-4 Source Proceedings of the ACM International Conference on Evaluation and Assessment in Software Engineering, EASE 2022: The International Conference on Evaluation and Assessment in Software Engineering 2022 Event EASE 2022, 2022-06-13 → 2022-06-15, Gothenburg, Sweden Series ACM International Conference Proceeding Series Part of collection Institutional Repository Document type conference paper Rights © 2022 Lukas Vermeer, K.S. Anderson, Mauricio Acebal Files PDF Automated_Sample_Ratio_Mi ... alysis.pdf 727.54 KB Close viewer /islandora/object/uuid:31d2806f-2bf3-43c3-9e71-3985fd7368a6/datastream/OBJ/view