Print Email Facebook Twitter Data vs. Model Machine Learning Fairness Testing Title Data vs. Model Machine Learning Fairness Testing: An Empirical Study Author Shome, A. (TU Delft Software Engineering) Cruz, Luis (TU Delft Software Engineering) van Deursen, A. (TU Delft Software Engineering) Date 2024 Abstract Although several fairness definitions and bias mitigation techniques exist in the literature, all existing solutions evaluate fairness of Machine Learning (ML) systems after the training stage. In this paper, we take the first steps towards evaluating a more holistic approach by testing for fairness both before and after model training. We evaluate the effectiveness of the proposed approach and position it within the ML development lifecycle, using an empirical analysis of the relationship between model dependent and independent fairness metrics. The study uses 2 fairness metrics, 4 ML algorithms, 5 real-world datasets and 1600 fairness evaluation cycles. We find a linear relationship between data and model fairness metrics when the distribution and the size of the training data changes. Our results indicate that testing for fairness prior to training can be a "cheap" and effective means of catching a biased data collection process early; detecting data drifts in production systems and minimising execution of full training cycles thus reducing development time and costs. Subject Datacentric AIEmpirical Software EngineeringML Fairness TestingSE4ML To reference this document use: http://resolver.tudelft.nl/uuid:b6618a7d-78b0-44b7-9dc1-b0a2455595cf DOI https://doi.org/10.1145/3639478.3643121 Publisher IEEE ISBN 9798400705021 Source Proceedings - 2024 ACM/IEEE 46th International Conference on Software Engineering: Companion, ICSE-Companion 2024 Event 46th International Conference on Software Engineering: Companion, ICSE-Companion 2024, 2024-04-14 → 2024-04-20, Lisbon, Portugal Series Proceedings - International Conference on Software Engineering, 0270-5257 Part of collection Institutional Repository Document type conference paper Rights © 2024 A. Shome, Luis Cruz, A. van Deursen Files PDF 3639478.3643121.pdf 560.03 KB Close viewer /islandora/object/uuid:b6618a7d-78b0-44b7-9dc1-b0a2455595cf/datastream/OBJ/view