Title
Are Concept Drift Detectors Reliable Alarming Systems?: A Comparative Study
Author
Poenaru-Olaru, L. (TU Delft Software Engineering)
Cruz, Luis (TU Delft Software Engineering) ![ORCID 0000-0002-1615-355X ORCID 0000-0002-1615-355X](/sites/all/themes/tud_repo3/img/icons/orcid_16x16.png)
van Deursen, A. (TU Delft Software Technology) ![ORCID 0000-0003-4850-3312 ORCID 0000-0003-4850-3312](/sites/all/themes/tud_repo3/img/icons/orcid_16x16.png)
Rellermeyer, Jan S. (Leibniz University Hannover) ![ORCID 0000-0003-3791-7114 ORCID 0000-0003-3791-7114](/sites/all/themes/tud_repo3/img/icons/orcid_16x16.png)
Contributor
Tsumoto, Shusaku (editor)
Ohsawa, Yukio (editor)
Chen, Lei (editor)
Van den Poel, Dirk (editor)
Hu, Xiaohua (editor)
Motomura, Yoichi (editor)
Takagi, Takuya (editor)
Wu, Lingfei (editor)
Xie, Ying (editor)
Abe, Akihiro (editor)
Raghavan, Vijay (editor)
Department
Software Technology
Date
2022
Abstract
As machine learning models increasingly replace traditional business logic in the production system, their lifecycle management is becoming a significant concern. Once deployed into production, the machine learning models are constantly evaluated on new streaming data. Given the continuous data flow, shifting data, also known as concept drift, is ubiquitous in such settings. Concept drift usually impacts the performance of machine learning models, thus, identifying the moment when concept drift occurs is required. Concept drift is identified through concept drift detectors. In this work, we assess the reliability of concept drift detectors to identify drift in time by exploring how late are they reporting drifts and how many false alarms are they signaling. We compare the performance of the most popular drift detectors belonging to two different concept drift detector groups, error rate-based detectors and data distribution-based detectors. We assess their performance on both synthetic and real-world data. In the case of synthetic data, we investigate the performance of detectors to identify two types of concept drift, abrupt and gradual. Our findings aim to help practitioners understand which drift detector should be employed in different situations and, to achieve this, we share a list of the most important observations made throughout this study, which can serve as guidelines for practical usage. Furthermore, based on our empirical results, we analyze the suitability of each concept drift detection group to be used as an alarming system.
Subject
concept drift detection
machine learning lifecycle management
To reference this document use:
http://resolver.tudelft.nl/uuid:bd41e26b-0d7d-4e59-a55d-671a2a02b8c7
DOI
https://doi.org/10.1109/BigData55660.2022.10020292
Publisher
IEEE
Embargo date
2023-07-26
ISBN
978-1-6654-8046-8
Source
Proceedings of the 2022 IEEE International Conference on Big Data (Big Data)
Event
2022 IEEE International Conference on Big Data, 2022-12-17 → 2022-12-20, Osaka, Japan
Bibliographical note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
Institutional Repository
Document type
conference paper
Rights
© 2022 L. Poenaru-Olaru, Luis Cruz, A. van Deursen, Jan S. Rellermeyer