A Monitoring System for Machine Learning Models in a Large-Scale Context

Abstract

Since building a machine learning model costs a lot while following 9 stages, the automated machine learning model creation became a crucial role in a large-scale context. At the same time, a monitoring system became an essential factor for machine learning models. This thesis presents the monitoring system for machine learning models at ING in an enterprise context with new features required by users. Moreover, the thesis describes a case study of ING, a large global banking company that develops software solutions in-house. We conducted a mixed-methods study, consisting of data collection of the monitoring system and a survey with the users of the monitoring system. Our research shows that challenges found by the actual users of the monitoring system and mapped challenges discovered by the Microsoft study are related to machine learning model monitoring, the perception of the users on the importance of the monitoring system, and the impact of the monitoring system. We found that the monitoring system at ING supports relatively efficient model management in terms of checking model validation and evaluation. Moreover, the users of the monitoring system perceived that it is an important system, and it supports the models regarding quality, the trust of the automated model creation, and usability. Additionally, compared to the existing solution, the monitoring system at ING supports useful model management.