Change Point Detection In Continuous Integration Performance Tests

Abstract

Software testing is an integral part of the development of embedded systems. Among other reasons, tests are frequently used to ensure that a system meets all the specifications, which is especially important when designing systems for the medical industry. Software changes that have a detrimental impact on a real-time system's performance can accumulate until the systems no longer meet the required performance levels. At this point, fixing accumulated defects can become a costly and complex endeavor. The goal of this thesis is to create a method that can detect changes in performance metrics from a continuous integration pipeline with minimal manual intervention.

To achieve this goal, we have created a novel online, univariate change detection method, which consists of a diverse ensemble of more than 10 existing change detection algorithms. The ensemble is robust against changes in distribution and requires little tuning. The contributions of this work include the proposal of a generic architecture to combine both statistics and decisions from individual algorithms. Related work uses simple majority voting for decision fusion in the ensemble - we demonstrate that an ensemble can benefit from more complex decision fusion, for example using a Random Forest.

Synthetic data and a case study, using a dataset provided by Philips, are used to demonstrate that the overall ensemble is consistently able to outperform the individual algorithms in the ensemble. In addition, unlike the individual algorithms, the ensemble generalizes well to data that is not normally distributed and have not been encountered during training. Compared to the monitoring system for performance metrics that is currently being used by Philips, the ensemble is able to detect 75\% more changes with a 50\% lower false positive rate.