Continuous Mutation Testing in Modern Software Development

Abstract

Modern software is being built in a continuously integrated fashion, in order to overcome the challenges that come with developing large software systems from many contributors. The cornerstone of continuous integration is the testing step, since it is supposed to protect the system from changes that might disrupt correct behavior. Mutation testing is a method that checks the fault finding capability of a test suite. Current CI settings do not implement a step that checks how thorough the test suite is.

Therefore, the goal of this thesis has been to explore how mutation testing can be applied to changes under analysis in a continuous integration setting. Since there is no infrastructure to support this, in order to conduct our study we developed OPi+, a prototype tool for experimenting the infrastructure required for a continuous mutation testing approach. Using real-world systems for analysis, we give initial evidence of the continuous mutation testing usefulness in terms of costs and benefits when applied to realistic software changes. The empirical study is based on analysis performed on the entire commit history of the popular open source Java Maven systems.

Through our study we defined 5 types of outcomes together with a continuous mutation testing behavior flow and additional analysis that streamlines current mutation testing practices.
We showed not only that mutation testing in a CI environment requires significantly fewer resources but they are also within the limits required by a CI pipeline. Through our study we also identify unmutable code for which we propose appropriate unimplemented operator set. We also study the evolution of surviving mutants with regards to their impact on the systems` technical debt.

In our study, we showed initial evidence that mutation testing can successfully be made compatible with a CI environment. We therefore propose a few ideas that could possibly further streamline continuous mutation testing.