CI told you: Exploring the role of testing strategies as part of CI pipelines and their impact on DevOp metrics in Open Source projects
K.Z. Panayotov (TU Delft - Electrical Engineering, Mathematics and Computer Science)
S. Huang – Mentor (TU Delft - Software Engineering)
Sebastian Proksch – Mentor (TU Delft - Software Engineering)
Marco A. Zuñiga Zamalloa – Graduation committee member (TU Delft - Networked Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Continuous Integration is a used extensively in modern software engineering for both proprietary and open-source projects. Many studies have studied its benefits and drawbacks, finding how it increases development productivity and stability. However, CI is a set of practices from static linting to calculating the code coverage of the underlying test suites. In order to choose whether to make use of that technology or to evaluate the overall performance of a project’s development, practitioners make use of certain measurements, DevOps metrics being one of the most significant ones. We aim to analyse the effects of testing strategies within the CI over a set of DevOps metrics. This is done by collecting over 5778608 executions of GitHub CI workflows that involve a test-running step from 476 open-source projects. We see that 69.48% of runs happen after a pull request or a push. In the end, we found that frequent CI test execution didn’t increase the project’s DevOps metrics, indicating that developers should try limiting the unnecessary execution of tests to save on resources. Further we see lack of Pearson statistical significance for the correlation between coverage in CI and the metrics in the smaller set of selected projects.