Oops, My Tests Broke the Build

An Explorative Analysis of Travis CI with GitHub

Conference paper (2017)

Authors

M.M. Beller Software Engineering - EEMCS
G. Gousios Software Engineering - EEMCS
A.E. Zaidman Software Engineering - EEMCS
Research Group
Software Engineering (EEMCS) (TU Delft)
Copyright: © 2017 M.M. Beller, G. Gousios, A.E. Zaidman
DOI
help_outline
https://doi.org/10.1109/MSR.2017.62
Java Software Programming Testing Best practices Tools
More Info
expand_more

Published Date

2017

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Software Technology

Research Group

Software Engineering

Abstract

AbContinuous Integration (CI) has become a best practice of modern software development. Yet, at present, we have a shortfall of insight into the testing practices that are common in CI-based software development. In particular, we
seek quantifiable evidence on how central testing is to the CI process, how strongly the project language influences testing, whether different integration environments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on
TRAVIS CI, we find that testing is the single most important reason why builds fail. Moreover, the programming language has a strong influence on both the number of executed tests, their run time, and proneness to fail. The use of multiple integration environments leads to 10% more failures being caught at build time. However, testing on TRAVIS CI does not seem an adequate
surrogate for running tests locally in the IDE. To further research on TRAVIS CI with GITHUB, we introduce TRAVISTORRENT.

Files

BellerMSR2017.pdf
(.pdf | 0.554 Mb)