Understanding Software Quality Assurance in Open-Source Communities: Awareness, Adoption, and Tool Utilization
Ali Khatami (TU Delft - Electrical Engineering, Mathematics and Computer Science, TU Delft - Electrical Engineering, Mathematics and Computer Science)
A.E. Zaidman – Promotor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
C.E. Brandt – Copromotor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Today's software development ecosystem offers an unprecedented array of quality assurance mechanisms. Continuous integration (CI) platforms, automated testing frameworks, automated static analysis tools, and code review systems are now readily available to developers, particularly in open-source software (OSS) projects hosted on platforms like GitHub. Yet this abundance does not automatically translate into effective adoption. Instead, it introduces a new challenge: the rapid expansion of Software Quality Assurance (SQA) practices and tools has created a complex decision space where teams must determine which practices to adopt, how to configure them effectively, how to integrate them into development workflows, and how to design processes that are relevant for their specific project context. This dissertation aims to construct a holistic understanding of software quality assurance by bringing together different aspects of open-source SQA practices that can inform future decision-making.
We begin by establishing an empirical understanding of how SQA practices are currently employed in the OSS world. Projects tend to adopt practices unevenly across dimensions, while broader investment across SQA practices is more common among mature projects. This raises a deeper question about whether developers actually understand the mechanisms already present in their projects. Our survey of open-source maintainers and contributors confirms this concern, uncovering an awareness gap: while developers broadly know that practices such as CI and testing are in place, their awareness is more limited when asked about specific mechanisms and tools.
To address the awareness gap, we design and evaluate \textsc{RepoInsights}, a software quality assurance analytics dashboard consolidating data from testing, code review, and automated workflow activities. While participants value the overview, they express a need for richer and more fine-grained contextual information. This motivates a deeper investigation into GitHub Actions (GHA), a platform deeply integrated into the GitHub ecosystem and a promising source of such contextual depth.
We investigate GitHub Actions through two complementary lenses, revealing two further findings:
workflow configurations suffer from recurring quality issues that require active maintenance, and configured workflows are not necessarily used in practice, as configuration does not imply actual utilization.
Taken together, our five studies contribute toward a deeper understanding of how software quality assurance operates in open-source software development on GitHub.
The empirical findings and conceptual contributions of this dissertation, including the awareness gap, the configuration–usage gap, and the characterization of developer responses to GHA workflow failures, contribute to the holistic understanding needed to navigate the complex SQA decision space in open-source software development.