Mind the Gap
What Working With Developers on Fuzz Tests Taught Us About Coverage Gaps
C.E. Brandt (TU Delft - Software Engineering)
Marco Castelluccio (Mozilla Corporation)
Christian Holler (Mozilla Corporation)
Jason Kratzer (Mozilla Corporation)
A.E. Zaidman (TU Delft - Software Technology)
Alberto Bacchelli (Universitat Zurich)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Can fuzzers generate partial tests that developers find useful enough to complete into functional tests (e.g., by adding assertions)? To address this question, we develop a prototype within the Mozilla ecosystem and open 13 bug reports proposing partial generated tests for currently uncovered code. We found that the majority of the reactions focus on whether the targeted coverage gap is actually worth testing. To investigate further which coverage gaps developers find relevant to close, we design an automated filter to exclude irrelevant coverage gaps before generating tests. From conversations with 13 developers about whether the remaining coverage gaps are worth closing when a partially generated test is available, we learn that the filtering indeed removes clearly non-test-worthy gaps. The developers propose a variety of additional strategies to address the coverage gaps and how to make fuzz tests and reports more useful for developers.
help_outline