Property-Based Testing in Haskell
An Analysis of QuickCheck usage in Open-Source Haskell Projects
Y. Zhao (TU Delft - Electrical Engineering, Mathematics and Computer Science)
S. Juhošová – Mentor (TU Delft - Programming Languages)
M.A. Costea – Mentor (TU Delft - Programming Languages)
Marco Zúñiga Zuñiga Zamalloa – Graduation committee member (TU Delft - Networked Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Property-Based Testing (PBT) with QuickCheck has become a cornerstone of reliable software development in Haskell, yet there is little systematic understanding of how developers employ it in production quality libraries. In this study, we perform an empirical analysis of QuickCheck usage in nine representative open-source Haskell projects (aeson, attoparsec, containers, hashable, lens, megaparsec, pandoc-type, text and vector). We extract thousands of
QuickCheck properties and manually sample 217 of them to classify property types such as invariants, roundtrip checks, algorithm correctness, and idempotence. We also analyze all extracted properties to identify patterns in generator and shrinking strategies. Our Results show that invariants and test-oracle tests dominate the sampled properties, while custom generators are used in 55.8% of all properties and custom shrinkers in only 25%. We discuss how these patterns vary across different domains (data structures, text processing, optics, hashing) and highlight the tension between default and custom test infrastructure. Finally, we reflect on the implications for tool support, education, and future research particularly automatic generator inference and improved shrinking utilities to lower
the barrier to effective PBT in large codebases.