How Reproducible Are Build and Test Outcomes of Dependency Updates in JavaScript/npm Projects?
An Empirical Study of Dependabot npm Update Pull Requests
V. Maksymiuk (TU Delft - Electrical Engineering, Mathematics and Computer Science)
C.R. Paulsen – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
S. Proksch – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
J.A. Pouwelse – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Automated dependency-update tools such as Dependabot help projects keep dependencies secure and compatible, and CI results give maintainers practical evidence for deciding whether such updates can be merged, delayed, or require closer inspection. This evidence is useful only if the same update state tends to produce the same observable outcome when evaluated again. Prior work has studied reproducible dependency-update failures in ecosystems such as Java/Maven; this thesis examines whether similar CI-like evidence can be obtained for JavaScript/npm projects, where lockfiles, package-manager behavior, browser tooling, native addons, and peer dependencies may introduce instability.
Building on prior work on automated dependency updates and reproducible
dependency-update benchmarks, this study evaluates 2,777 npm dependency-update pull requests from 3,083 Dependabot candidates. The results show that most outcomes are reproducible, but 152 experiments (4.8\%) and 111 PRs (4.0\%) arenon-reproducible. Instability is concentrated in tests and appears more often in browser-coupled, native-addon-risk, peer-dependency-risk, and Yarn-based projects. These findings suggest that CI remains useful evidence for npm dependency-update decisions, but practitioners and researchers should treat single-run outcomes as incomplete evidence in instability-prone settings.