How does LLM-based test generation for Java libraries perform when full source code is available?

Abstract

As software projects depend heavily on open-source libraries, developers use tests to ensure that dependency updates remain behaviourally compatible. However, such library tests are often incomplete or unavailable. Although automated test generation tools such as EvoSuite exist and Large Language Models (LLMs) have shown promise in generating more readable tests, most evaluations have been conducted on benchmark datasets or popular GitHub projects. This creates a gap in understanding how effective LLM-generated tests are for released library artifacts. In this paper, we evaluate LLM-based test generation for released Java libraries from Maven Central to assess its feasibility in dependency validation workflows. We implement a pipeline that provides source code and method context to a locally hosted LLM, validates generated tests, and applies iterative repair when needed. Our results show that tests generated by the local model achieve substantially lower coverage than EvoSuite, primarily due to compilation failures, highlighting that symbol resolution errors remain a key challenge in generating tests with LLMs. We further show that iterative repair is effective at improving the coverage of generated tests and a stronger cloud-hosted model even surpasses EvoSuite in coverage. Overall, the findings indicate that LLM-based test generation from source code is a promising approach for dependency update validation when combined with sufficiently capable models and iterative repair mechanisms.