Leveraging E2E Test Context for LLM-Enhanced Test Data and Descriptions
Enhancing Automated Software Testing with Runtime Data Integration
M.C.A. de Wit (TU Delft - Electrical Engineering, Mathematics and Computer Science)
A. Deljouyi – Mentor (TU Delft - Software Engineering)
Andy Zaidman – Mentor (TU Delft - Software Technology)
A. Katsifodimos – Graduation committee member (TU Delft - Data-Intensive Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Automated software testing plays a critical role in improving software quality and reducing manual testing expenses. However, generating understandable and meaningful unit tests remains challenging, especially with frameworks optimized for coverage like Search-Based Software Testing (SBST). Large Language Models (LLMs) have the capability to generate human-like text, while capture/replay techniques can provide realistic data scenarios through trace logs, contributing to meaningful test case generation. This study introduces UTGen+, an approach that enhances LLM-based SBST by integrating trace logs from end-to-end tests, aiming to further improve test case understandability.
We conducted a comparative user study with 9 participants using UTGen+, original UTGen, and conventional SBST (EvoSuite), focusing on the effects of trace log inclusion on the naturalness and relevancy of comments, identifiers, and test data across several projects. The results indicated that while UTGen+ did not improve the naturalness and relevancy of comments and identifiers, it significantly enhanced the relevancy of test data. These findings suggest that incorporating contextual data can indeed benefit the generation of more relevant and understandable automated test cases.