AD
A. Drăgoi
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Master thesis
(2026)
-
Andrei Drăgoi, A. Panichella, M.J.G. Olsthoorn, Pouria Derakhshanfar, J. Yang
Autonomous coding agents such as Claude Code and Codex are now used broadly to write code across entire repositories. They rely on large language models (LLMs), which are known to be sensitive to small behavior-preserving changes in their input. Renaming a variable or reordering code can reduce the accuracy of a standalone model, even though the program's functionality remains the same. It is not clear whether agents have this same weakness, since they can explore the codebase, run tests, and correct any issues through a feedback loop.
This work studies the robustness of coding agents to metamorphic transformations applied across the repository. We use three behavior-preserving transformations that rename identifiers, restructure control flow, and reorder methods. We transform 25 instances from the SWE-rebench-V2 dataset and run Claude Code and Codex on both the original and transformed versions.
The results show that coding agents appear more robust than standalone LLMs. Still, both agents tend to explore the codebase more in the transformed version. For Claude Code, we see an increase in the success rate, while Codex shows greater run-to-run variability in the input tokens, making it less predictable. We read these trends as evidence that the agents are relying less on memorized patterns and more on freshly-read code. Still, this tendency toward robustness should be considered a practical observation, not definitive proof. The run-to-run variance of the agents is large, often larger than the effect we look for, so we can only show we don't see a difference, but cannot prove there is none. So, to answer the question in our title: in practice, most likely robust. ...
This work studies the robustness of coding agents to metamorphic transformations applied across the repository. We use three behavior-preserving transformations that rename identifiers, restructure control flow, and reorder methods. We transform 25 instances from the SWE-rebench-V2 dataset and run Claude Code and Codex on both the original and transformed versions.
The results show that coding agents appear more robust than standalone LLMs. Still, both agents tend to explore the codebase more in the transformed version. For Claude Code, we see an increase in the success rate, while Codex shows greater run-to-run variability in the input tokens, making it less predictable. We read these trends as evidence that the agents are relying less on memorized patterns and more on freshly-read code. Still, this tendency toward robustness should be considered a practical observation, not definitive proof. The run-to-run variance of the agents is large, often larger than the effect we look for, so we can only show we don't see a difference, but cannot prove there is none. So, to answer the question in our title: in practice, most likely robust. ...
Autonomous coding agents such as Claude Code and Codex are now used broadly to write code across entire repositories. They rely on large language models (LLMs), which are known to be sensitive to small behavior-preserving changes in their input. Renaming a variable or reordering code can reduce the accuracy of a standalone model, even though the program's functionality remains the same. It is not clear whether agents have this same weakness, since they can explore the codebase, run tests, and correct any issues through a feedback loop.
This work studies the robustness of coding agents to metamorphic transformations applied across the repository. We use three behavior-preserving transformations that rename identifiers, restructure control flow, and reorder methods. We transform 25 instances from the SWE-rebench-V2 dataset and run Claude Code and Codex on both the original and transformed versions.
The results show that coding agents appear more robust than standalone LLMs. Still, both agents tend to explore the codebase more in the transformed version. For Claude Code, we see an increase in the success rate, while Codex shows greater run-to-run variability in the input tokens, making it less predictable. We read these trends as evidence that the agents are relying less on memorized patterns and more on freshly-read code. Still, this tendency toward robustness should be considered a practical observation, not definitive proof. The run-to-run variance of the agents is large, often larger than the effect we look for, so we can only show we don't see a difference, but cannot prove there is none. So, to answer the question in our title: in practice, most likely robust.
This work studies the robustness of coding agents to metamorphic transformations applied across the repository. We use three behavior-preserving transformations that rename identifiers, restructure control flow, and reorder methods. We transform 25 instances from the SWE-rebench-V2 dataset and run Claude Code and Codex on both the original and transformed versions.
The results show that coding agents appear more robust than standalone LLMs. Still, both agents tend to explore the codebase more in the transformed version. For Claude Code, we see an increase in the success rate, while Codex shows greater run-to-run variability in the input tokens, making it less predictable. We read these trends as evidence that the agents are relying less on memorized patterns and more on freshly-read code. Still, this tendency toward robustness should be considered a practical observation, not definitive proof. The run-to-run variance of the agents is large, often larger than the effect we look for, so we can only show we don't see a difference, but cannot prove there is none. So, to answer the question in our title: in practice, most likely robust.
Exploring Test Suite Coverage of Large Language Model–Enhanced Unit Test Generation
A Study on the Ability of Large Language Models to Improve the Understandability of Generated Unit Tests Without Compromising Coverage
Automated software testing is a frequently studied topic in specialized literature. Search-based software testing tools, like EvoSuite, can generate test suites using genetic algorithms without the developer’s input. Large Language Models (LLMs) have recently attracted significant attention in the software engineering domain for their potential to automate test generation. UTGen, a tool integrating LLMs with EvoSuite, produces more understandable tests than EvoSuite; however, the generated tests suffer a coverage drop.
To streamline bug detection by developers, we propose UTGenCov, a concept that focuses on improving the understandability of EvoSuite-generated tests without compromising on coverage. This approach builds upon UTGen by thoroughly analyzing the reasons behind the decrease in coverage and proposing an alternative approach.
Our investigation determined that the leading cause of coverage reduction in UTGen is LLM hallucination in the Understandability phase. UTGenCov aims to address hallucinations by providing the source code of the methods used in the test to the LLM. Yet, our experiment results indicate inconsistent performance and a further decrease in branch coverage of 0.74% compared to UTGen. ...
To streamline bug detection by developers, we propose UTGenCov, a concept that focuses on improving the understandability of EvoSuite-generated tests without compromising on coverage. This approach builds upon UTGen by thoroughly analyzing the reasons behind the decrease in coverage and proposing an alternative approach.
Our investigation determined that the leading cause of coverage reduction in UTGen is LLM hallucination in the Understandability phase. UTGenCov aims to address hallucinations by providing the source code of the methods used in the test to the LLM. Yet, our experiment results indicate inconsistent performance and a further decrease in branch coverage of 0.74% compared to UTGen. ...
Automated software testing is a frequently studied topic in specialized literature. Search-based software testing tools, like EvoSuite, can generate test suites using genetic algorithms without the developer’s input. Large Language Models (LLMs) have recently attracted significant attention in the software engineering domain for their potential to automate test generation. UTGen, a tool integrating LLMs with EvoSuite, produces more understandable tests than EvoSuite; however, the generated tests suffer a coverage drop.
To streamline bug detection by developers, we propose UTGenCov, a concept that focuses on improving the understandability of EvoSuite-generated tests without compromising on coverage. This approach builds upon UTGen by thoroughly analyzing the reasons behind the decrease in coverage and proposing an alternative approach.
Our investigation determined that the leading cause of coverage reduction in UTGen is LLM hallucination in the Understandability phase. UTGenCov aims to address hallucinations by providing the source code of the methods used in the test to the LLM. Yet, our experiment results indicate inconsistent performance and a further decrease in branch coverage of 0.74% compared to UTGen.
To streamline bug detection by developers, we propose UTGenCov, a concept that focuses on improving the understandability of EvoSuite-generated tests without compromising on coverage. This approach builds upon UTGen by thoroughly analyzing the reasons behind the decrease in coverage and proposing an alternative approach.
Our investigation determined that the leading cause of coverage reduction in UTGen is LLM hallucination in the Understandability phase. UTGenCov aims to address hallucinations by providing the source code of the methods used in the test to the LLM. Yet, our experiment results indicate inconsistent performance and a further decrease in branch coverage of 0.74% compared to UTGen.