Are AI Agents for Code Robust to Metamorphic Transformations?

Abstract

Autonomous coding agents such as Claude Code and Codex are now used broadly to write code across entire repositories. They rely on large language models (LLMs), which are known to be sensitive to small behavior-preserving changes in their input. Renaming a variable or reordering code can reduce the accuracy of a standalone model, even though the program's functionality remains the same. It is not clear whether agents have this same weakness, since they can explore the codebase, run tests, and correct any issues through a feedback loop.

This work studies the robustness of coding agents to metamorphic transformations applied across the repository. We use three behavior-preserving transformations that rename identifiers, restructure control flow, and reorder methods. We transform 25 instances from the SWE-rebench-V2 dataset and run Claude Code and Codex on both the original and transformed versions.

The results show that coding agents appear more robust than standalone LLMs. Still, both agents tend to explore the codebase more in the transformed version. For Claude Code, we see an increase in the success rate, while Codex shows greater run-to-run variability in the input tokens, making it less predictable. We read these trends as evidence that the agents are relying less on memorized patterns and more on freshly-read code. Still, this tendency toward robustness should be considered a practical observation, not definitive proof. The run-to-run variance of the agents is large, often larger than the effect we look for, so we can only show we don't see a difference, but cannot prove there is none. So, to answer the question in our title: in practice, most likely robust.