Investigating Mutation-Guided Refactoring Using Large Language Models

Abstract

In legacy systems, changing existing software is risky when developers cannot easily understand or test the behavior of the code, which limits evolvability. Tests can reduce this risk, and with mutation testing, surviving mutants can indicate where test oracles can be strengthened and where potential observability issues in production code are situated. This thesis investigates to what extent LLM-guided refactoring, guided by mutation testing results, can increase the observability of production code while not decreasing readability. A case study is performed on two open-source Java projects, JFreeChart and Bukkit, for 12 classes in total, with an LLM that uses an execution-validation workflow. The mutation score increased in all runs. In JFreeChart, the average increase was 5.96%, while in Bukkit it was 47.89%. However, the results show that a higher mutation score does not always mean that production-code observability improved, because some mutants were killed by stronger tests for already observable behavior. The readability impact was limited overall. This suggests that mutation-guided LLM refactoring is most useful when surviving mutants reveal behavior that is genuinely difficult to observe, and when the refactoring exposes it at the intended level of observability. Thus, a surviving mutant should first be interpreted as a decision problem: it may require refactoring, stronger tests, or no change.