DN
Dan Nguyen Le Kha Dan
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
1 records found
1
Spectrum-based Fault Localization for LLM-based Multi-Agent Systems
Identifying Faulty Agent Roles through Spectrum Analysis of Execution Traces
Bachelor thesis
(2026)
-
Dan Nguyen Le Kha Dan, B. Özkan, A. Panichella, Z. Seyedghorban, M.T.J. Spaan
Large Language Model-based Multi-Agent Systems (LLM-MAS) are promising frameworks for automating complex, real-world tasks. However, when these systems fail, failure attribution is challenging due to the stochastic behavior of Large Language Models (LLMs) and the distributed decision-making process of multi-agent collaboration. This paper investigates whether Spectrum-based Fault Localization (SBFL), a well established technique in software testing and debugging, can be applied to identify faulty agent roles in LLM-MAS. We evaluate SBFL on HyperAgent across five SWE-bench Verified tasks, defining the spectra based on role message frequency and semantic output overlap. Agent roles are ranked by their computed suspiciousness scores and fault localization performance is measured using Top-1 and Top-3 accuracy against ground truth labels established by an LLM-as-a-judge. Our results show that semantic output overlap achieves the highest Top-1 accuracy of 60%, consistently outperforming raw message frequency spectra. However, none of the evaluated spectrum representations produces reliable Top-3 rankings, no SBFL formula consistently outperforms the others, and adding more execution runs does not consistently improve fault localization performance. These findings suggest that SBFL can support role level fault localization in LLM-MAS, but its effectiveness depends strongly on spectrum design and remains limited for reliably identifying multiple faulty roles.
...
Large Language Model-based Multi-Agent Systems (LLM-MAS) are promising frameworks for automating complex, real-world tasks. However, when these systems fail, failure attribution is challenging due to the stochastic behavior of Large Language Models (LLMs) and the distributed decision-making process of multi-agent collaboration. This paper investigates whether Spectrum-based Fault Localization (SBFL), a well established technique in software testing and debugging, can be applied to identify faulty agent roles in LLM-MAS. We evaluate SBFL on HyperAgent across five SWE-bench Verified tasks, defining the spectra based on role message frequency and semantic output overlap. Agent roles are ranked by their computed suspiciousness scores and fault localization performance is measured using Top-1 and Top-3 accuracy against ground truth labels established by an LLM-as-a-judge. Our results show that semantic output overlap achieves the highest Top-1 accuracy of 60%, consistently outperforming raw message frequency spectra. However, none of the evaluated spectrum representations produces reliable Top-3 rankings, no SBFL formula consistently outperforms the others, and adding more execution runs does not consistently improve fault localization performance. These findings suggest that SBFL can support role level fault localization in LLM-MAS, but its effectiveness depends strongly on spectrum design and remains limited for reliably identifying multiple faulty roles.