How can a human-in-the-loop feedback mechanism improve the accuracy and reliability of medical text annotations generated by LLM-based agents?

Abstract

Large language models evaluated on medical text annotation tasks are typically scored against a single reference label, such as the primary billed ICD-9 code in MIMIC-III. This standard imposes a measurement ceiling on strict accuracy: it conflates genuine clinical extraction failure with cases where the model extracted a valid clinical comorbidity rather than the administratively billed primary code. To address this, we propose a dual-metric evaluation protocol that pairs strict accuracy (matching the primary billed code) with lenient accuracy (matching any billed code for the admission), scoped to clinical ICD-9 chapters, to separate model capability from billing-convention noise. Exercising the protocol through a human-in-the-loop (HITL) framework, we demonstrate an 18 percentage-point gap between strict (0.67) and lenient (0.85) accuracy on a human-reviewed baseline (n = 100); direct on-corpus analysis further shows that approximately 10% of discharge summaries (up to 14% in early iterations) contain gold-label-ambiguous coding artefacts rather than recoverable errors. On the same HITL framework, lenient accuracy converges to a tight [0.86, 0.90] band by iteration 3 across five prompt-engineering conditions, a cross-condition convergence we read as a framework-level reliability gain, and a human-reviewer trajectory shows strict accuracy trending from 0.62 → 0.78 under rich free-text feedback (a directional signal, confounded with batch difficulty at this sample size, that motivates the higher-powered replication we propose). Future LLM-based ICD-coding evaluations on MIMIC-III should adopt the dual-metric protocol so that model capability and billing-convention noise are reported separately.