LLM-Orchestrated Iterative Retrieval for Contract Clause Extraction

Abstract

Legal contract review requires practitioners to find specific provisions
in documents that can span hundreds of pages, and errors in this task
carry direct legal consequences.
Standard retrieve-and-extract pipelines reduce hallucination by grounding
language model outputs in retrieved document text, but their single
retrieval step cannot follow cross-references, check whether the gathered
evidence is complete, or reliably confirm that a clause is absent.
Whether having a language model actively direct each retrieval step,
rather than passively reading a fixed set of retrieved chunks, improves
clause extraction accuracy has not, to the best of our knowledge,
been tested.
This thesis provides a controlled comparison of retrieve-and-extract
and LLM-orchestrated iterative retrieval on the Contract Understanding
Atticus Dataset (CUAD), keeping all components except the retrieval
architecture constant and evaluating both systems across all 38 clause
categories under Exact Match, token-level F1, and ROUGE-L.
Iterative retrieval consistently outperforms single-pass retrieval.
The main experiment, a 5427-pair run on Gemma4~e4b, a 4-billion-parameter
model run locally via Ollama, improves Exact Match from 0.687 to 0.744, Token
F1 from 0.341 to 0.505, and ROUGE-L from 0.308 to 0.467, with every gain
statistically significant under paired tests.
A smaller balanced run on both models confirms this and shows the gain again on
the larger closed model GPT-5-mini, which improves from Exact Match 0.711 to
0.800, Token F1 0.314 to 0.538, and ROUGE-L 0.283 to 0.490.
The F1 gains substantially exceed the Exact Match gains in every case,
indicating that iterative retrieval improves the completeness of
extracted clause text, not only presence-or-absence decisions.
The comparison isolates the contribution of LLM involvement at the
retrieval step in a domain and evaluation setting where, to the best
of our knowledge, it has not previously been measured.