Scaling Deliberative-Quality Measurement with Large Language Models
Validating a Four-Model LLM Ensemble as a Coder for a Theory-Justified Three-Dimension Discourse Quality Index Sub-Codebook on UK House of Commons Debate
F. Liu (TU Delft - Electrical Engineering, Mathematics and Computer Science)
W.P. Brinkman – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
M.N.J. Grauwde – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Evaluating the quality of public deliberation is a prerequisite for governing it on evidence rather than impression, yet the bottleneck is measurement. The Discourse Quality Index (DQI), the standard instrument in the field, requires trained human coders — slow, costly, and applied inconsistently across research teams — which means most empirical deliberation studies contain only a few thousand speeches. Scaling that measurement is the problem this paper addresses. Large language models (LLMs) can apply a structured rubric rapidly and uniformly, so the central question is whether an LLM coder is reliable enough on theory-grounded deliberative constructs to stand in for a trained human and raise the data-size ceiling. We make
two contributions. First, we construct a theory-justified three-dimension DQI sub- codebook — Level of Justification, Respect for Groups, and Counterarguments — defending each inclusion and exclusion from deliberative-democracy theory. Second,
we benchmark a four-model LLM-as-judge ensemble against two trained coders on
200 UK House of Commons public-safety debate acts. We report Gwet’s linear- weighted AC1: trained coders reach AC1 = 0.78, 0.94, and 0.48 on the three dimensions, and the ensemble reaches 0.74, 0.89, and 0.48 against the lead coder.
We also tested whether an LLM can substitute for a human coder, framed as an equivalence test: whether the paired difference in agreement falls within a ±0.10 band of the human–human baseline, judged by where its confidence interval lies rather than by a test against zero. The difference falls within 0.05 of the baseline
on every dimension, and its confidence interval lies inside the band on all three.
The results suggest that on justification and recognition the ensemble can serve as reliable independent measurement, and that on Counterarguments it can serve as a second coder in a doubly-coded design — where the binding constraint appears to
be the codebook anchor language rather than the model.