Elicitation in the classical model

Abstract

The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according to metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to determine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.