Is every expert equal?

Is every expert equal?: An analysis of the differences in performance in structured expert judgement

Harkema, Jan (TU Delft Electrical Engineering, Mathematics and Computer Science)

Nane, G.F. (mentor)
Cooke, R.M. (graduation committee)
Spandaw, J.G. (graduation committee)

Delft University of Technology

Electrical Engineering

2022-01-17

In this thesis the differences in performance scores of experts in the Classical Model for structured expert judgement are analyzed. The underlying assumption in the Classical Model is that variance in performances of experts in a panel is at least partly resultant of the expert's ability to quantify uncertainty. This assumption is tested against the so called Random Expert Hypothesis, that states that these differences are solely resultant of random fluctuations. At the five percent significance level it is concluded that the variation in the combined score of experts cannot exclusively be explained by random fluctuations. When the assumption is tested individually for three different subject fields, health, policy and science, the Random Expert Hypothesis cannot be rejected for both health and policy related studies. Lastly it is shown that the variation in performances between the best and worst expert in a panel strongly correlates with the performance of the best expert against random panels. This indicates that the aggregation of experts according to the scoring rule in the Classical Model may primarily work to diminish the influence of low performing experts.

Structured Expert Judgement
SEJ
expert judgement
Random Expert Hypothesis

http://resolver.tudelft.nl/uuid:d17c7ced-70bd-42cf-92b9-9709c1dcb649

Student theses

bachelor thesis

© 2022 Jan Harkema