Print Email Facebook Twitter A retrospective assessment of COVID-19 model performance in the USA Title A retrospective assessment of COVID-19 model performance in the USA Author Colonna, Kyle J. (Harvard University) Nane, G.F. (TU Delft Applied Probability) Choma, Ernani F. (Harvard University) Cooke, R.M. (TU Delft Applied Probability; Resources for the Future) Evans, John S. (Harvard University) Date 2022 Abstract Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that - (i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling. Subject Cooke's classical modelCOVID-19decision sciencesensemble modellingforecastinguncertainty analysis To reference this document use: http://resolver.tudelft.nl/uuid:0ef6f46f-e8d3-40a3-b640-ff0123bd082e DOI https://doi.org/10.1098/rsos.220021 ISSN 2054-5703 Source Royal Society Open Science, 9 (10) Part of collection Institutional Repository Document type journal article Rights © 2022 Kyle J. Colonna, G.F. Nane, Ernani F. Choma, R.M. Cooke, John S. Evans Files PDF rsos.220021.pdf 929.07 KB Close viewer /islandora/object/uuid:0ef6f46f-e8d3-40a3-b640-ff0123bd082e/datastream/OBJ/view