The deceptive simplicity of the brier skill score

Abstract

The quality of morphodynamic predictions is often indicated by a skill score that weighs the mean-squared error of the prediction by that of the initial bed as the reference prediction. As simple as this Brier skill score (BSS) or meansquared- error skill score (MSESS) may seem, it is not well understood and, hence, sometimes misinterpreted. This chapter aims at improving the understanding of the MSESS. We review existing MSESS formulations and classifications, with and without an account of the measurement error. Using simple examples, we illuminate which aspects of prediction quality the MSESS actually measures. It is shown that the MSESS tends to favor model results that underestimate the variance of cumulative bed changes. We further demonstrate that the normalization by the observed cumulative change, which follows from the choice of the initial bed as the reference, is not effective in creating a level playing field over a wide range of prediction situations (trend, episodic event, different seasons). Also, it is shown that the combined presence of larger, persistent scales and smaller, intermittent scales in the cumulative bed changes may lead to an apparent increase of skill with time, although the prediction of neither of these scales becomes more skilful with time. Finally, in order to obtain a balanced appreciation of model performance, the use and development of a more extensive suite of validation measures is advocated.