Quantifying the quality of coastal morphological predictions

Abstract

This thesis investigates the behaviour of the often
used point-wise skill score, the MSESSini a.k.a. BSS, and develops new
error metrics that, as opposed to point-wise metrics, take the spatial
structure of morphological patterns into account. The MSESSini measures
the relative accuracy of a morphological prediction over a prediction of
zero morphological change, using the mean-squared error (MSE) as the
accuracy measure. The main findings about the MSESSini are: 1) a generic
ranking, based on values for MSESSini, has limited validity, since the
zero change reference model fails to make model performance comparable
across different prediction situations; 2) the combination of larger,
persistent and smaller, intermittent scales of cumulative change may
lead to an increase of skill with time, without the prediction on either
of these scales becoming more skilful with time; 3) in the presence of
inevitable location errors, the MSESSini favours predictions that
underestimate the variance of cumulative bed changes and 4) existing
methods to correct for measurement error are inconsistent in either
their skill formulation or their suggested classification scheme. In
order to overcome the inherent limitations of point-wise metrics, three
novel diagnostic tools for the spatial validation of 2D morphological
predictions are developed. First, a field deformation or warping method
deforms the predictions towards the observations, minimizing the squared
point-wise error. Error measures are formulated based on both the
smooth displacement field between predictions and observations and the
residual point-wise error field after the deformation. In contrast with
the RMSE, the method captures the visual closeness of morphological
patterns. Second, an optimal transport method defines the distance
between predicted and observed morphological fields in terms of an
optimal sediment transport field. The optimal corrective transport field
moves the misplaced sediment from the predicted to the observed
morphology at the lowest quadratic transportation cost. The
root-mean-squared value of the optimal transport field, the RMSTE, is
proposed as a new error metric. As opposed to the field deformation
method, the optimal transport method is mass-conserving, parameter-free
and symmetric. The RMSTE, unlike the RMSE, is able to discriminate
between predictions that differ in the misplacement distance of
predicted morphological features. It also avoids the consistent reward
of the underestimation of morphological variability that the RMSE is
prone to. Third, a scale-selective validation approach allows any metric
to selectively address multiple spatial scales. It employs a smoothing
filter in such a way that, in addition to the domain-averaged
statistics, localized validation statistics and maps of prediction
quality are obtained per scale (geographic extent or areal size of
focus). The employed skill score weights how well the morphological
structure and variability are simulated, while avoiding to reward the
underestimation of variability. To fully describe prediction quality
multiple metrics are required with a weighting determined by the goal of
the simulation. Point-wise metrics should be supplemented with an error
decomposition, as to avoid undesired underestimation of variability.
Further, a set of performance metrics must include a metric, e.g. the
RMSTE, that accounts for the spatial structure of the observed and
predicted morphological fields.