Foodborne pathogens represent a significant public health burden. Quantifying the relative importance of various potential sources of foodborne infection is challenging due to data scarcity and uncertainty in empirical studies. Structured expert judgment (SEJ) provides a valuable methodological alternative to gain insights into source attribution of foodborne pathogens. We conducted a SEJ study to attribute human cases of 26 foodborne pathogens in the Netherlands to seven major transmission pathways, 20 food groups, and two animal groups, in a typical post-COVID19 year, using Cooke's classical model. The elicitation process involved snowball recruitment, expertise self-assessment, and a workshop where experts answered calibration questions to capture their uncertainty as input for the model. Subsequently, experts completed the target questions to obtain attributable proportions at the ‘kitchen door’ level. Results indicated that transmission was predominantly (>50%) foodborne for Staphylococcus aureus , Listeria monocytogenes , Yersinia spp., Bacillus cereus , Clostridium perfringens , certain non-typhoidal Salmonella serotypes, Campylobacter spp., hepatitis E virus and Toxoplasma gondii , whereas person-to-person transmission was the primary pathway for astrovirus, rotavirus, norovirus, and sapovirus. Brucella spp. and typhoidal Salmonella were attributed primarily (>85%) to international travel. All other pathogens showed attributions of <50% to any individual pathway. Substantial differences were observed when dividing foodborne transmission into food groups. Key contributors included food handlers and vermin, various meats (e.g., pork, beef, chicken), and shellfish. These SEJ-derived estimates complement existing data by providing pathogen-specific insights in the Dutch context.

Robust estimates of the global burden of foodborne diseases are essential for guiding food safety policy and prioritizing interventions. The release of the first World Health Organization (WHO) global burden estimates in 2015 (WHO, 2015; Havelaar et al., 2015) represented a major step toward quantifying the public health impact of contaminated food and highlighted substantial gaps in both surveillance data and knowledge about how different hazards are transmitted. To continue strengthening the evidence base for food safety decision-making, WHO reconvened the Foodborne Disease Burden Epidemiology Reference Group (FERG) in 2021 with a mandate to update the global estimates, expand analyses to the national level, and develop indicators for monitoring progress in reducing foodborne disease (WHO, 2022)...

This study evaluates five scoring rules, or measures of statistical accuracy, for assessing uncertainty estimates from expert judgment studies and model forecasts. These rules — the Continuously Ranked Probability Score ((Formula presented.)), Kolmogorov-Smirnov ((Formula presented.)), Cramer-von Mises ((Formula presented.)), Anderson Darling ((Formula presented.)), and chi-square test — were applied to 6864 expert uncertainty estimates from 49 Classical Model (CM) studies. We compared their sensitivity to various biases and their ability to serve as performance-based weight for expert estimates. Additionally, the piecewise uniform and Metalog distribution were evaluated for their representation of expert estimates because four of the five rules require interpolating the experts' estimates. Simulating biased estimates reveals varying sensitivity of the considered test statistics to these biases. Expert weights derived using one measure of statistical accuracy were evaluated with other measures to assess their performance. The main conclusions are (1) (Formula presented.) overlooks important biases, while chi-square and (Formula presented.) behave similarly, as do (Formula presented.) and (Formula presented.). (2) All measures except (Formula presented.) agree that performance weighting is superior to equal weighting with respect to statistical accuracy. (3) Neither distributions can effectively predict the position of a removed quantile estimate. These insights show the behavior of different scoring rules for combining uncertainty estimates from expert or models, and extent the knowledge for best-practices.

A review of scoring rules highlights the distinction between rewarding honesty and rewarding quality. This motivates the introduction of a scale-invariant version of the Continuous Ranked Probability Score (CRPS) which enables statistical accuracy (SA) testing based on an exact rather than an asymptotic distribution of the density of convolutions. A recent data set of 6761 expert probabilistic forecasts for questions for which the actual values are known is used to compare performance. New insights include that (a) variance due to assessed variables dominates variance due to experts, (b) performance on mean absolute percentage error (MAPE) is weakly related to SA (c) scale-invariant CRPS combinations compete with the Classical Model (CM) on SA and MAPE, and (d) CRPS is more forgiving with regard to SA than the CM as CRPS is insensitive to location bias.

The work presented here marks a further advance in expert uncertainty quantification. In a recent probabilistic evaluation of ice sheet process contributions to sea level rise, tail dependence was elicited and propagated through an uncertainty analysis for the first time. The elicited correlations and tail dependencies concerned pairings of three processes: Accumulation, Discharge and Run-off, which operate on major ice sheets in the West and East Antarctic and in Greenland. The elicitation enumerated dependencies between these processes under selected global temperature change scenarios over different future time horizons. These expert judgments allowed us to populate a Paired Copula Bayesian network model to obtain the estimated contributions of these ice sheets for future sea level rise. Including positive central tendency dependence and tail dependence increases the fatness of the upper tails of projected sea level rise distributions, an amplification important for designing and evaluating possible mitigation strategies. Detailing and jointly computing distributional dependencies and tail dependencies can be crucial components of good practice for assessing the influence of uncertainties on extreme values when modelling stochastic multifactorial processes.

The problem of failure rate estimation is considered in the light of data acquired from the ATV reporting system. A number of different models for preventive maintenance are discussed which make the failure rate identifiable, as are more general bounding methods which do not require identifiability. These techniques are illustrated on ATV data which is shown in graphs of subsurvival functions. These graphs show, amongst other things, the effect of different maintenance policies on identical systems.

The ice sheets covering Antarctica and Greenland present the greatest uncertainty in, and largest potential contribution to, future sea level rise. The uncertainty arises from a paucity of suitable observations covering the full range of ice sheet behaviors, incomplete understanding of the influences of diverse processes, and limitations in defining key boundary conditions for the numerical models. To investigate the impact of these uncertainties on ice sheet projections we undertook a structured expert judgement study. Here, we interrogate the findings of that study to identify the dominant drivers of uncertainty in projections and their relative importance as a function of ice sheet and time. We find that for the 21st century, Greenland surface melting, in particular the role of surface albedo effects, and West Antarctic ice dynamics, specifically the role of ice shelf buttressing, dominate the uncertainty. The importance of these effects holds under both a high-end 5°C global warming scenario and another that limits global warming to 2°C. During the 22nd century the dominant drivers of uncertainty shift. Under the 5°C scenario, East Antarctic ice dynamics dominate the uncertainty in projections, driven by the possible role of ice flow instabilities. These dynamic effects only become dominant, however, for a temperature scenario above the Paris Agreement 2°C target and beyond 2100. Our findings identify key processes and factors that need to be addressed in future modeling and observational studies in order to reduce uncertainties in ice sheet projections.

Epidemiologic cohort studies have consistently demonstrated that long-term exposure to ambient fine particles (PM_2.5) is associated with mortality. Nevertheless, extrapolating results to understudied locations may involve considerable uncertainty. To explore this issue, this review discusses the evidence for (i) the associated risk of mortality, (ii) the shape of the concentration-response function, (iii) a causal interpretation, and (iv) how the source mix/composition of PM_2.5and population characteristics may alter the effect. The accumulated evidence suggests the following: (i) In the United States, the change in all-cause mortality risk per μg/m³is about 0.8%. (ii) The concentration-response function appears nonlinear. (iii) Causation is overwhelmingly supported. (iv) Fossil fuel combustion-related sources are likely more toxic than others, and age, race, and income may modify the effect. To illustrate the use of our findings in support of a risk assessment in an understudied setting, we consider Kuwait. However, given the complexity of this relationship and the heterogeneity in reported effects, it is unreasonable to think that, in such circumstances, point estimates can be meaningful. Consequently, quantitative probabilistic estimates, which cannot be derived objectively, become essential. Formally elicited expert judgment can provide such estimates, and this review provides the evidence to support an elicitation.

Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that - (i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.

Averaging quantiles as a way of combining experts' judgments is studied both mathematically and empirically. Quantile averaging is equivalent to taking the harmonic mean of densities evaluated at quantile points. A variance shrinkage law is established between equal and harmonic weighting. Data from 49 post-2006 studies are extended to include harmonic weighting in addition to equal and performance-based weighting. It emerges that harmonic weighting has the highest average information and degraded statistical accuracy. The hypothesis that the quantile average is statistically accurate would be rejected at the 5% level in 28 studies and at the 0.1% level in 15 studies. For performance weighting, these numbers are 3 and 1, for equal weighting 2 and 1.

Regular vines (R-vines) copulas build high dimensional joint densities from arbitrary one-dimensional margins and (conditional) bivariate copula densities. Vine densities enable the computation of all conditional distributions, though the calculations can be numerically intensive. Saturated continuous nonparametric Bayes nets (CNPBN) are regular vines. Computing regression functions from the vine copula density is termed vine regression. The epicycles of regression–including/excluding covariates, interactions, higher order terms, multicollinearity, model fit, transformations, heteroscedasticity, bias–are dispelled. One simply computes the regressions from the vine copula density. Only the question of finding an adequate vine copula remains. Vine regression is applied to a data set from the National Longitudinal Study of Youth relating breastfeeding to IQ. The expected effects of breastfeeding on IQ depend on IQ, on the baseline level of breastfeeding, on the duration of additional breastfeeding and on the values of other covariates. A child given two weeks breastfeeding can expect to increase his/her IQ by 1.5–2 IQ points by adding 10 weeks of breastfeeding, depending on values of other covariates. A child given two years breastfeeding can expect to gain from 0.48–0.65 IQ points from 10 additional weeks. Adding 10 weeks breastfeeding to each of the 3,179 children in this data set has a net present value $50,700,000 according to the Bayes net, compared to $29,000,000 according to the linear regression.

Drawing on risk methods from volcano crises, we developed a rapid COVID-19 infection model for the partial return of pupils to primary schools in England in June and July 2020, and a full return in September 2020. The model handles uncertainties in key parameters, using a stochastic re-sampling technique, allowing us to evaluate infection levels as a function of COVID-19 prevalence and projected pupil and staff headcounts. Assuming average national adult prevalence, for the first scenario (as at 1 June 2020) we found that between 178 and 924 [90% CI] schools would have at least one infected individual, out of 16 769 primary schools in total. For the second return (July), our estimate ranged between 336 (2%) and 1873 (11%) infected schools. For a full return in September 2020, our projected range was 661 (4%) to 3310 (20%) infected schools, assuming the same prevalence as for 5 June. If national prevalence fell to one-quarter of that, the projected September range would decrease to between 381 (2%) and 900 (5%) schools but would increase to between 2131 (13%) and 9743 (58%) schools if prevalence increased to 4× June level. When regional variations in prevalence and school size distribution were included in the model, a slight decrease in the projected number of infected schools was indicated, but uncertainty on estimates increased markedly. The latter model variant indicated that 82% of infected schools would be in areas where prevalence exceeded the national average and the probability of multiple infected persons in a school would be higher in such areas. Post hoc, our model projections for 1 September 2020 were seen to have been realistic and reasonable (in terms of related uncertainties) when data on schools' infections were released by official agencies following the start of the 2020/2021 academic year.

Post-2006 expert judgment data has been extended to 530 experts assessing 580 calibration variables from their fields. New analysis shows that point predictions as medians of combined expert distributions outperform combined medians, and medians of performance weighted combinations outperform medians of equal weighted combinations. Relative to the equal weight combination of medians, using the medians of performance weighted combinations yields a 65% improvement. Using the medians of equally weighted combinations yields a 46% improvement. The Random Expert Hypothesis underlying all performance-blind combination schemes, namely that differences in expert performance reflect random stressors and not persistent properties of the experts, is tested by randomly scrambling expert panels. Generating distributions for a full set of performance metrics, the hypotheses that the original panels’ performance measures are drawn from distributions produced by random scrambling are rejected at significance levels ranging from E−6 to E−12. Random stressors cannot produce the variations in performance seen in the original panels. In- and out-of-sample validation results are updated.

Despite considerable advances in process understanding, numerical modeling, and the observational record of ice sheet contributions to global mean sea-level rise (SLR) since the Fifth Assessment Report (AR5) of the Intergovernmental Panel on Climate Change, severe limitations remain in the predictive capability of ice sheet models. As a consequence, the potential contributions of ice sheets remain the largest source of uncertainty in projecting future SLR. Here, we report the findings of a structured expert judgement study, using unique techniques for modeling correlations between inter- and intra-ice sheet processes and their tail dependences. We find that since the AR5, expert uncertainty has grown, in particular because of uncertain ice dynamic effects. For a +2 °C temperature scenario consistent with the Paris Agreement, we obtain a median estimate of a 26 cm SLR contribution by 2100, with a 95th percentile value of 81 cm. For a +5 °C temperature scenario more consistent with unchecked emissions growth, the corresponding values are 51 and 178 cm, respectively. Inclusion of thermal expansion and glacier contributions results in a global total SLR estimate that exceeds 2 m at the 95th percentile. Our findings support the use of scenarios of 21st century global total SLR exceeding 2 m for planning purposes. Beyond 2100, uncertainty and projected SLR increase rapidly. The 95th percentile ice sheet contribution by 2200, for the +5 °C scenario, is 7.5 m as a result of instabilities coming into play in both West and East Antarctica. Introducing process correlations and tail dependences increases estimates by roughly 15%.

The increase of multidrug resistance and resistance to last-line antibiotics is a major global public health threat. Although surveillance programs provide useful current and historical information on the scale of the problem, the future emergence and spread of antibiotic resistance is uncertain, and quantifying this uncertainty is crucial for guiding decisions about investment in antibiotics and resistance control strategies. Mathematical and statistical models capable of projecting future rates are challenged by the paucity of data and the complexity of the emergence and spread of resistance, but experts have relevant knowledge. We use the Classical Model of structured expert judgment to elicit projections with uncertainty bounds of resistance rates through 2026 for nine pathogen-antibiotic pairs in four European countries and empirically validate the assessments against data on a set of calibration questions. The performance-weighted combination of experts in France, Spain, and the United Kingdom projected that resistance for five pairs on the World Health Organization’s priority pathogens list (E. coli and K. pneumoniae resistant to third-generation cephalosporins and carbapenems and MRSA) would remain below 50% in 2026. In Italy, although upper bounds of 90% credible ranges exceed 50% resistance for some pairs, the medians suggest Italy will sustain or improve its current rates. We compare these expert projections to statistical forecasts based on historical data from the European Antimicrobial Resistance Surveillance Network (EARS-Net). Results from the statistical models differ from each other and from the judgmental forecasts in many cases. The judgmental forecasts include information from the experts about the impact of current and future shifts in infection control, antibiotic usage, and other factors that cannot be easily captured in statistical forecasts, demonstrating the potential of structured expert judgment as a tool for better understanding the uncertainty about future antibiotic resistance.

Probabilistic thinking can often be unintuitive. This is the case even for simple problems, let alone the more complex ones arising in climate modelling, where disparate information sources need to be combined. The physical models, the natural variability of systems, the measurement errors and their dependence upon the observational period length should be modelled together in order to understand the intricacies of the underlying processes. We use Bayesian networks (BNs) to connect all the above-mentioned pieces in a climate trend uncertainty quantification framework. Inference in such models allows us to observe some seemingly nonsensical outcomes. We argue that they must be pondered rather than discarded until we understand how they arise. We would like to stress that the main focus of this paper is the use of BNs in complex probabilistic settings rather than the application itself.

Validation is the hallmark of science. For expert judgment to contribute to science-based uncertainty quantification, it must become amenable to empirical validation. Using data in which experts quantify uncertainty on variables from their fields whose true values are known post hoc, this chapter explains how validation is performed in the Classical Model for structured expert judgment and reviews results for different combination methods.

The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according to metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to determine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.

The sharp rise in Oklahoma seismicity since 2009 is due to wastewater injection. The role of injection depth is an open, complex issue, yet critical for hazard assessment and regulation. We developed an advanced Bayesian network to model joint conditional dependencies between spatial, operational, and seismicity parameters. We found that injection depth relative to crystalline basement most strongly correlates with seismic moment release. The joint effects of depth and volume are critical, as injection rate becomes more influential near the basement interface. Restricting injection depths to 200 to 500 meters above basement could reduce annual seismic moment release by a factor of 1.4 to 2.8. Our approach enables identification of subregions where targeted regulation May mitigate effects of induced earthquakes, aiding operators and regulators in wastewater disposal regions.

The goal of this study is to show how to quantify the benefits of accelerated learning about key parameters of the climatic system and use this knowledge to improve decision-making on climate policy. The US social cost of carbon (SCC) methodology is used in innovative ways to value new Earth observing systems (EOSs). The study departs from the strict US SCC methodology, and from previous work, in that net benefits are used instead of only damages to calculate the value of information of the enhanced systems. In other respects the US SCC methodology is followed closely. We compute the surfeit expected net benefits of learning the actionable information earlier, with the enhanced system, versus learning later with existing systems. The enhanced systems are designed to give reliable information about climate sensitivity on accelerated timescales relative to existing systems; therefore, the decision context stipulates that a global reduced emissions path would be deployed upon receiving suitable information on the rate of temperature rise with a suitable level of confidence. By placing the enhanced observing system in a decision context, the SCC enables valuing this system as a real option. Policy relevance Uncertainty in key parameters of the climatic system is often cited as a barrier for near-term reductions of carbon emissions. It is a truism among risk managers that uncertainty costs money, and its reduction has economic value. Advancing policy making under uncertainty requires valuing the reduction in uncertainty. Using CLARREO, a new proposed EOS,as an example, this article applies value of information/real option theory to value the reduction of uncertainty in the decadal rate of temperature rise. The US interagency social cost of carbon directive provides the decision context for the valuations. It is shown that the real option value of the uncertainty reduction, relative to existing observing systems, is a very large multiple of the new system's cost.