This study evaluates five scoring rules, or measures of statistical accuracy, for assessing uncertainty estimates from expert judgment studies and model forecasts. These rules — the Continuously Ranked Probability Score ((Formula presented.)), Kolmogorov-Smirnov ((Formula presented.)), Cramer-von Mises ((Formula presented.)), Anderson Darling ((Formula presented.)), and chi-square test — were applied to 6864 expert uncertainty estimates from 49 Classical Model (CM) studies. We compared their sensitivity to various biases and their ability to serve as performance-based weight for expert estimates. Additionally, the piecewise uniform and Metalog distribution were evaluated for their representation of expert estimates because four of the five rules require interpolating the experts' estimates. Simulating biased estimates reveals varying sensitivity of the considered test statistics to these biases. Expert weights derived using one measure of statistical accuracy were evaluated with other measures to assess their performance. The main conclusions are (1) (Formula presented.) overlooks important biases, while chi-square and (Formula presented.) behave similarly, as do (Formula presented.) and (Formula presented.). (2) All measures except (Formula presented.) agree that performance weighting is superior to equal weighting with respect to statistical accuracy. (3) Neither distributions can effectively predict the position of a removed quantile estimate. These insights show the behavior of different scoring rules for combining uncertainty estimates from expert or models, and extent the knowledge for best-practices.

Bayesian networks (BNs) are popular models that represent complex relationships among variables. In the discrete case, these relationships can be quantified by conditional probability tables (CPTs). CPTs can be derived from data, but if data are not sufficient, experts can be involved to assess the probabilities in the CPTs through Structured Expert Judgment (SEJ). This is often a burdensome task due to the large number of probabilities that need to be assessed and the structured protocols that need to be followed. To lighten the elicitation burden, several methods have previously been developed to construct CPTs using a limited number of input parameters, such as InterBeta, the Ranked Nodes Method (RNM), and Functional Interpolation. In this study, the burden/accuracy trade-off of InterBeta is researched by applying the method to reconstruct previously elicited CPTs and simulated CPTs, first by comparing these CPTs to ones constructed using RNM and Functional Interpolation. After that, InterBeta extensions are proposed and tested, including an extra mean function (shifted geometric mean), the elicitation of additional middle rows, and the newly proposed extension ExtraBeta. InterBeta with parent weights is found to be the best-performing method, and the ExtraBeta extension is found to be promising and is proposed for further exploration.

Wearable sensors enable remote, continuous patient monitoring at home, offering a promising approach for early detection of postoperative complications. However, analyzing continuous long-term physiological data remains challenging, particularly in the absence of precisely labeled deterioration events. Unsupervised change point detection methods can address this issue by identifying physiological deviations without requiring predefined event labels. This study investigates the feasibility of using a Long-Short-Term Memory (LSTM) autoencoder for detecting postoperative complications from continuous heart rate and respiration rate data using a wearable patch sensor while monitoring patients in their homes. The autoencoder was applied to identify physiological deviations that may indicate potential complications after major abdominal oncological surgeries in ten patients. The model was trained on data from seven patients to recognize deviations from normal physiological patterns and evaluated on three patients. The proposed model detected change points preceding the clinically documented complication time in two test patients, identifying these deteriorations an average of 3.25 hours earlier than the standard Remote Early Warning Score (REWS) alarm system. These findings suggest that LSTM autoencoder-based change point detection could be a valuable tool for identifying postoperative complications early in remote patient monitoring settings, to support timely intervention and potentially improving patient outcomes.

A review of scoring rules highlights the distinction between rewarding honesty and rewarding quality. This motivates the introduction of a scale-invariant version of the Continuous Ranked Probability Score (CRPS) which enables statistical accuracy (SA) testing based on an exact rather than an asymptotic distribution of the density of convolutions. A recent data set of 6761 expert probabilistic forecasts for questions for which the actual values are known is used to compare performance. New insights include that (a) variance due to assessed variables dominates variance due to experts, (b) performance on mean absolute percentage error (MAPE) is weakly related to SA (c) scale-invariant CRPS combinations compete with the Classical Model (CM) on SA and MAPE, and (d) CRPS is more forgiving with regard to SA than the CM as CRPS is insensitive to location bias.

This study focuses on measuring the influence of critical Human and Organizational Factors (HOFs) on human error occurrence in structural design and construction tasks within the context of the Dutch construction industry. The primary research question addressed in this paper concerns the extent of HOFs’ contribution to human error occurrence. To answer this question, the Classical Model for Structured Expert Judgement (SEJ) is employed, enabling experts to provide their judgments on task Human Error Probability (HEP) influenced by different HOFs, which are subsequently aggregated mathematically. SEJ is chosen as a suitable approach due to the limited availability of applicable data in the construction sector. As a result, the impacts of HOFs are quantified as multipliers, representing the ratio between the observed or evaluated task HEP and its baseline value. These multipliers are then compared with corresponding multipliers from existing Human Reliability Analysis methods and studies. The findings reveal that fitness-for-duty, organizational characteristics and fragmentation exhibit the most pronounced negative effects, whereas complexity, attitude and fitness-for-duty demonstrate the most significant positive impacts on task performance. These results offer valuable insights that can be applied to enhance structural safety assurance practices.

Mathematical aggregation of probabilistic expert judgments often involves weighted linear combinations of experts’ elicited probability distributions of uncertain quantities. Experts’ weights are commonly derived from calibration experiments based on the experts’ performance scores, where performance is evaluated in terms of the calibration and the informativeness of the elicited distributions. This is referred to as Cooke’s method, or the classical model (CM), for aggregating probabilistic expert judgments. The performance scores are derived from experiments, so they are uncertain and, therefore, can be represented by random variables. As a consequence, the experts’ weights are also random variables. We focus on addressing the underlying uncertainty when calculating experts’ weights to be used in a mathematical aggregation of expert elicited distributions. This paper investigates the potential of applying an empirical Bayes development of the James–Stein shrinkage estimation technique on the CM’s weights to derive shrinkage weights with reduced mean squared errors. We analyze 51 professional CM expert elicitation studies. We investigate the differences between the classical and the (new) shrinkage CM weights and the benefits of using the new weights. In theory, the outcome of a probabilistic model using the shrinkage weights should be better than that obtained when using the classical weights because shrinkage estimation techniques reduce the mean squared errors of estimators in general. In particular, the empirical Bayes shrinkage method used here reduces the assigned weights for those experts with larger variances in the corresponding sampling distributions of weights in the experiment. We measure improvement of the aggregated judgments in a cross-validation setting using two studies that can afford such an approach. Contrary to expectations, the results are inconclusive. However, in practice, we can use the proposed shrinkage weights to increase the reliability of derived weights when only small-sized experiments are available. We demonstrate the latter on 49 post-2006 professional CM expert elicitation studies.

Evaluation systems have been long criticized for abusing and misusing bibliometric indicators. This has created a culture by which academics are constantly exposing their daily work to the standards they are expected to perform. In this study, we investigate whether researchers’ own values and expectations are in line with the expectations of the evaluation system. We conduct a multiple case study of five departments in two Dutch universities to examine how they balance between their own valuation regimes and the evaluation schemes. For this, we combine curriculum analysis with a series of semi-structured interviews. We propose a model to study the diversity of academic activities and apply it to the multiple case study to understand how such diversity is shaped by discipline and career stage. We conclude that the observed misalignment is not only resulting from an abuse of metrics but also by a lack of tools to evaluate performance in a contextualized and adaptable way.

BACKGROUND: The Girinka program in Rwanda has contributed to an increase in milk production, as well as to reduced malnutrition and increased incomes. But dairy products can be hazardous to health, potentially transmitting diseases such as bovine brucellosis, tuberculosis, and cause diarrhea. We analyzed the burden of foodborne disease due to consumption of raw milk and other dairy products in Rwanda to support the development of policy options for the improvement of the quality and safety of milk. METHODS: Disease burden data for five pathogens (Campylobacter spp., nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., and Mycobacterium bovis) were extracted from the 2010 WHO Foodborne Disease Burden Epidemiology Reference Group (FERG) database and merged with data of the proportion of foodborne disease attributable to consuming dairy products from FERG and a separately published Structured Expert Elicitation study to generate estimates of the uncertainty distributions of the disease burden by Monte Carlo simulation. RESULTS: According to WHO, the foodborne disease burden (all foods) of these five pathogens in Rwanda in 2010 was like or lower than in the Africa E subregion as defined by FERG. There were 57,500 illnesses occurring in Rwanda owing to consumption of dairy products, 55 deaths and 3,870 Disability Adjusted Life Years (DALYs) causing a cost-of-illness of $3.2 million. 44% of the burden (in DALYs) was attributed to drinking raw milk and sizeable proportions were also attributed to traditionally (16-23%) or industrially (6-22%) fermented milk. More recent data are not available, but the burden (in DALYs) of tuberculosis and diarrheal disease by all causes in Rwanda has declined between 2010 and 2019 by 33% and 46%, respectively. CONCLUSION: This is the first study examining the WHO estimates of the burden of foodborne disease on a national level in Rwanda. Transitioning from consuming raw to processed milk (fermented, heat treated or otherwise) may prevent a considerable disease burden and cost-of-illness, but the full benefits will only be achieved if there is a simultaneous improvement of pathogen inactivation during processing, and prevention of recontamination of processed products.

Background According to the World Health Organization, 600 million cases of foodborne disease occurred in 2010. To inform risk management strategies aimed at reducing this burden, attribution to specific foods is necessary. Objective We present attribution estimates for foodborne pathogens (Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Crypto-sporidium spp., Brucella spp., and Mycobacterium bovis) in three African countries (Burkina Faso, Ethiopia, Rwanda) to support risk assessment and cost-benefit analysis in three projects aimed at increasing safety of beef, dairy, poultry meat and vegetables in these countries. Methods We used the same methodology as the World Health Organization, i.e., Structured Expert Judgment according to Cooke’s Classical Model, using three different panels for the three countries. Experts were interviewed remotely and completed calibration questions during the interview without access to any resources. They then completed target questions after the interview, using resources as considered necessary. Expert data were validated using two objective measures, calibration score or statistical accuracy, and information score. Per-formance-based weights were derived from the two measures to aggregate experts’ distributions into a so-called decision maker. The analysis was made using Excalibur software, and resulting distributions were normalized using Monte Carlo simulation. Results Individual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable propor-tion of ETEC illness was attributed to animal source foods as experts considered contamina-tion of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection. Conclusions Cooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries. Attribution estimates will be combined with country-level estimates of the burden of foodborne disease to inform decision making by national authorities.

We model the growth of scientific literature related to COVID-19 and forecast the expected growth from 1 June 2021. Considering the significant scientific and financial efforts made by the research community to find solutions to end the COVID-19 pandemic, an unprecedented volume of scientific outputs is being produced. This questions the capacity of scientists, politicians and citizens to maintain infrastructure, digest content and take scientifically informed decisions. A crucial aspect is to make predictions to prepare for such a large corpus of scientific literature. Here we base our predictions on the Autoregressive Integrated Moving Average (ARIMA) and exponential smoothing models using the Dimensions database. This source has the particularity of including in the metadata information on the date in which papers were indexed. We present global predictions, plus predictions in three specific settings: by type of access (Open Access), by domain-specific repository (SSRN and MedRxiv) and by several research fields. We conclude by discussing our findings.

Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that - (i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.

Foodborne disease is a significant global health problem, with low- and middle-income countries disproportionately affected. Given that most fresh animal and vegetable foods in LMICs are bought in informal food systems, much the burden of foodborne disease in LMIC is also linked to informal markets. Developing estimates of the national burden of foodborne disease and attribution to specific food products will inform decision-makers about the size of the problem and motivate action to mitigate risks and prevent illness. This study provides estimates for the burden of foodborne disease caused by selected hazards in two African countries (Burkina Faso and Ethiopia) and attribution to specific foods. Country-specific estimates of the burden of disease in 2010 for Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli and non-typhoidal Salmonella enterica were obtained from WHO and updated to 2017 using data from the Global Burden of Disease study. Attribution data obtained from WHO were complemented with a dedicated Structured Expert Judgement study to estimate the burden attributable to specific foods. Monte Carlo simulation methods were used to propagate uncertainty. The burden of foodborne disease in the two countries in 2010 was largely similar to the burden in the region except for higher mortality and disability-adjusted life years (DALYs) due to Salmonella in Burkina Faso. In both countries, Campylobacter caused the largest number of cases, while Salmonella caused the largest number of deaths and DALYs. In Burkina Faso, the burden of Campylobacter and ETEC increased from 2010 to 2017, while the burden of Salmonella decreased. In Ethiopia, the burden of all hazards decreased. Mortality decreased relative to incidence in both countries. In both countries, the burden of poultry meat (in DALYs) was larger than the burden of vegetables. In Ethiopia, the burdens of beef and dairy were similar, and somewhat lower than the burden of vegetables. The burden of foodborne disease by the selected pathogens and foods in both countries was substantial. Uncertainty distributions around the estimates spanned several orders of magnitude. This reflects data limitations, as well as variability in the transmission and burden of foodborne disease associated with the pathogens considered.

The aim of this chapter is to show how a structured approach to elicit expert judgement (SEJ) can guide the practice of early internationalization. We applied SEJ to forecast some critical issues upon which an innovative start-up wished to base their decision of whether to expand their initial operations in Poland and Czech Republic to Brazil. Sixteen participants of an Executive MBA program acted as experts and underwent the procedure for eliciting their judgements. The performance of experts was quantified in terms of statistical accuracy and informativeness, which were combined to provide a performance-based weight for each expert according to Classical Model. The combination of weighted expert judgements led to improved statistical accuracy and informativeness of the forecast. The procedure demonstrates how entrepreneurs can take advantage of expert knowledge in deciding about risky endeavours when lacking their own experiences and reliable data that can guide their choices.

We present a forecasting analysis on the growth of scientific literature related to COVID-19 expected for 2021. Considering the paramount scientific and financial efforts made by the research community to find solutions to end the COVID-19 pandemic, an unprecedented volume of scientific outputs is being produced. This questions the capacity of scientists, politicians and citizens to maintain infrastructure, digest content and take scientifically informed decisions. A crucial aspect is to make predictions to prepare for such a large corpus of scientific literature. Here we base our predictions on the ARIMA model and use two different data sources: the Dimensions and World Health Organization COVID-19 databases. These two sources have the particularity of including in the metadata information the date in which papers were indexed. We present global predictions, plus predictions in three specific settings: type of access (Open Access), NLM source (PubMed and PMC), and domain-specific repository (SSRN and MedRxiv). We conclude by discussing our findings.

An extension of the D-vine based forward regression procedure to a R-vine forward regression is proposed. In this extension any R-vine structure can be taken into account. Moreover, a new heuristic is proposed to determine which R-vine structure is the most appropriate to model the conditional distribution of the response variable given the covariates. It is shown in the simulation that the performance of the heuristic is comparable to the D-vine based approach. Furthermore, it is explained how to extend the heuristic into a situation when more than one response variable are of interest. Finally, the proposed R-vine regression is applied to perform a stress analysis on the manufacturing sector which shows its impact on the whole economy.

The Classical Model (CM) or Cooke’s method for performing Structured Expert Judgement (SEJ) is the best-known method that promotes expert performance evaluation when aggregating experts’ assessments of uncertain quantities. Assessing experts’ performance in quantifying uncertainty involves two scores in CM, the calibration score (or statistical accuracy) and the information score. The two scores combine into overall scores, which, in turn, yield weights for a performance-based aggregation of experts’ opinions. The method is fairly demanding, and therefore carrying out a SEJ elicitation with CM requires careful consideration. This chapter aims to address the methodological and practical aspects of CM into a comprehensive overview of the CM elicitation process. It complements the chapter “Elicitation in the Classical Model” in the book Elicitation (Quigley et al. 2018). Nonetheless, we regard this chapter as a stand-alone material, hence some concepts and definitions will be repeated, for the sake of completeness.

A recent ample Structured Expert Judgment (SEJ) study quantified the source attribution of 33 distinct pathogens in the United States. The source attribution for five transmission pathways: food, water, animal contact, person-to-person, and environment has been considered. This chapter will detail how SEJ has been applied to answer questions of interest by discussing the process used, strengths identified, and lessons learned from designing a large SEJ study. The focus will be on the undertaken steps that have prepared the expert elicitation.

With the advent of large-scale application of hydrogen, transportation becomes crucial. Reusing the existing natural gas transmission system could serve as catalyst for the future hydrogen economy. However, a risk analysis of hydrogen transmission in existing pipelines is essential for the deployment of the new energy carrier. This paper focuses on the individual risk (IR) associated with a hazardous hydrogen jet fire and compares it with the natural gas case. The risk analysis adopts a detailed flame model and state of the art computational software, to provide an enhanced physical description of flame characteristics. This analysis concludes that hydrogen jet fires yield lower lethality levels, that decrease faster with distance than natural gas jet fires. Consequently, for large pipelines, hydrogen transmission is accompanied by significant lower IR. Howbeit, ignition effects increasingly dominate the IR for decreasing pipeline diameters and cause hydrogen transmission to yield increased IR in the vicinity of the pipeline when compared to natural gas.

Illnesses transmitted by food and water cause a major disease burden in the United States despite advancements in food safety, water treatment, and sanitation. We report estimates from a structured expert judgment study using 48 experts who applied Cooke's classical model of the proportion of disease attributable to 5 major transmission pathways (foodborne, waterborne, person-to-person, animal contact, and environmental) and 6 subpathways (food handler-related, under foodborne; recreational, drinking, and nonrecreational/nondrinking, under waterborne; and presumed person-to-person-associated and presumed animal contact-associated, under environmental). Estimates for 33 pathogens were elicited, including bacteria such as Salmonella enterica, Campylobacter spp., Legionella spp., and Pseudomonas spp.; protozoa such as Acanthamoeba spp., Cyclospora cayetanensis, and Naegleria fowleri; and viruses such as norovirus, rotavirus, and hepatitis A virus. The results highlight the importance of multiple pathways in the transmission of the included pathogens and can be used to guide prioritization of public health interventions.