Expert opinions are crucial to quantify unknown variables when it is not feasible to use traditional research methods (limited resources or questions about future values). Implementing structured expert elicitation (SEE) is important for the quality and credibility of assumptions made by these experts or the researchers who enlist them. However, application in practice has been limited.

When research is performed to project policy effects, the context is so volatile that the accuracy of experts' judgements cannot be validated. Moreover, experts may focus on quantifying uncertainty while ignoring extreme uncertainties (war, future technological inventions), operating from a status quo point of view, in order to support the policy relevance of their research. This leads to a unique context of the expert judgement process. It is unclear how experts address expert judgement in this practice. Moreover, it is not yet known what problems may limit application of SEE protocols in this context.

I worked with the PBL Netherlands Environmental Assessment Agency to evaluate their expert judgement process for the KEV (The Climate and Energy Outlook). On a yearly basis their (sector) experts are required to judge dozens of uncertain variables for the projections of the progress on climate and energy goals. This includes policy effects, external factors like weather, and also scientific uncertainties like the inherent uncertainty of the models that are used for the projections.

How can practitioners in the policy research field be supported in implementing structured expert elicitation to improve their expert judgement?

To answer this research question and address it through communication design, I followed the Double Diamond method for design-based research (Discover-Define-Develop-Deliver). For the duration of a year, I was an intern at PBL, allowing me to observe and discuss their methodology, while inquiring after what problems they face. I helped organise a pilot for implementation of the IDEA protocol for structured expert elicitation. To evaluate this pilot I designed a focus group session with all different actors involved in the KEV project. Next, to discuss challenges and develop possible solutions I organised a co-design workshop with (sector) experts from different teams that work on the KEV.

The results show that experts face several unique challenges in their expert judgement process, which are not directly addressed by SEE protocols. The IDEA protocol shows promise to improve quality of judgements and transparency through increased documentation. However, the protocol lacks support experts say they require to select what uncertainties to address in their research, as well as adequate support to quantify their uncertainty interval.

Siloed teams lead to diverging methodology and definitions, as well as missing overall problem demarcation. Because there is limited opportunity for feedback on the accuracy of experts' judgements, there is no universal measure of what makes a judgement 'good'.

To address the problems faced by the practitioners at PBL, I designed an instructional video. It informs experts about SEE and provides tangible examples and actionable steps the practitioners can take on the short and long term, on the individual and organisational level. This provides the researchers with the knowledge and control to improve their own practice. At the same time, the video also acknowledges the unique challenges the experts face and reassures them of the confidence they can have in their practice to make decisions. By targeting both teamleaders and sector experts, the facilitating party and the practicing party are involved and given agency in this process.

Therefore, research requiring expert judgement on policy uncertainties presents some unique challenges in the expert judgement process. When accuracy cannot be used as a measure of quality, it is important that the researchers invest in the credibility of the methodology. SEE can provide structure and transparency, but additional instructions and methods are needed to support experts to confidently make judgements that balance accuracy and information.

Though this case study focuses on a specific organisation and group of experts, the lessons learned from this study indicate a variety of challenges when applying SEE in different research domains and contexts. This could possible indicate the stagnated implementation in practice. Additionally, this research and the development of structured expert elicitation for policy assessment support credibility of expert judgement in this political environment. ... Read more

This study explores perceptions of uncertainty and its effects on the policy-process on the science-policy interface in the Netherlands. Practitioners experience a misalignment, which leads to misunderstanding, decreased usefulness of policy assessments, and in the end affect policy design and investments.
Policy analysis can provide valuable information to understand what policy options are effective in complex policy problems, taking into account the uncertainty of human behaviour, economic development, and even weather changes. Previous studies have shown however that the applicability and relevance of uncertainty communication has been limited, specifically for policy-makers.
The case studied is the perception of uncertainty by researchers and policy officers that produce and use the Climate and EnergyOutlook in theNetherlands, which presents projections of emissions and energy use with an uncertainty bandwidth.
A conceptual framework was developed to capture how uncertainty is understood in scientific literature: Framing, Characterisation, Impact, and methods of Dealing with uncertainty. Using an adapted mental model method, 9 researchers and 6 policy officers were interviewed, and their perceptions qualitatively coded (deductive and inductive) and compared.
It is concluded that two distinct perspectives were indicated on each aspect of uncertainty (definition, characterisation, and interpretation), shaped by values and institutional context. The differences were just more nuanced than a split between the participant groups. Probability statements helped align interpretation, but this may risk neglecting other important (qualitative) aspects of uncertainty. As uncertainty is also used strategically, there is a limitation to how much researchers can influence the impact of uncertainty information.
The observations help clarify the difference in uncertainty perception experienced on the science-policy interface, how they effect communication and policy-making, and suggest ways to improve alignment. Though this case study focuses on a specific presentation of uncertainty in projections of greenhousegas emissions, the findings offer a new perspective on uncertainty perception in policy analysis. ... Read more

Layman Summary
The goal of this paper is to study how waves move over a fluid in with an underlying current. Oceans displace debris and heat. Ocean models can be used to predict the movement of debris and even predict the effect of the movements of oceans on the climate. Modeling water can be improved by introducing more information from observations into the models. The goal of this paper is to combine the effect of varying density and an underlying current in a model for ocean water near the equator. Our model focuses on long, shallow waves. This allows us to simplify the model and solve it more easily. We combine methods from different sources to create a model for our problem. The model we create shows there are two types of solution waves: Periodic travelling waves and Solitons. Solitons are single waves that don’t change shape through time. We find that density influences the amplitude of waves and the perturbation of the current. We find that the underlying current influences the wave profile’s width as well as the amplitude.

Summary
Models of oceans can be used to predict the displacement of debris and even trace its path back to its origin. Oceans are a large influence on the weather and climate all over the world. Improving these models is therefore very useful.
There isn’t a general equation that describes all water dynamics. Even if there would be, we would not have a computer good enough to make all the necessary calculations for the model. Though there will not be a perfect model, there is still a lot of room to improve the current models. The accuracy of water models, specifically for oceans, can be improved in different ways. We can increase the resolution to model smaller, more intricate behavior. The model can also be improved by coupling more different phenomena.
The goal of this paper is to combine existing methods for modeling the propagation of waves in a model that describes the propagation of waves over a current in a stratified fluid. We assume the fluid to be inviscid and incompressible. We also assume there is no thermal conductivity. We choose to focus the model on long and shallow waves and changes in the current. These
assumptions mean we can choose to base the governing equations on the Euler equations for inviscid fluids. Through non-dimensionalisation and scaling transformations, we transform our model to unitless equations. We examine only the behavior of the leading order solutions by expanding the unknown variables as asymptotic series. The final equations that describe the wave propagation belong to the family of Korteweg-deVries equations. One solution to these equations is a sech2(θ) function. They represent soliton waves. These are
solitary waves that hold their shape through the combination of dispersion and the non-linear character of the waves.
We find that the density influences the amplitude of waves and the perturbation of the current because it shows up as a multiplicative factor. We find that the underlying current influences the wave profile’s width as well as the amplitude because it also shows up inside the θ term of the sech^2(θ) function. ... Read more