Prediction models are popular in medical research and practice. Many expect that by predicting patient-specific outcomes, these models have the potential to inform treatment decisions, and they are frequently lauded as instruments for personalized, data-driven healthcare. We show, however, that using prediction models for decision-making can lead to harm, even when the predictions exhibit good discrimination after deployment. These models are harmful self-fulfilling prophecies: their deployment harms a group of patients, but the worse outcome of these patients does not diminish the discrimination of the model. Our main result is a formal characterization of a set of such prediction models. Next, we show that models that are well calibrated before and after deployment are useless for decision-making, as they make no change in the data distribution. These results call for a reconsideration of standard practices for validation and deployment of prediction models that are used in medical decisions. ... Read more

Clinicians increasingly rely on prediction models to guide treatment choices. Most prediction models, however, are developed using observational data that include some patients who have already received the treatment the prediction model is meant to inform. Special attention to the causal role of those earlier treatments is required when interpreting the resulting predictions.

“Causal blind spots” were identified in 3 common approaches to handling treatment when developing a prediction model: including treatment as a predictor, restricting to persons taking a certain treatment, and ignoring treatment. Through several real examples, this article illustrates how the risks obtained from models developed using such approaches may be misinterpreted and can lead to misinformed decision making. The discussion covers issues attributable to confounding, selection, mediation, and changes in treatment protocols over time.

An extension of guidelines for the development, reporting, and evaluation of prediction models is advocated to avoid such misinterpretations. Developers must ensure that the intended target population for the model, and the treatment conditions under which predictions hold, are clearly communicated. When prediction models are intended to inform treatment decisions, they need to provide estimates of risk under the specific treatment (or intervention) options being considered, known as “prediction under interventions.” Next to suitable data, this requires causal reasoning and causal inference techniques during model development and evaluation. Being clear about what a given prediction model can and cannot be used for prevents misinformed treatment decisions and thereby prevents potential harm to patients. ... Read more

Prediction models are used among others to inform medical decisions on interventions. Typically, individuals with high risks of adverse outcomes are advised to undergo an intervention while those at low risk are advised to refrain from it. Standard prediction models do not always provide risks that are relevant to inform such decisions: for example, an individual may be estimated to be at low risk because similar individuals in the past received an intervention which lowered their risk. Therefore, prediction models supporting decisions should target risks belonging to defined intervention strategies. Previous works on prediction under interventions assumed that the prediction model was used only at one time point to make an intervention decision. In clinical practice, intervention decisions are rarely made only once: they might be repeated, deferred, and reevaluated. This requires estimated risks under interventions that can be reconsidered at several potential decision moments. In the current work, we highlight key considerations for formulating estimands in sequential prediction under interventions that can inform such intervention decisions. We illustrate these considerations by giving examples of estimands for a case study about choosing between vaginal delivery and cesarean section for women giving birth. Our formalization of prediction tasks in a sequential, causal, and estimand context provides guidance for future studies to ensure that the right question is answered and appropriate causal estimation approaches are chosen to develop sequential prediction models that can inform intervention decisions. ... Read more

Objective: To evaluate if immediate catheter removal (ICR) after laparoscopic hysterectomy is associated with similar retention outcomes compared with delayed removal (DCR). Study design: Non-inferiority randomised controlled trial. Population: Women undergoing laparoscopic hysterectomy in six hospitals in the Netherlands. Methods: Women were randomised to ICR or DCR (between 18 and 24 hours after surgery). Primary outcome: The inability to void within 6 hours after catheter removal. Results: One hundred and fifty-five women were randomised to ICR (n = 74) and DCR (n = 81). The intention-to-treat and per-protocol analysis could not demonstrate the non-inferiority of ICR: ten women with ICR could not urinate spontaneously within 6 hours compared with none in the delayed group (risk difference 13.5%, 5.6–24.8, P = 0.88). However, seven of these women could void spontaneously within 9 hours without additional intervention. Regarding the secondary outcomes, eight women from the delayed group requested earlier catheter removal because of complaints (9.9%). Three women with ICR (4.1%) had a urinary tract infection postoperatively versus eight with DCR (9.9%, risk difference −5.8%, −15.1 to 3.5, P = 0.215). Women with ICR mobilised significantly earlier (5.7 hours, 0.8–23.3 versus 21.0 hours, 1.4–29.9; P ≤ 0.001). Conclusion: The non-inferiority of ICR could not be demonstrated in terms of urinary retention 6 hours after procedure. However, 70% of the women with voiding difficulties could void spontaneously within 9 hours after laparoscopic hysterectomy. It is therefore questionable if all observed urinary retention cases were clinically relevant. As a result, the clinical advantages of ICR may still outweigh the risk of bladder retention and it should therefore be considered after uncomplicated laparoscopic hysterectomy. Tweetable abstract: The advantages of immediate catheter removal after laparoscopic hysterectomy seem to outweigh the risk of bladder retention. ... Read more

Background: Pelvic endometriosis is often mentioned as one of the variables influencing surgical outcomes of laparoscopic hysterectomy (LH). However, its additional surgical risks have not been well established. The aim of this study was to analyze to what extent concomitant endometriosis influences surgical outcomes of LH and to determine if it should be considered as case-mix variable. Results: A total of 2655 LH’s were analyzed, of which 397 (15.0%) with concomitant endometriosis. For blood loss and operative time, no measurable association was found for stages I (n = 106) and II (n = 103) endometriosis compared to LH without endometriosis. LH with stages III (n = 93) and IV (n = 95) endometriosis were associated with more intra-operative blood loss (p = <.001) and a prolonged operative time (p = <.001) compared to LH without endometriosis. No significant association was found between endometriosis (all stages) and complications (p =.62). Conclusions: The findings of our study have provided numeric support for the influence of concomitant endometriosis on surgical outcomes of LH, without bowel or bladder dissection. Only stages III and IV were associated with a longer operative time and more blood loss and should thus be considered as case-mix variables in future quality measurement tools. ... Read more