When accurate prediction models yield harmful self-fulfilling prophecies

Journal Article (2025)
Author(s)

Wouter van Amsterdam (University Medical Center Utrecht)

N. van Geloven (Leiden University Medical Center)

JH Krijthe (TU Delft - Pattern Recognition and Bioinformatics)

Rajesh Ranganath (New York University)

Giovanni Cinà (Universiteit van Amsterdam, Pacmed)

Research Group
Pattern Recognition and Bioinformatics
DOI related publication
https://doi.org/10.1016/j.patter.2025.101229
More Info
expand_more
Publication Year
2025
Language
English
Research Group
Pattern Recognition and Bioinformatics
Issue number
4
Volume number
6
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Prediction models are popular in medical research and practice. Many expect that by predicting patient-specific outcomes, these models have the potential to inform treatment decisions, and they are frequently lauded as instruments for personalized, data-driven healthcare. We show, however, that using prediction models for decision-making can lead to harm, even when the predictions exhibit good discrimination after deployment. These models are harmful self-fulfilling prophecies: their deployment harms a group of patients, but the worse outcome of these patients does not diminish the discrimination of the model. Our main result is a formal characterization of a set of such prediction models. Next, we show that models that are well calibrated before and after deployment are useless for decision-making, as they make no change in the data distribution. These results call for a reconsideration of standard practices for validation and deployment of prediction models that are used in medical decisions.