Illusory generalizability of clinical prediction models.
- Resource Type
- Article
- Authors
- Chekroud, Adam M.; Hawrilenko, Matt; Loho, Hieronimus; Bondar, Julia; Gueorguieva, Ralitza; Hasan, Alkomiet; Kambeitz, Joseph; Corlett, Philip R.; Koutsouleris, Nikolaos; Krumholz, Harlan M.; Krystal, John H.; Paulus, Martin
- Source
- Science. 1/12/2024, Vol. 383 Issue 6679, p164-167. 4p. 1 Diagram, 1 Chart.
- Subject
- *MACHINE learning
*PREDICTION models
*STATISTICAL decision making
*ARIPIPRAZOLE
*STATISTICAL models
*ANTIPSYCHOTIC agents
*THERAPEUTICS
- Language
- ISSN
- 0036-8075
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a modelÕs success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability. [ABSTRACT FROM AUTHOR]