Study evaluates the effectiveness of predictive AI in healthcare settings, identifies shortcomings

Editor's Note

Implementing predictive AI models in healthcare settings can alter the baseline assumptions the models were trained on in ways that cause the models to perform worse, according to a recent simulation study published on October 6 by the Annals of Internal Medicine. 

The study analyzed 130,000 critical care admissions at two major healthcare institutions, the Mount Sinai Health System in New York and Beth Israel Deaconess Medical Center in Boston. The researchers investigated three key scenarios: model retraining after initial use; creating a new model after one has already been in use; and concurrent use of two predictive models.

While current practice suggests retraining models can address performance degradation over time, the study shows it can actually lead to further degradation by disrupting the learned relationships between presentation and outcome. Similarly, while an AI model can predict adverse outcomes like sepsis, it will also seek to prevent death which can follow sepsis, disrupting the effectiveness of the model. 

The researchers say the findings underscore the importance of performing regular impact monitoring on these AI tools and thoughtful use of predictive models.