Research study assessing accuracy of ChatGPT in medical diagnoses

Editor's Note

A May 2023 study shows that ChatGPT–a large language model artificial intelligence chatbot–can make accurate diagnoses and care management decisions but is less adept at differential diagnosis. The findings were published in the Journal of Medical Internet Research. 

The research team from Mass General Brigham inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme Clinical Manual into ChatGPT and compared its accuracy on a number of metrics, including differential diagnoses, diagnostic testing, final diagnosis, and management. 

Accuracy was measured by human scorers determining the proportion of correct responses to the questions posed within the clinical vignettes tested. They found that ChatGPT was about 72% accurate in overall clinical decision making, including making diagnoses and care management decisions. The chatbot performed equally well in primary care and emergency care settings across all medical specialties. 

Ultimately, the researchers placed the accuracy level of ChatGPT as equivalent to someone who has just graduated from medical school, such as an intern or resident.