Another study touting the diagnostic capabilities of GPT-4, this one from the University of Birmingham:

The study, which was published in the journal PLOS Digital Health, saw a “large language model” called GPT-4 tested on a variety of medical professionals, including junior doctors without specialisation and expert and trainee eye doctors. Each was asked to provide a diagnosis or treatment advice after being shown a series of 87 patient scenarios with a particular eye condition and selecting from four options. 

Following the assessment, GPT-4 showcased a markedly superior performance when compared to unspecialised junior doctors, who possess a proficiency level comparable to general practitioners with specialist eye knowledge. The study found that GPT-4’s scores were closely aligned with those of both trainee and expert eye doctors, although top-performing doctors had achieved higher scores. 

GPT-4 helps generalist physicians, but scores lower than experts. That’s what we’d expect to see.

Read the report here.