Microsoft Study Claims Its Diagnostic AI Outperformed Generalist Doctors in a Benchmark Setting

The research was peer-reviewed and published in the journal Science, and showed that the medical model, called the Microsoft AI Diagnostic Orchestrator (MAI-DxO), was over twice as accurate as generalist physicians at diagnosing patients. The tests were conducted in a controlled benchmark setting, meaning its usefulness in a true medical environment is up for debate, and Microsoft was eager to declare victory.

Microsoft CEO of its AI lab Mustafa Suleyman said the age of AI improving lives through healthcare is finally here, telling The Verge that further advances are right around the corner. He said that next year we are going to see extremely impressive deployments of intelligent systems in production, and that this is just going to change the way care is delivered in a big way.

For the study, researchers had the DxO model examine 304 clinicopathological conference cases from the New England Journal of Medicine and 325 emergency department-to-admission encounters from a database called MIMIC-IV-ED. With the former cohort, DxO achieved an 83.2 percent diagnostic accuracy, compared to 36.1 percent for generalist physicians who examined the same cases and were allowed to view reference materials. With the emergency department cases, the AI scored an 80.3 percent accuracy, while the human doctors came in at 46.8 percent.

The AI models were graded not just by the accuracy of their diagnoses, but the estimated costs of the tests ordered to reach them, and DxO overall managed to do this at half or even one third of the cost of the human doctors, Microsoft said in a blog post. The findings, if borne out, are an impressive testament to AI's capabilities and usefulness in medical applications, though there is a lot more to being a doctor than analyzing symptoms.

The study addressed the sequential nature of diagnosis somewhat by having DxO work sequentially rather than being given a list of complete symptoms, working off a patient's initial complaint, coming up with a working diagnosis, and choosing follow-up questions to ask and deciding what tests to order while considering the costs involved. But the study could only imperfectly simulate the back and forth a doctor has with a patient by having DxO talk to a gatekeeper agent that mediates all information requests drawing from the underlying case record, and human doctors already did the leg work of putting together these neat, highly detailed case files for the AI to pore over in the first place.