Microsoft’s new AI diagnostic system outscored individual doctors on tough medical cases in a study published this week, though it needed the most expensive models to pull it off.
TL;DR: Microsoft’s MAI-DxO system, built by Mustafa Suleyman’s AI health team, hit 83% accuracy on hard diagnostic cases from the New England Journal of Medicine, compared to 36% for primary care doctors and internists working alone with reference material. The results, published Thursday in the journal Science, mostly track how good the underlying AI model already is. GPT-5.5 jumped from 72% to nearly 87% once Microsoft’s orchestration layer got involved. A real-world trial is now running at Beth Israel Deaconess in Boston.
The headline number is the one Microsoft wants you to remember: 83% versus 36%. That’s the gap between MAI-DxO, a system that runs several AI agents like a panel of specialists, and solo primary care doctors working through the same complex cases pulled from NEJM case reports. Dominic King, who leads health at Microsoft AI, called it “something very substantial and exciting.”
Read the fine print and the comparison gets murkier. Cases this hard would normally land with specialists or a full physician team, not a single generalist searching reference material. The 36% figure describes a mismatch Microsoft itself points to as the real problem: “the growing mismatch between demand and supply of medical expertise,” as King put it.
The orchestration layer did less than the headline suggests
Microsoft tested MAI-DxO against models from OpenAI, Google, Anthropic, Meta, and DeepSeek. The pattern that shows up across all of them is that the base model matters more than Microsoft’s coordination system sitting on top of it.
GPT-5 alone scored 78.9% on complex cases, at roughly $8,400 per case. Running it through DxO trimmed about $3,000 off that cost and added only about four points of accuracy. That’s a modest return for a lot of engineering.
GPT-5.5, released in April, told a different story. Alone it scored 72% on the same complex cases. With DxO coordinating it, that climbed to nearly 87%, at a slightly lower cost than running GPT-5 solo. Google’s Gemini 2.5 Flash went from 46.4% by itself to almost 69% with DxO wrapped around it.
In the emergency department scenarios, the gap mostly disappeared. Both the standalone models and the DxO-coordinated versions landed around 85%, at similar cost. Fast-moving, lower-ambiguity cases apparently don’t give an orchestration layer much room to help.
Microsoft isn’t alone in this race
Google published a Lancet study with Beth Israel Deaconess this month on its own AMIE system. OpenAI has made similar claims about matching physician performance. Microsoft’s framing, that it’s chasing “medical superintelligence,” which it defines as surpassing the strongest human-led health systems across the full arc of care, puts it squarely in a three-way race with companies that have very different approaches to the same problem.
The next test is a live one. Microsoft is running DxO alongside standard care at Beth Israel Deaconess, where patients who consent get a DxO-generated opinion that’s checked against what their actual doctors conclude. King described a Boston emergency department on a Friday night as exactly the kind of chaotic, varied caseload the system needs to prove itself against. “We think this will be the biggest study of its kind,” he said.
Suleyman, who co-founded DeepMind before joining Microsoft in 2024, is betting the pace of improvement keeps accelerating. He expects healthcare-specific models to get “an order of magnitude better in capability within the next 24 to 36 months,” and says Microsoft is working with the Mayo Clinic on a dedicated healthcare foundation model slated for release next year. “We have to have the absolute number one state-of-the-art medical clinical decision support products over the next few years,” he said. “We’re just true believers.”
That kind of talk is easy to say and hard to verify from the outside. What’s checkable right now is the Science paper’s own numbers, and those numbers say the orchestration system is real but secondary. The model underneath it is doing most of the work, which means the real race Microsoft, Google, and OpenAI are running might just be whoever ships the best base model first.
What this means for hospitals, not just headlines
Buried in the cost figures is the part hospital administrators will actually care about. An $8,400-per-case price tag for GPT-5 alone is not something any clinic budgets for routinely, which is probably why Microsoft is leading with cost reduction as much as accuracy gains. Shaving $3,000 off a diagnostic workup, even with only a modest accuracy bump, changes the math on whether a tool like this gets deployed widely or stays a research curiosity used on the hardest cases a hospital already struggles to staff for.
Microsoft’s own definition of “medical superintelligence” sets a high bar: surpassing the strongest human-led health systems in medical reasoning and coordination across the full arc of care, not just beating a single doctor on a single case type. By that standard, an 83% hit rate on NEJM case studies is an early data point, not a finish line. The Beth Israel trial will be the first time this kind of system gets measured against real doctors making real decisions on real patients, rather than a retrospective case archive built for exactly this kind of benchmarking.
Related: Hugging Face’s recent open-weight genomics model (Carbon-X) shows the same pattern playing out in biology, and Unitree’s push toward self-improving robots raises a similar question about how much credit goes to the model versus the system wrapped around it.
The bottom line
MAI-DxO is a real result, not vaporware, and the Beth Israel trial will tell us whether it holds up outside a curated case set. But the study’s own data undercuts the “AI beats doctors” framing it’s being sold with. Give a model like GPT-5.5 to a system like DxO and performance jumps. Give it a model that’s already strong on its own, like GPT-5, and the orchestration barely moves the needle. The honest takeaway is that the base model is the product, and Microsoft’s panel-of-agents idea is a multiplier that only works when there’s something worth multiplying.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
