🔍 Read the full analysis: Microsoft’s AI Doctor Study: A Closer Read Of The Results on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A study published in Science on Oct. 22 reports that Microsoft’s MAI-DxO system achieved 83.2% diagnostic accuracy on 304 difficult NEJM cases, compared with 36.1% for generalist physicians using reference materials. The results come from a text-based simulation, not care of real patients, and GPT-5 alone scored 78.9% on the same complex-case benchmark. A prospective study alongside routine care is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult medical cases in a study published Oct. 22 in Science, compared with 36.1% for generalist physicians who could use textbooks and the internet. The peer-reviewed result is based on historical written cases and a simulated testing process, not treatment of real patients, so it does not establish how the system would perform in hospitals.
The paper, titled Sequential diagnosis with language models, evaluated Microsoft’s Sequential Diagnosis Benchmark. Each case began with a brief summary. The system or physician then had to request additional details, such as answers to questions or test results, from a simulated gatekeeper holding the case file. The benchmark scored both the final diagnosis and the estimated cost of tests used to reach it.
Microsoft reported 83.2% accuracy for MAI-DxO on the 304 complex cases drawn from the New England Journal of Medicine. Generalist physicians scored 36.1% with reference materials and 22.3% without them. On a separate set of 325 emergency-department cases, the reported scores were 80.3% for the AI system and 46.8% for physicians.
The comparison does not show that MAI-DxO outperformed specialist teams or doctors working together. The physician group consisted of primary-care doctors and hospital generalists working individually. The supplied account says the cases were challenging teaching cases that would commonly be referred to specialists in real practice. The study also used text records rather than physical examinations, actual scans or live patient interactions.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Mean for Care
The results offer evidence that language models can perform well on a specific kind of difficult diagnostic exercise: deciding what information to request and when to order tests. That matters because diagnosis is not only a matter of matching a complete case history to a disease; clinicians also choose which questions and tests may help resolve uncertainty. The benchmark attempts to capture that process while assigning costs to requested tests.
But a high score in this setting is not evidence that the tool is ready to diagnose patients independently. The benchmark contains selected, difficult cases and does not establish how the system handles routine complaints, patients with more than one condition, incomplete records or the consequences of a wrong recommendation. The source account also says the NEJM set contains no healthy patients or benign explanations, limiting what the study can say about false alarms. Hospitals, insurers and patients should read the result as a research finding, not as a clinical-performance guarantee.
The reported performance of GPT-5 alone was 78.9% on the complex cases, close to MAI-DxO’s 83.2%. Microsoft’s system adds an orchestration layer that has virtual physician agents propose diagnoses, choose questions or tests, and challenge one another. In the company’s reported comparison, that layer added about four percentage points and reduced estimated test costs. The result leaves open how much of the performance comes from the underlying model and how much from the multi-agent method.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Microsoft Tested the System
The paper followed a 2025 arXiv preprint, which was not peer-reviewed and reported an earlier version of the system using OpenAI’s o3 model. The Science paper was published Oct. 22, 2026; the supplied source says it was received in April and accepted in August. All 15 listed authors were affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. Publication in a peer-reviewed journal means the work underwent scholarly review, but it is not independent replication of Microsoft’s findings.
Some methodological details in the supplied account come from the preprint, and may not match the final paper. In that version, a model judged diagnoses against the recorded answer, while clinicians re-graded a sample with good, but imperfect, agreement. When a requested test result was absent from the case file, the gatekeeper generated a synthetic result. Test prices were estimated using billing codes and a 2023 U.S. hospital price list, with $300 added for each physician visit; the authors described these as first-order approximations.
The preprint’s physician comparison involved 21 U.S. and U.K. generalists, including 17 primary-care doctors and four hospital generalists, with a median of 12 years’ experience. Those doctors did not use search engines or language models. They assessed 56 held-out cases, while the AI was assessed on all 304. The Science paper’s reported physician results include an open-book group, but the supplied material does not provide enough detail to treat every preprint procedure as identical to the final study.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
As an affiliate, we earn on qualifying purchases.
Limits of the Clinical Evidence
The study does not show whether MAI-DxO improves patient outcomes, reduces diagnostic delays or avoids unnecessary testing in routine clinical care. It also does not compare the system with specialists or multidisciplinary teams, and the benchmark design described in the supplied material cannot measure false alarms on harmless conditions because those cases were absent.
Other questions concern the scoring and costs. A model judged the diagnoses, with clinician agreement reported as good but not perfect in the preprint. Some missing test results were generated synthetically, while test prices were estimates. The full Science article is behind a paywall, according to the source account, so it is not clear from the supplied material which preprint details changed in the final version. Independent replication has not been reported here.
As an affiliate, we earn on qualifying purchases.
A Study With Patients Planned
Microsoft plans a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the supplied source, consenting patients will interact with the system alongside their usual care. That design could provide evidence about how the tool operates in a live clinical setting, though the source does not give a start date, enrollment target or planned outcome measures.
Until results from that work and independent evaluations are available, the published scores should be treated as evidence about performance on a defined simulated benchmark. Whether MAI-DxO can support clinicians safely, how it performs across less unusual cases and whether its recommendations improve care remain open questions.
AI medical decision support system
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s MAI-DxO study find?
Microsoft reported that MAI-DxO achieved 83.2% diagnostic accuracy on 304 complex historical cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those references.
Was the study conducted with real patients?
No. The reported benchmark used historical, text-based cases and simulated requests for additional information and tests. The source says a separate prospective study involving consenting patients is planned.
Did MAI-DxO outperform medical specialists?
The comparison described in the source was with generalist physicians working alone, not specialist doctors or teams. The results do not establish how MAI-DxO compares with those groups.
How much did the MAI-DxO system add beyond GPT-5?
Microsoft reported 78.9% accuracy for GPT-5 alone on the complex cases, compared with 83.2% for MAI-DxO. The source describes the orchestration layer’s gain as about four percentage points; it does not establish how that difference would translate to clinical practice.
What happens next?
A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients are expected to interact with the system alongside normal care. Timing and study details were not provided in the supplied material.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
