This article introduces the research paper “Sequential Diagnosis with Language Models” by Harsha Nori, Mayank Daswani, Christopher Kelly, Scott Lundberg, Marco Tulio Ribeiro, Marc Wilson, Xiaoxuan Liu, Viknesh Sounderajah, Jonathan Carlson, Matthew P Lungren, Bay Gross, Peter Hames, Mustafa Suleyman, Dominic King, and Eric Horvitz from Microsoft AI, published on June 30, 2025. The paper highlights the potential of artificial intelligence to expand access to expert medical knowledge and reasoning, particularly in the complex domain of medical diagnosis.
The Challenge of Real-World Medical Diagnosis
The authors contend that most existing evaluations of language models (LMs) in medicine rely on static vignettes and multiple-choice questions, which fail to capture the complexity of real-world clinical practice. In clinical settings, physicians engage in sequential diagnosis, an iterative process where they formulate and revise diagnostic hypotheses, adapt questions and tests based on new information, and weigh evolving evidence before making a final diagnosis. This dynamic process requires crucial skills such as identifying informative next steps, balancing diagnostic yield against cost and patient burden, and knowing when sufficient evidence has been gathered. Current static benchmarks, by reducing this process to a single-turn quiz, risk overstating model competence and obscuring weaknesses like premature diagnostic closure or indiscriminate test ordering.
Introducing the Sequential Diagnosis Benchmark (SDBench)
To address these limitations, the paper introduces the Sequential Diagnosis Benchmark (SDBench), an interactive framework designed to evaluate diagnostic agents (human or AI) through realistic sequential clinical encounters. SDBench transforms 304 diagnostically challenging New England Journal of Medicine (NEJM) clinicopathological conference (CPC) cases into stepwise diagnostic encounters.
In SDBench, a diagnostic agent begins with a brief case abstract and can take one of three actions:
- Ask questions about patient history or physical examination details.
- Request diagnostic tests such as labs, imaging, or procedures.
- Diagnose, making a one-time commitment to a final diagnosis.
Information is revealed by an information Gatekeeper, a language model (o4-mini) with access to the full NEJM CPC case file. The Gatekeeper discloses findings only when explicitly queried, synthesizing additional case-consistent information for tests not described in the original narrative to maintain realism and prevent information leakage.
Performance in SDBench is assessed along two critical axes:
- Diagnostic accuracy: Evaluated by a Judge agent (o3 model) against the ground truth diagnosis using a detailed, physician-authored 5-point Likert rubric, where a score of ≥4 indicates a “correct” diagnosis. The Judge also accounts for clinical synonyms to focus on substantive correctness.
- Cumulative estimated real-world cost of all requested diagnostic tests and physician visits. This metric deters unrealistic behaviors like indiscriminate test ordering, reflecting real-world constraints. Each physician visit (sequential questions) is assigned a fixed cost of $300, and diagnostic test costs are determined using a language model-based system that translates free-text requests into CPT codes matched to 2023 U.S. health system pricing data.
This dual evaluation aligns with the goals of the Triple Aim, which seeks high-quality care delivered at sustainable costs.
The MAI Diagnostic Orchestrator (MAI-DxO)
Complementing the benchmark, the paper introduces the MAI Diagnostic Orchestrator (MAI-DxO), a model-agnostic orchestrated system co-designed with physicians that consistently outperforms both human physicians and commercial language models. MAI-DxO emulates a virtual panel of doctors working collaboratively, aiming to replicate the benefits of team-based clinical reasoning while mitigating individual cognitive biases and minimizing costs.
A single language model role-plays five distinct medical personas within MAI-DxO:
- Dr. Hypothesis: Maintains and updates a probability-ranked differential diagnosis.
- Dr. Test-Chooser: Selects diagnostic tests that maximally discriminate between leading hypotheses.
- Dr. Challenger: Acts as devil’s advocate, identifying potential anchoring bias and highlighting contradictory evidence.
- Dr. Stewardship: Enforces cost-conscious care by advocating for cheaper alternatives and vetoing low-yield expensive tests.
- Dr. Checklist: Performs silent quality control to ensure valid test names and internal consistency.
After internal deliberation, the panel reaches a consensus on whether to ask questions, order tests, or commit to a diagnosis. MAI-DxO was evaluated in five variants, ranging from most cost-conscious (“Instant Answer,” “Question Only,” “Budgeted”) to maximum accuracy (“No Budget,” “Ensemble”).
Performance and Implications
MAI-DxO, when paired with OpenAI’s o3 model, achieved 80% diagnostic accuracy, which is four times higher than the 20% average of generalist physicians on SDBench. Furthermore, MAI-DxO significantly reduces diagnostic costs by 20% compared to physicians, and 70% compared to off-the-shelf o3. When configured for maximum accuracy using ensembling techniques, MAI-DxO reached 85.5% accuracy.
These performance gains are attributed to MAI-DxO’s structured reasoning, which helps models overcome limitations by ensuring comprehensive differentials, reducing anchoring bias, and encouraging cost-conscious testing. The framework proved remarkably model-agnostic, boosting the accuracy of off-the-shelf models from various providers (OpenAI, Gemini, Claude, Grok, DeepSeek, and Llama families) by an average of 11 percentage points. The results were also robust across dataset splits, including a hidden test set of recent cases, suggesting the gains are not due to memorization.
The paper posits that frontier AI language models challenge the traditional medical structure by combining the generalist’s breadth with a specialist’s depth, allowing them to outperform individual physicians on complex diagnostic problems. This raises a critical question for future AI evaluation: should AI systems be compared to individual physicians or to entire hospital-like teams?
Limitations and Future Directions
The authors acknowledge several limitations, including that the SDBench cases are pedagogically curated and do not reflect real-world disease prevalence (e.g., no healthy patients or benign syndromes), meaning false positive rates could not be measured. The cost estimates, while standardized, are approximations and don’t account for all real-world factors like invasiveness, patient discomfort, or provider-specific variations. Additionally, the physician cohort consisted of generalists who, in reality, might refer complex cases to specialists and are free to use external resources during practice, unlike in the study.
Despite these limitations, the findings underscore AI’s increasing potential in healthcare delivery, particularly in addressing clinical workforce shortages and variability in care. Future work includes validating MAI-DxO in everyday clinical environments, developing diagnostic corpora that mirror real-world prevalence, supporting large-scale interactive medical benchmarks through synthetic findings, enhancing medical education, and incorporating visual and other sensory modalities for higher diagnostic accuracy.
References: Nori, H., Daswani, M., Kelly, C., Lundberg, S., Ribeiro, M. T., Wilson, M., … Horvitz, E. (2025). Sequential Diagnosis with Language Models. arXiv preprint arXiv:2506.22405v1.
