Picture a student driver. On the school’s practice course, they are brilliant. Parallel parking on the first try, hill starts clean as a whistle, weaving through slalom cones like a bird. The instructor says, “Our best student.” Then the student takes the wheel in real traffic. They panic at the first intersection, lose control of the tires in the rain, cannot change lanes on the highway.
You turn to the instructor and say, “But on the course they were doing all the maneuvers.” The instructor shrugs.
Now think of the practice course as hospital data, and the student driver as a medical artificial intelligence (AI) model. In their editorial in the International Journal of Medical Informatics, Cabitza and colleagues say exactly this: the vast majority of machine learning (ML) models in medicine are perfect on the course and fragile in the field. The number of publications grows faster than the body of evidence. Journals ask “what is the model’s accuracy score?” while the question that really matters — “does this model actually work in a real hospital, with a real patient, at a real decision point?” — gets forgotten.
Section 1 — Six fragilities, six everyday analogies
The authors identify six points at which medical ML models tend to crack. Let us open them up one by one through everyday examples.
1. Even the referees disagree (the ground truth problem)
If you gave the same essay to five different literature teachers, you might get a 95 from one and a 70 from another. So if we are going to teach an AI to “recognize a good essay,” what is the correct answer?
Medicine works the same way. The same breast biopsy shown to two pathologists can yield different conclusions; the same X-ray can be interpreted differently by two radiologists. The “correct answer” itself is contested. When you train a model on a single expert’s label, you are also teaching the model that expert’s mistakes. The authors say: collect labels from multiple experts, do not hide the disagreement but model it explicitly, and where possible fall back on more definitive references such as autopsy findings.
2. The fire alarm’s sensitivity setting (the threshold problem)
If the smoke alarm in your kitchen is too sensitive, it goes off when you make toast; if it is not sensitive enough, it stays silent during an actual fire. Saying it is “well calibrated on average” means nothing. What matters is its performance in your kitchen, given your cooking habits.
Models are evaluated using a score called AUC. AUC gives the average performance across all threshold values. For a clinician, this is not enough. If you are triaging chest pain in the emergency department, a false negative (sending a patient home) is far more costly; in a routine screening program, a false positive (an unnecessary test) is far more costly. The threshold is not neutral; it is a decision, a value, an ethical position. A model should be evaluated at the threshold at which it will actually be used.
3. The airport metal detector paradox (the prevalence trap)
Imagine an airport where only 1 in 100,000 passengers carries a dangerous object. Even with a 99 percent accurate detector, the overwhelming majority of alarms will be false alarms. Why? Because real cases are already so rare.
Medicine works the same way. Metrics such as accuracy, F1, MCC, and PPV shift dramatically with disease prevalence. The rarer a condition is, the lower the chance that a “positive” prediction is truly positive in the real world. If a paper reports 95 percent accuracy and you deploy the model in a different hospital with a different case mix, that 95 percent can turn into a completely different number. The authors urge us to look instead at calibration: when the model says “70 percent probability,” do seven out of ten such patients actually turn out positive?
4. The false precision of the bathroom scale (the point estimate problem)
Suppose your bathroom scale reads “72.347 kg.” Is that number really accurate to the third decimal place? No. The true value probably lies somewhere between 71.5 and 73.
Papers report model performance the same way, as point estimates like “AUC = 0.847,” then compare two models as if “0.847 vs 0.832” were a meaningful gap. With small samples, that difference is meaningless. The authors call for confidence intervals and statistical tests alongside every metric. Hiding uncertainty makes a model look better than it is.
5. Grandma’s home cooking has never been tested elsewhere (the lack of external validation)
Your grandmother’s menemen is legendary — on the stove at home. But if you put her in a restaurant kitchen, with a different pan, a different burner, and the need to produce thirty portions in ten minutes, can she still pull it off? We do not know. She has never cooked anywhere else.
Most models are tested in their own kitchen. They are validated on a piece of data split off from the same hospital, the same population. This is called internal validation, and the authors say it is not enough. Models need to be tested in another hospital, in another city, in another health system, and ideally prospectively (that is, before the model is actually deployed). Otherwise the reported figures are not real-world estimates but upper bounds under idealized conditions.
6. The GPS on a 2010-model phone (the time decay problem)
If you tried to navigate the city using the GPS app on a phone from 2010, you would not know about the new roads, the closed shops, the changed one-way systems. The map is the same; the world has changed.
Medical data is no different. Diagnostic criteria are updated, populations age, new laboratory methods emerge, disease prevalence shifts. Training a model on 2020 data and continuing to use it the same way in 2026 is like driving with that 2010 phone. The fix is temporal validation, continuous monitoring, and recalibration when needed. The model is not “deployed and done”; it is “deployed and watched.” This is what MLOps means.
Section 2 — The authors’ prescription
The editorial does not just complain; it offers a concrete list. Let us simplify the items.
First, take labels seriously. Multiple expert annotations; modeling disagreement rather than hiding it; reference standards such as autopsy where possible. Second, align the threshold with clinical reality. Evaluate the model at the right sensitivity setting for the fire alarm. Third, look at calibration. Replace “how accurate is the model?” with “when the model says 70 percent, is it actually 70 percent?” Fourth, report uncertainty. Ranges, not single numbers. Fifth, test it in another kitchen. External validation, prospective validation. Sixth, do not forget the model once it is deployed. MLOps, continuous monitoring, adverse event tracking.
Above all this comes a call: the responsibility does not lie with researchers alone. Journals must tighten their acceptance criteria; reviewers must refuse to approve work that lacks external validation; vendors must document the calibration of their models; institutions must direct their funds not to the highest accuracy score but to models that can be tracked and audited. When the author stands alone, so does the reform.
Section 3 — What the paper does best
The smartest move is reframing the conversation from “better algorithms” to “better evidence.” This does for medical AI what evidence-based medicine (EBM) did for clinical practice in the twentieth century. EBM said that clinical judgment cannot be taken seriously unless backed by systematic evidence. This editorial sets out the same agenda for AI: not the models themselves, but the evidence we have about them.
The second strength is specificity. Where many similar calls remain stuck in moral high language, the authors name names: the TRIPOD+AI reporting guideline, decision curve analysis, out-of-distribution (OOD) testing, local calibration. There is a prescription alongside the diagnosis.
The third strength is shared responsibility. Authors, reviewers, users, institutions, vendors — five links in the same chain. No single one of them can solve the problem alone, but if any one of them breaks, the chain breaks.
Section 4 — Where it falls short
The editorial is strong, but not perfect. It strains at three points.
First: the incentive structure. The authors invite journals to adopt stricter acceptance criteria. Yet the journal business model — built on impact factor, rapid publication, and high article volume — makes it difficult to voluntarily enforce a rule like “no external validation, no acceptance.” The call is right; the mechanism is unclear.
Second: foundation models go unmentioned. By 2026, much of the medical AI conversation has moved beyond classical supervised ML to multimodal foundation models, large language models, and generative clinical decision support systems. These are evaluated on entirely different criteria than AUC. The six fragilities apply to classical prediction models; part of the new wave sits outside this frame.
Third: the structural barrier to external validation. The call is clear: test in another hospital, on another dataset. But the fragmented nature of health data, the limits set by data protection laws such as GDPR, and the cost of multicenter trials place the “large independent dataset” bar out of reach for many groups. The risk is this: the new standard could create a world in which only large consortia produce “evidence” and everyone else produces only “publications.” That would be a new kind of inequality.
Section 5 — Why this matters for health management
For health managers, the real significance of this editorial is its move from the language of engineering to the language of governance. When a hospital buys an AI product, the questions we should be asking change. We used to ask, “What is the AUC?” From now on, we will ask, “Is this model’s external validation documented? Has it been calibrated to our patient mix? Is there a performance monitoring layer once it is deployed? What is the recall process when it triggers an adverse event?”
Recall Donabedian’s classic structure-process-outcome triad. It applies directly to medical AI. Structure: the methodological infrastructure of the team developing the model, the quality of the data, the ethical approvals. Process: validation, calibration, threshold selection, continuous monitoring. Outcome: clinical benefit, reduction in adverse events, patient satisfaction, cost-effectiveness.
For a healthcare institution, in other words, medical AI is no longer simply a technology procurement decision; it is a quality management question.
Conclusion
A stack of academic publications, however tall, is not clinical evidence. Cabitza and colleagues argue that the medical AI literature is currently exactly such a stack. A student driver who is 97 percent successful on the practice course is not a safe driver in real traffic. The coming years will be years of closing that gap: better labels, more realistic thresholds, more transparent uncertainty, broader validation, more continuous monitoring.
The question is this: the next time a vendor walks into your hospital to sell you a “97 percent accurate AI product,” what will you ask them?
Source: Cabitza, F., Jurman, G., Molinari, F., & Bellazzi, R. (2026). Why almost all ML models for medicine are wrong—and what we need for evidence-based medical AI. International Journal of Medical Informatics, 219, 106538. https://doi.org/10.1016/j.ijmedinf.2026.106538
Glossary of Abbreviations
| Abbreviation | Full Form | Meaning / Explanation |
| AI | Artificial Intelligence | — |
| ML | Machine Learning | — |
| EBM | Evidence-Based Medicine | — |
| AUC | Area Under the Curve | A model’s overall discrimination score |
| ROC | Receiver Operating Characteristic | The curve plotting true vs false positive rates |
| AUROC | Area Under the ROC Curve | The full name of AUC |
| PPV | Positive Predictive Value | The proportion of positive predictions that are truly positive |
| F1 | F1 Score | The harmonic mean of precision and recall |
| MCC | Matthews Correlation Coefficient | A balanced classification metric |
| OOD | Out-of-Distribution | Data that differs from what the model was trained on |
| MLOps | Machine Learning Operations | Processes for monitoring models in live deployment |
| XAI | Explainable AI | — |
| TRIPOD | Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis | A reporting guideline for clinical prediction models |
| TRIPOD+AI | — | The 2024 update of TRIPOD for AI models |
| GDPR | General Data Protection Regulation | The European Union’s data protection law |
| Calibration | — | The property that when a model says “X percent,” the real-world rate is X percent |
| Prospective validation | — | Testing the model on forward-looking, unused data |
| External validation | — | Testing the model on a dataset from outside where it was trained |
| Decision Curve Analysis | — | A method for measuring the net clinical benefit of a model |
| Net Benefit | — | A measure of the real clinical gain of using a model |
| Data Shift | — | A change over time in the distribution of the data |
| Calibration Drift | — | A model losing its calibration over time |
| Hold-out | — | A portion of the training data set aside for testing |