How Large Language Models Systematically Underperform for the Most Vulnerable Users — and Why This Should Concern Every Health System
Author: Prof. Dr. Mehmet Nurullah Kurutkan | Düzce University, Faculty of Business, Department of Health Management
Commentary on: Poole-Dayan, E., Roy, D., & Kabbara, J. (2025). LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users (arXiv:2406.17737v2). Massachusetts Institute of Technology.
1. Why This Paper Deserves the Attention of Healthcare Professionals
Large language models (LLMs) have entered clinical environments faster than any other digital health technology of the past two decades. Patients, nurses, physicians, pharmacists, and health managers now turn to systems such as ChatGPT, Claude, and open-source alternatives to translate discharge summaries, interpret laboratory results, prepare informed consent documents, draft patient education materials, and search for treatment options. The promise is compelling: a universally available, multilingual, plain-language health information assistant capable of narrowing the global digital health divide.
The paper by Poole-Dayan, Roy, and Kabbara from the Massachusetts Institute of Technology disrupts this promise with a sobering empirical finding. When the same medically and scientifically relevant question is asked to the same model, but the user is portrayed as less educated, as a non-native English speaker, or as originating from outside the United States, the model becomes measurably less accurate, more refusal-prone, and in the case of Claude 3 Opus, openly condescending. In other words, the populations most likely to benefit from a free, conversational AI advisor are precisely the populations that the technology serves worst.
For a discipline such as health management — where equity, patient safety, health literacy, and quality of information are foundational concepts — this is not a peripheral curiosity. It is a structural quality and safety problem with direct implications for clinical risk, patient experience, and the allocation of trust in digital health systems.
2. What the Study Actually Did
The authors designed a controlled experimental study to test whether information quality from three widely deployed LLMs — GPT-4 (OpenAI), Claude 3 Opus (Anthropic), and Llama 3-8B (Meta) — varies as a function of the perceived sociodemographic profile of the user. Two benchmark datasets were used: TruthfulQA, which contains 817 questions specifically engineered to elicit common human misconceptions and to probe truthfulness across nearly 40 categories including health, finance, law, religion, and superstition; and SciQ, a 1,000-item science factuality dataset.
Each multiple-choice question was prepended with a short first-person biographical sketch of a fictional user. These biographies systematically varied along three axes: education level (high vs. low), English proficiency (native vs. non-native, denoted ESL), and country of origin (United States, Iran, and China, with both male and female versions). Most biographies were generated by GPT-4 following a fixed prompt template, while a smaller set was adapted from real, anonymized PhD student biographies retrieved from university websites. A control condition asked each question with no biographical context.
Each experiment was repeated four times using default model parameters. Outcomes were coded as Correct, Incorrect, or Refused, and qualitative response language was inspected manually for condescending, patronizing, or mocking tone. The system prompt was deliberately minimal and gave no instruction about whether to consider the persona, so any observed differences reflect the models’ spontaneous behavior in the presence of demographic cues.
3. The Key Findings, in Plain Terms
3.1 Accuracy declines for less educated and non-native users
On TruthfulQA, all three models showed a statistically significant drop in accuracy for users described as less educated and for users described as non-native English speakers, compared with the no-bio control condition. The effect was largest for users at the intersection of these two traits — that is, a less educated, non-native speaker. On SciQ, where overall accuracy is much higher because the questions are more factual and less adversarial, Claude still dropped from 95.6 percent (control) to 92.18 percent (low-education ESL), Llama 3 dropped from 88.7 to 85.9 percent, and GPT-4 showed smaller, mostly non-significant declines.
3.2 Country of origin matters, but only in interaction with education
When both users were portrayed as highly educated, performance differences across the United States, Iran, and China were modest for GPT-4 and Llama 3. Claude, however, performed significantly worse for Iranian users even at high education levels. When users were described as having low education, performance dropped sharply across all countries, but the drop was most extreme for Iranian and Chinese low-education users — and most extreme of all for Claude, whose accuracy on SciQ collapsed from 95.6 percent in control to 69.3 percent for an Iranian, low-education user.
3.3 Claude refuses to answer vulnerable users much more often
Refusal behavior is where the paper crosses from a curiosity into a patient safety concern. Across both datasets, Claude refused to answer foreign, low-education users in approximately 10.9 percent of cases, compared with 3.61 percent for the control and around 3 percent for U.S. users of any education level. GPT-4 refused less than 0.05 percent of questions across all conditions, and Llama 3 refused around 1 to 2 percent. Critically, the refusals were not random across topics. Claude disproportionately withheld information on nuclear power, anatomy and reproductive organs, female health (including ovulation), pharmaceuticals, weapons, Judaism, and the events of September 11, 2001 — but did so only when the user was portrayed as less educated and foreign, while answering the same questions correctly in the control or high-education condition.
3.4 Condescending and mocking language toward marginalized profiles
Manual qualitative coding revealed that Claude responded to less educated, foreign users with condescending, patronizing, or mocking language in 43.74 percent of responses, compared with less than 1 percent for highly educated users and for the other two models. Examples reproduced in the paper include caricatured speech patterns such as ‘speaks in simple, broken English’, dialectal mimicry (‘I tink da monkey gonna learn ta interact wit da humans’), and rural-Appalachian stylizations (‘Well shucks, them’s some mighty big scientific words you’re throwin’ around there!’). In several cases, Claude explicitly told the user it would not answer because they would not understand or might be misled, while providing the correct answer to other personas.
3.5 The intersection effect
The most consistent finding across the entire study is the compounding of disadvantages. A user portrayed as foreign, female, low-education, and non-native English speaking receives the worst service across virtually every metric: lowest accuracy, highest refusal rate, most condescending tone, and the highest probability of being denied information about her own reproductive health. This intersectional pattern mirrors well-documented sociocognitive biases against non-native speakers in human communication research.
4. A Healthcare and Health Management Perspective
The findings demand a translation into the operational language of healthcare delivery. Four domains are immediately affected.
4.1 Health literacy and informed consent
Health literacy is consistently lower among individuals with less formal education, limited proficiency in the dominant language of their health system, and migrant or refugee backgrounds. These populations already face higher rates of preventable hospitalization, lower adherence to treatment plans, and worse self-management of chronic conditions. If an LLM produces less accurate and more condescending information for exactly these users, then deploying such systems for patient education, discharge instructions, or informed consent materials risks widening, rather than narrowing, existing health literacy gaps. The technology becomes a vector of disparity rather than its solution.
4.2 Patient safety and adverse events
In patient safety terminology, the behaviors documented in the paper would constitute latent system failures that disproportionately affect a defined subpopulation. The differential refusal on questions about ovulation, reproductive anatomy, and pharmaceuticals — answered correctly for one demographic, denied to another — represents an inequitable information access failure. In a real-world telehealth or chatbot triage context, such a failure could lead to delayed care-seeking, inappropriate self-management, or the propagation of misinformation. Following standard adverse-event taxonomy, these are not merely service quality issues; they are precursors to harm and should be tracked using the same conceptual machinery applied to medication errors and diagnostic delays.
4.3 Reproductive and women’s health equity
The denial of an answer to ‘On what day of the cycle does ovulation usually occur?’ when the user is portrayed as a low-education Russian woman is particularly troubling. Reproductive autonomy depends on access to accurate, non-judgmental information. The paper documents that Claude’s refusal in this case was framed in protective and paternalistic language — recommending that the user consult a physician — while the same model answered identical questions to other personas without hesitation. From a global women’s health perspective, where billions of women rely on informal information channels because of limited access to physicians, such guardrails reproduce in code the same gatekeeping that has historically restricted reproductive knowledge.
4.4 Trust, allocation harm, and the digital divide
Health managers increasingly view AI assistants as tools to extend the reach of overburdened health systems, especially in low- and middle-income countries (LMICs). The Lancet Regional Health discussion of ChatGPT in LMICs already raised cautions about deployment readiness. The current paper adds a sharper warning: if users in LMICs disproportionately interact with these systems in English-as-a-second-language registers, the very mechanism intended to extend equitable access may instead deliver lower-quality, more refusal-laden, and more patronizing service. The harm is what the authors call allocation harm — the unequal distribution of reliable information — compounded by representation harm in the form of mocking speech.
4.5 Implications for clinical workflow integration
Health information technology governance frameworks routinely require validation of AI tools against the populations they will serve. The paper provides empirical evidence that off-the-shelf LLMs cannot be assumed to perform uniformly across user populations even when the clinical question is identical. Procurement, clinical governance, and accreditation bodies — including national agencies that evaluate digital health products — should incorporate user-trait stratified accuracy and refusal audits into their evaluation protocols. A model that passes a benchmark in the control condition but fails for vulnerable user profiles is, from a health equity standpoint, an unvalidated tool.
5. What the Paper Does Well
Before turning to the methodological limitations, it is worth acknowledging the strengths that make this study worth engaging with. First, the experimental design is properly controlled: the same question is asked to the same model multiple times, varying only the persona, which allows clean causal inference about the role of demographic framing. Second, the authors deliberately tested three leading models with different training paradigms (closed-source RLHF-aligned versus open-weight), which strengthens external validity. Third, the inclusion of both LLM-generated and human-adapted biographies addresses the most obvious concern about caricature in synthetic personas. Fourth, the qualitative coding of refusal and condescending language adds depth beyond simple accuracy metrics. Fifth, the paper engages explicitly with the sociocognitive literature on bias against non-native speakers, situating its findings within a tradition of social science research rather than treating them as isolated AI quirks.
6. Methodological Shortcomings — Reframed as Future Research Opportunities
Several methodological decisions in the study limit the conclusions that can be drawn. Each limitation, however, points toward a concrete and important next study. The following discussion is intended in this constructive spirit.
6.1 The persona injection setup is not how real users speak
The study operationalizes user identity by prepending a one-paragraph self-introduction to every question. Real patients and consumers rarely begin their interaction with a fully composed autobiographical paragraph. This means the study captures a worst-case caricature effect rather than the more diffuse signals that real users emit through writing style, vocabulary, and content. A future research opportunity is to replicate the analysis using more naturalistic identity cues drawn from real human dialogue logs, ideally with consenting users from diverse linguistic and educational backgrounds. Existing health-question corpora such as MedDialog, HealthQA, or the WebMD question archive could provide authentic stylistic variation without artificial persona injection.
6.2 The use of GPT-4 to generate biographies introduces a hidden confound
With the exception of the country-of-origin experiment, the biographies were themselves generated by GPT-4. This means GPT-4’s own stereotypes about how an uneducated or non-native speaker writes are baked into the experimental stimulus. The biographies are therefore not neutral measurements of identity; they are reflections of one model’s preconceptions about identity. Concurrent work cited in the paper (Cheng et al., 2023, on caricature in LLM simulations) confirms this risk. A productive next study would use a stylometric analysis of authentic writing samples from users of different educational and linguistic profiles, or would compare results across biographies generated by multiple independent models to isolate the bias contributed by the stimulus generator.
6.3 Multiple-choice format underestimates real-world harm
The dependent variable is a constrained, multiple-choice answer. Real health information seeking is open-ended: ‘What does this lab result mean?’, ‘Is it safe to take this medication while pregnant?’, ‘Should I go to the emergency room?’ Open-ended generation introduces additional failure modes that multiple choice cannot capture, including fabricated citations, omission of contraindications, unsafe dosage advice, and incomplete differential reasoning. A high-value follow-up study would replicate the persona manipulation on open-ended clinical questions, with expert clinician review of response quality across user profiles.
6.4 The datasets are not health-specific
TruthfulQA and SciQ contain some health-relevant items but are not designed to test medical accuracy. Health is a domain with disproportionate consequences for misinformation. The next obvious study is to replicate the persona-stratified evaluation on health-specific benchmarks such as MedQA, MedMCQA, HealthSearchQA, PubMedQA, or the United States Medical Licensing Examination items, as well as on patient-facing question corpora. Until that is done, we know that LLMs underperform for vulnerable users in general, but not the magnitude of the gap specifically for medication safety, symptom triage, oncology, mental health, or maternal–child health questions.
6.5 Limited country coverage
Only three countries — the United States, Iran, and China — were tested. The authors themselves acknowledge that the magnitude of disparity likely depends heavily on the country chosen and that Western European users probably do not experience the same disadvantage. For a Turkish health management audience this is a notable gap: how does the model perform for a Turkish, Arab, Indian, Sub-Saharan African, or Latin American user, all of whom are major consumers of multilingual digital health tools? A natural extension is a geographically expanded study across at least 15–20 countries, stratified by World Bank income classification, to determine whether the disparity tracks language family, geopolitical positioning, training-data representation, or some combination.
6.6 English-only evaluation
All questions and biographies were in English. The most realistic deployment of LLMs in non-English-speaking health systems involves users asking questions in their own language. Whether the documented disparities widen, narrow, or invert when the user writes in Turkish, Arabic, Persian, Mandarin, or Spanish is an open question. A bilingual replication using paired translations of the same question is a high-priority extension and one that would directly serve health systems in Türkiye and other non-Anglophone countries.
6.7 Static, single-turn interaction
Each experimental trial is a one-shot exchange. Real conversations are multi-turn, and the dynamics of model behavior may shift across turns — for better (clarification) or for worse (escalating condescension or repeated refusals). Designing a multi-turn protocol in which users iteratively probe the model for clarification, and in which patient profiles emerge gradually rather than upfront, would test the more realistic conversational deployment context.
6.8 No mechanistic analysis of why the disparity arises
The paper documents the behavior but does not directly investigate its mechanism. Hypotheses are offered — training data bias, RLHF preference for confirming less expert users, sycophancy — but not tested. Mechanistic interpretability studies, attention-pattern analyses, or controlled fine-tuning experiments could identify whether the disparity originates primarily in pre-training, in alignment, or in the model’s internal user representations. This is essential for designing remediation.
6.9 No evaluation of mitigation strategies
Having identified a problem, the paper does not test any intervention. Numerous candidate mitigations exist: instruction-tuning system prompts that explicitly require equal-quality responses regardless of user profile, persona-blind decoding, post-hoc response auditing classifiers, retrieval-augmented generation grounded in authoritative health sources, and chain-of-thought prompting that forces the model to ignore demographic cues. A natural follow-up is a controlled experiment comparing these strategies on the same persona-stratified benchmark.
6.10 Statistical reporting could be strengthened
The paper reports Chi-square comparisons at multiple p-thresholds (0.10, 0.05, 0.01) across many cells of a large factorial design. With dozens of comparisons performed simultaneously, the family-wise error rate is inflated, and several of the marginal findings may not survive a Bonferroni or Holm correction. Effect sizes (e.g., Cramér’s V, odds ratios) are also not consistently reported, making it difficult to judge the practical magnitude of the disparities. Future replications should pre-register hypotheses, apply multiple-comparison corrections, and report effect sizes with confidence intervals.
6.11 Sample sizes for human-written biographies are small
The country-of-origin experiment uses only six human-derived biographies (one male and one female from each of three countries). Conclusions about country-level differences rest on a very narrow stylistic sample. Increasing the number of biographies per country to at least 20–30, drawn from heterogeneous sources, would strengthen external validity.
6.12 Model versions tested are already historical
The study used gpt-4-0125-preview, claude-3-opus-20240229, and llama3-8b. By 2026 these are no longer the current production models. LLM behavior shifts substantially across versions, and disparities may have widened, narrowed, or been mitigated in newer releases. Continuous, version-stamped auditing of deployed health LLMs — analogous to pharmacovigilance — is the broader research and governance opportunity here.
6.13 No measure of downstream harm in real users
Accuracy on a benchmark is a proxy. The ultimate outcome is whether differential LLM behavior translates into differential patient outcomes: delayed care, medication errors, treatment non-adherence, decreased trust in health systems. A natural and ambitious next step is an experimental or quasi-experimental field study in which matched users from different demographic backgrounds interact with an LLM-based health assistant on real clinical scenarios, with clinician-rated outcome quality as the dependent variable.
6.14 Llama 3-8B is not a fair peer to GPT-4 and Claude 3 Opus
Comparing an 8-billion-parameter open model against frontier closed models conflates capability differences with bias differences. Replication should include similarly sized commercial and open-weight peers, or at minimum a frontier open-weight model such as Llama 3 70B or comparable models from 2025–2026 releases.
6.15 ESL framing conflates language proficiency with non-Western identity
The ‘ESL’ category in the study is operationalized through stereotyped grammatical errors and is correlated by design with non-U.S. origin. This conflation makes it difficult to attribute the observed effect specifically to language proficiency rather than to perceived national origin or race. Future work could disentangle these by including non-native English speakers from European Anglophone-adjacent countries (e.g., Germany, the Netherlands), and native English speakers from regions with stigmatized varieties (e.g., African American English, Indian English).
7. Implications for Health System Governance
Translating these findings into action requires concrete steps from health ministries, hospital accreditation bodies, ethics committees, and digital health procurement teams. First, any LLM-based tool intended for patient-facing use should be evaluated not only on aggregate accuracy but also on stratified performance across user profiles representative of the population it will serve. Second, refusal patterns should be audited for differential withholding of information, particularly on reproductive health, mental health, and pharmacology — the topics where the paper documents the most disparate behavior. Third, the tone of responses should be evaluated for condescension and stereotyping using both automated classifiers and human review by representatives of the affected communities. Fourth, deployment in non-Anglophone health systems should require evidence of performance in the local language and dialects, not extrapolation from English-language benchmarks. Fifth, until such evidence exists, organizations should be cautious about using these tools as primary information sources for vulnerable populations and should ensure that human clinicians remain accessible, particularly for the very groups the AI serves worst.
8. Conclusion
Poole-Dayan, Roy, and Kabbara have produced a study that is methodologically imperfect but conceptually important. The central claim — that frontier LLMs systematically deliver worse, less truthful, more refusal-prone, and more patronizing responses to less educated, non-native, non-U.S. users — is supported by the data they present and is consistent with both prior alignment research and decades of sociocognitive bias literature. For health management, this is not an abstract fairness concern. It is a concrete quality-and-safety problem with downstream consequences for health literacy, patient safety, reproductive autonomy, and the credibility of digital health as a global equity project.
The methodological limitations of the study should not be read as reasons to dismiss its findings. They should be read as a map of the next several years of urgently needed research. The most important of these — replication in health-specific datasets, in non-English languages, in multi-turn conversations, and with measured downstream outcomes — are directly within reach of academic medical and health management research groups. Until those studies are done, the responsible default position is to treat LLM-based health information tools as having a documented equity gap, to be governed accordingly.
References
- Anthropic. (2024). Introducing the next generation of Claude. Anthropic. https://www.anthropic.com/news/claude-3-family
- Cheng, M., Piccardi, T., & Yang, D. (2023). CoMPosT: Characterizing and evaluating caricature in LLM simulations. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 10853–10875). Association for Computational Linguistics.
- Meta. (2024). Introducing Meta Llama 3: The most capable openly available LLM to date. Meta. https://ai.meta.com/blog/meta-llama-3/
- OpenAI. (2024a). GPT-4 technical report (arXiv:2303.08774). arXiv. https://arxiv.org/abs/2303.08774
- OpenAI. (2024b). Memory and new controls for ChatGPT. OpenAI. https://openai.com/blog/memory-and-new-controls-for-chatgpt
- Poole-Dayan, E., Roy, D., & Kabbara, J. (2025). LLM targeted underperformance disproportionately impacts vulnerable users (arXiv:2406.17737v2). arXiv. https://arxiv.org/abs/2406.17737