Artificial intelligence (AI) and machine learning (ML) hold immense potential to revolutionize healthcare by leveraging vast amounts of health data and advancing evidence-based care. These technologies promise greater efficiency, improved workflow, and enhanced prediction of health outcomes, with some algorithms even outperforming clinicians in specific diagnostic tasks in controlled settings.
However, this transformative potential is significantly threatened by biases inherent in health data, which can severely impact the effectiveness and equity of AI/ML systems. A critical factor contributing to this challenge is “health data poverty,” defined as the inability for individuals, groups, or populations to benefit from innovations due to insufficient or unrepresentative data. This often unseen factor directly contributes to the perpetuation and amplification of existing health disparities, leading to worsened inequity for already vulnerable populations.
The core issue lies in the fact that when the data used to train AI/ML models are biased, the digital tools themselves become corrupted. Rather than mitigating disparities, continuing to build technologies on marginalized datasets can perpetuate or even amplify them, meaning communities with lower health outcomes may continue to suffer despite technological advancements. Computers, often perceived as unbiased machines, will most frequently perpetuate or amplify the existing biases found in their source data. These biases can infiltrate the AI/ML development process at any of its three stages: Data, Model, and Implementation. They can originate from deficiencies in the data or tool itself (source), be magnified (amplifier), or emerge from unforeseen interactions of seemingly unbiased components. The full impact of these biases may not become apparent until later stages of development or may not be equally distributed across implementations.
Specific examples highlight the pervasive nature of these biases:
- Underrepresentation and Data Divides: Health data poverty results from certain people groups being underrepresented in generated health data, potentially leading to harm from new digital tools. The majority of available digital health data currently originates from wealthy regions with widespread adoption of electronic health records (EHR) and devices, leading to a “data divide” where poor and low-accessibility populations are not digitally captured and are thus underrepresented.
- Inaccurate Risk Assessments: Risk scores, such as the Framingham Risk Score, have been shown to predict a 20% lower risk of cardiovascular events for Black individuals compared to White individuals with similar clinical characteristics. This indicates that such scores may not adequately capture risk factors for some minority groups, making technologies developed with these datasets not generalizable to diverse populations, including children, ethnic minorities, older adults, and individuals with disabilities.
- Diagnostic and Treatment Errors:
- Clinical Diagnostics: Diagnostic tools are often designed for dominant populations. For instance, the rarity of darker skin in dermatology can lead to the underdiagnosis of various diseases. Similarly, genetic data frequently excludes minorities, denying these populations access to personalized treatment or diagnostic tools.
- Device Underperformance: Certain clinical devices, like pulse oximeters, can underperform in creating equitable assessments across different races and sexes; a study revealed that darker skin can lead to an overestimation of arterial oxygen saturation, affecting treatment interventions.
- Genetic Variations: Underrepresentation in genetics data leads to diagnostic mistakes and inappropriate pharmacological treatments, as genetic variations are crucial and must be adequately represented. Notably, genomic data from African, Polynesian, and Brazilian populations are often underrepresented or ignored.
- Radiology Challenges: AI algorithms trained on biased datasets in radiology may recognize the site where a radiograph was taken rather than the disease itself (e.g., COVID-19 pneumonia), resulting in low sensitivity and challenges in generalizing findings across different devices, clinicians, and institutions.
- Exacerbated Disparities in Outcomes: Across various medical specialties, applying AI models to populations not adequately represented in the training data frequently drives disparities in outcomes. This has been observed in cancer treatment due to inappropriate risk assessment and in ophthalmology due to a lack of demographic and pathological representation in public datasets.
- Trust and Access Issues: A lack of trust, particularly among historically disadvantaged groups, can make them reluctant to share data or participate in studies, further exacerbating data poverty. Unequal internet access, affecting the elderly, certain global regions, and rural areas, also limits digital data capture and access to digital healthcare solutions.
- Synthetically Generated Biased Data: Even attempts to address dataset imbalances by generating synthetic data can be problematic. If the underlying real data are biased, the synthetic data may fail to capture true proportions and could introduce new biases, potentially leading to poorer recognition rates for specific subgroups (e.g., black female faces in facial recognition algorithms).
To address these critical issues, the sources recommend a multi-pronged approach. There must be a strengthened effort to generate unbiased equitable data, ensuring that the data used for AI models are accurate and representative of diverse populations’ needs, including every race and sociodemographic group. Furthermore, it is crucial to cultivate an improved understanding of the limitations of AI/ML tools among clinicians and researchers throughout the entire development process, from input data generation to implementation. Finally, there is a call for rigorous regulation and continuous monitoring of the clinical outcomes of deployed AI/ML tools to identify factors needing rebalancing and to ensure the benefits truly extend to society. Building trust by assuring data confidentiality and security is also paramount, although caution is advised regarding the potential misuse of race/ethnicity-based data, even as it is recognized as necessary for uncovering systemic biases.
Reference for this article:
Paik, K. E., Hicklen, R., Kaggwa, F., Puyat, C. V., Nakayama, L. F., Ong, B. A., Shropshire, J. N. I., & Villanueva, C. (2023). Digital Determinants of Health: Health data poverty amplifies existing health disparities—A scoping review. PLOS Digital Health, 2(10), e0000313. https://doi.org/10.1371/journal.pdig.0000313
