AI is a rapidly developing field with significant potential to transform healthcare delivery and improve health outcomes. From enhancing diagnostic accuracy to supporting precision medicine, AI is increasingly being integrated into modern healthcare systems. However, AI is not being integrated into healthcare systems uniformly: bias, technological limitations and the availability of supporting infrastructure all affect how widely it is adopted.
The effectiveness of AI systems depends on the quality and applicability of the training data. Where data is sparse – that is, limited or unrepresentative of a particular population – some groups end up underrepresented. Examples include patients with rare diseases and populations in parts of the world where health data is not collected in sufficient quantity or frequency, most often in low- and middle-income countries (LMICs). Underrepresentation in health datasets increases the risk of inaccurate and biased outputs resulting in poorer healthcare outcomes. These disparities raise concerns about data inequity in healthcare AI. Data inequity refers to differences in the availability and representativeness of data about different population groups, which can lead to unequal clinical outcomes. This blog post explores how unequal data representation affects healthcare delivery and why clinical AI literacy is an essential safeguard against data inequity in low-resource settings.
Data inequity in health AI
Although data is often perceived as objective, samples are rarely fully representative, and unrepresentative data can perpetuate and even amplify existing health disparities if not handled with critical awareness.
While AI offers possibilities for advanced clinical prediction and decision-making in healthcare, models trained on relatively homogeneous datasets that poorly represent global diversity have limited generalizability and increase the risk of incorrect AI-driven decisions. An AI system developed in one context cannot be assumed to work equally well across diverse healthcare settings. This is not a theoretical concern: it is already affecting real patient outcomes.
A 2022 global review found that U.S. and Chinese datasets were disproportionately overrepresented in clinical AI research, with almost all of the top ten databases and author nationalities originating from high-income or upper-middle-income countries. Consequently, LMICs remain significantly underrepresented, further perpetuating global health inequities.
Health data poverty: data representation gaps and their impact on healthcare delivery
Despite the rapid growth of AI in healthcare, many datasets still fail to adequately reflect the diversity of the populations they are intended to serve. This has been described as “health data poverty” — a situation in which underrepresented groups are less able to benefit from data-driven healthcare innovations. In practice, non-diverse datasets contribute to biased algorithms, diagnostic inaccuracies, and unreliable outputs in certain patient populations.
One notable example is the use of convolutional neural networks (a type of AI model widely used for image analysis) in skin lesion classification. Many of these systems are trained predominantly on images of white patients, with Black patients comprising only about 5–10% of datasets. As a result, these systems have been shown to be significantly less accurate for Black patients than their reported performance suggests.
These representation gaps reduce the effectiveness of AI-assisted healthcare delivery and pose particular risks for low-resource healthcare systems, where limited safeguards may increase reliance on potentially biased technologies. In this sense, biased AI systems risk perpetuating existing global health inequities rather than reducing them.
Why AI literacy matters more in low-resource settings
For healthcare professionals, AI literacy means understanding enough about how an AI tool was built and validated to use it safely: what it was designed to do, which populations its training data came from, what its known limitations are, and when its outputs should be questioned. How to build these skills across the healthcare workforce is already widely discussed. Our question here is why they matter even more in low-resource settings.
AI could be even more transformative in low-resource settings, where health burdens are disproportionately high. As one recent commentary puts it, these are “precisely the environments in which the efficiency and predictive capacity of AI could yield greatest dividends.” However, as the World Economic Forum has warned: “Without representative data, AI-powered tools can misdiagnose or fail to recognize conditions in underrepresented populations, deepening global health inequalities rather than reducing them.”
AI literacy is particularly important for healthcare professionals, who remain ultimately responsible for interpreting and applying AI-generated outputs in clinical settings. Accordingly, AI systems should be viewed as decision-support tools that require human oversight. Developing AI literacy among healthcare professionals is therefore both a patient safety requirement and an ethical necessity in ensuring safe, equitable healthcare delivery.
This is especially important in low-resource settings, where underlying data may be less representative and contextual biases more pronounced, increasing the potential for harm. The focus is therefore not on attributing responsibility to patients or underrepresented populations, but on strengthening clinicians’ capacity to recognize limitations and critically evaluate AI outputs. AI literacy does not require clinicians to understand a model’s architecture or inspect its training data, any more than a radiologist needs to be able to build an MRI scanner. It means knowing which questions to ask: was this tool validated on patients like mine, what are its known failure modes, and when should its output be overridden? Where the data behind a model is opaque, these questions are the main safeguard against being misled by a confident but biased output. This is why we see AI literacy as one of the most practical ways to counter data inequity at the point of care.
Conclusion
The growing integration of AI into healthcare highlights a persistent challenge: data inequity continues to shape unequal health outcomes. When training datasets fail to reflect global diversity, the benefits of AI risk being unevenly distributed, reinforcing existing healthcare inequalities rather than reducing them.
AI literacy will not fix unrepresentative data, but it changes what happens when that data reaches the bedside. Clinicians who know which questions to ask of an AI tool are better placed to recognise when its output may not apply to the patient in front of them, and to override it when it does not. That makes AI literacy one of the most immediate and practical safeguards against data inequity, particularly in low-resource settings where other safeguards are thin.
It is not the whole answer. Closing the data gap itself will take governance that supports fairer data collection and representation and holds developers accountable for how their models perform across populations, a question that deserves its own discussion. In the meantime, investing in clinicians’ AI literacy is something health systems can start on now.
Rob White is Head of Data Engineering at STARK UK and a Board Member of Melanoma Patient Network Europe, where he works on patient data rights and the regulation of health data and AI. White is a member of the Nanopoulos Foundation’s Task Force on Healthcare Systems & Regulation.
Priscilla Ayanda is a final-year law student in Kwara, Nigeria, and a member of the Medical Law Hub, specialising in health law, health systems regulation and global health governance. Ayanda is a member of the Nanopoulos Foundation’s Task Force on Healthcare Systems & Regulation.
