Who Oversees the Overseer? AI Literacy and the Physician in the Loop

Consider a hospitalist at three in the morning. A sepsis alert fires for a patient who, to her eye, looks fine. The model behind it is proprietary, was validated at a hospital two countries away, and is right often enough to be taken seriously. She has, roughly, three options. She can follow the alert, she can ignore it, or she can question it: ask what the score is actually measuring, whether this patient resembles the ones it was trained on, and how much weight it deserves against her own clinical judgement. The law increasingly assumes she will take the third option. Whether she can depends less on the algorithm than on what she knows about it. That, in a sentence, is the problem of AI literacy, and it is among the most underappreciated problems in the governance of health AI.

The human in the loop is only as good as the human

The problem is not hypothetical, because the tools are already in use. In the American Medical Association’s 2026 survey of nearly 1,700 physicians, 81% reported using AI professionally, more than double the share in 2023. Europe is moving in the same direction. What the numbers cannot tell us is whether physicians are competent users or merely frequent ones.

The early evidence is sobering. In a reader study from Germany, radiologists read mammograms alongside suggestions they believed came from an AI system. When the suggestions were deliberately wrong, the accuracy of inexperienced readers fell below 20%, and even readers with more than 15 years of experience fell to about 45%. Experience softened the effect but did not remove it. A multicentre study in Poland pointed to a second risk. After AI-assisted colonoscopy was introduced, endoscopists’ adenoma detection rate in procedures done without AI fell from 28.4% to 22.4%. The design was observational and the finding needs replicating, but the direction is worrying. So we face two failure modes at once: physicians who defer too much to the machine, and physicians who become less able to practise without it.

Generative AI adds a third failure mode. In one study, researchers planted a single false detail, such as an invented laboratory value, in each of 300 clinical cases and gave them to six large language models. Depending on the model, between half and four-fifths of the answers built on the false detail instead of flagging it. Better prompting lowered the average rate from about 66% to 44%, which helps but still does not solve the problem. And because these language models are fluent and use confident prose, even a well-trained physician may miss the errors, let alone if the same physician is exhausted and sleep deprived after long hours of in-hospital duty. Training therefore has to be paired with safeguards that do not depend on one person’s vigilance at three in the morning.

Governance research finds the same pattern at the level of institutions. A recent review of well-known failures in sepsis prediction, care-management algorithms and oncology decision support identified a pattern of “compliance without understanding.” Organisations met the formal requirements, but the people operating the systems could not tell whether they were working. In the care-management case, an algorithm used past healthcare spending as a stand-in for health need. Because Black patients had historically received less care, and so cost less, the algorithm concluded they were healthier than they were. The same review names six competences that existing digital health literacy measures leave out: algorithmic literacy, bias awareness, data governance, critical appraisal of AI outputs, trust calibration, and interpreting explanations. Part of the reason these gaps go unnoticed is that nobody is measuring them. The most widely used tool, the eHealth Literacy Scale, has eight questions, was published in 2006, before the smartphone, and measures how confident people feel rather than what they can actually do. No validated instrument for AI health literacy exists yet. The lesson is uncomfortable for anyone who has treated “human in the loop” as a solution. A human in the loop who cannot evaluate the machine is not oversight. At best it is a formality, and at worst it gives false reassurance.

What the law asks, and what it quietly stopped asking

European law has, to its credit, noticed. Article 4 of the AI Act, which has applied since February 2025, originally required providers and deployers to ensure a sufficient level of AI literacy among their staff. For healthcare organisations, that meant general AI knowledge for clinical users, knowledge of the specific systems they use, and role-appropriate competences for technical staff and managers. Then came the Digital Omnibus on AI (Regulation (EU) 2026/1744), in force since July 2026. In the name of simplification, it replaced the duty to ensure literacy with a duty to take measures to support it, and made clear that no particular level must be guaranteed.

Europe has seen this before. The GDPR was meant to harmonise data protection, yet institutional scientific advisory boards and ethics committees applying the same provisions to health research reached very different conclusions, and data protection officers sometimes demanded consent the law did not require. The latitude the GDPR left to Member States explains part of that divergence, but not all of it: institutions bound by the same rules still read them differently. A duty to “support” AI literacy, with no level to reach, inevitably leads to the same outcome.

The Omnibus left Article 26 untouched, and that matters. Article 26 still requires hospitals that deploy high-risk AI to assign human oversight to people with “the necessary competence, training and authority,” to use systems according to their instructions, to monitor their performance, to keep logs, and to tell patients when such systems help make decisions about them. Article 14 still requires those systems to be designed so that their overseers can remain aware of automation bias. These obligations will apply from December 2027, or August 2028 for AI built into medical devices. Meanwhile, the European Health Data Space Regulation still requires Member States to train health professionals (Art. 83). The result is a curious one. The legislature softened the general literacy obligation while keeping a specific competence requirement that cannot be met without literacy. A hospital cannot credibly assign oversight to a “competent” physician who does not understand what she is overseeing.

Rights that need readers

There is a deeper reason to care, and it goes beyond compliance. As access to care comes to depend on digital tools such as electronic health records, telemedicine and AI-assisted triage, digital health literacy stops being a matter of convenience. Researchers have argued that digital inclusion is now a social determinant of health in its own right. In European legal terms, that makes digital health literacy a prerequisite for the effective enjoyment of the right to health protected by Article 35 of the EU Charter of Fundamental Rights. The organisational obligations that Article 35 places on health systems should be read with this in mind. A health system built around digital tools, which fails to make sure people can use them, risks undermining the very access it is meant to guarantee.

The right to explanation shows the point most clearly. The Court of Justice in Dun & Bradstreet Austria (C-203/22) confirmed that people subject to automated decisions are entitled to a clear and intelligible account of how the decision was reached, and Article 86 of the AI Act extends a similar right to certain high-risk systems. But an explanation that nobody can understand satisfies the letter of the law and nothing more. That is true for the patient, and it is just as true for the physician who must give the explanation. American lawyers will recognise the structure. The physician is, in effect, a learned intermediary between the algorithm and the patient. If the intermediary is not learned, the whole chain of explanation, consent and contestation breaks down. The costs of that breakdown also fall unevenly. Only 60% of Europeans have basic digital skills, and the gaps are widest among older people, migrants and people with less education, who are often the same patients most exposed to algorithmic bias. Generative AI has not removed that bias. In fact, when nine large language models were given the same emergency cases with only the patient’s background changed, cases labelled as Black, unhoused or LGBTQIA+ were more often sent for urgent care, invasive procedures or mental health assessment, and high-income patients were more often offered advanced imaging. Low AI literacy acts as a multiplier of disadvantage. A physician who can translate an AI output into plain language is frequently the only thing that interrupts that multiplication. Averages hide exactly these gaps, so AI literacy needs to be measured group by group rather than across the population as a whole.

From principle to practice

Diagnosing the problem is the easy part. The harder question is what a hospital should actually do on Monday morning. One practical answer so far is M-SHALF, the Modular & Stratified Healthcare AI Literacy Framework proposed by Palmieri and colleagues in npj Health Systems. It was built to turn the open-textured language of Article 4 into something operational. It arranges AI literacy into four domains (awareness, application, reflection and governance) which the authors describe as “successive stages of maturity in AI literacy and organisational capability.” Everyone starts with a shared core module, which then branches into tracks for clinicians, for administrative, legal and governance staff, and for technical staff. Its most useful feature may be its approach to assessment. Alongside quizzes and simulations, it uses audit reports, procurement checklists and records of when clinicians override AI recommendations. That last measure is the right one, because trust calibration is revealed in behaviour, not in self-report.

M-SHALF pairs naturally with a recent three-tier framework proposed by Tzortzatou-Nanopoulou. Its first tier is assessment infrastructure that measures demonstrated rather than perceived competence, which, she notes, does not yet exist for AI health literacy. Its second is an educational ecosystem running from medical school to continuing education. Its third is governance integration, including literacy impact assessments as a precondition for deploying AI. This need not mean a new layer of bureaucracy. The assessment can be built into procurement and run alongside technical validation, much as the NHS already uses its Digital Technology Assessment Criteria (DTAC) as a pre-deployment checklist. For medical education in particular, the building blocks are already on the table. The Commission’s 2021 study on digital health recommended baseline AI education for every medical student plus specialised training by field. That framework, drawing on the American Medical Association’s November 2025 policy on AI literacy in medical education, spreads this across training, from algorithmic bias in the pre-clinical years to hands-on decision support in clinical rotations and participation in hospital AI governance during residency. It also proposes OSCE stations, practical clinical exams in which trainees must interpret AI outputs.

Harder questions and some recommendations

None of this settles the hard questions, and it would be a mistake to pretend otherwise. What exactly counts as “necessary competence” under Article 26(2)? Is it a certificate, a demonstrated skill, or a track record of sensible overrides? Who pays for the training, especially in resource-strained public systems? How should liability be allocated when a trained physician reasonably overrides an AI recommendation and turns out to be wrong, or reasonably follows one that was wrong? And if relying on AI erodes skills, should we require regular practice without AI, much as pilots are required to fly manually?

These questions deserve their own debate, but some steps are already clear. Hospitals should treat Article 26(2) as a competence to be demonstrated rather than presumed, and should give clinicians who oversee AI the time and authority to do it properly. The monitoring and log-keeping duties in Article 26 should feed clinical governance, so that when a clinician notices a tool drifting or failing for particular patients, that observation reaches the provider and changes practice. Training should include deliberate practice without AI so that core skills are kept up. Patient portals that show AI-flagged results should explain them in layers, with a plain summary by default and more detail for those who want it, at the moment the result appears. Member States should use the EHDS rollout, which runs to 2031, as what Tzortzatou-Nanopoulou calls “a structured window of opportunity,” and use it together with the training obligation in Article 83, to build this capacity before the Article 26 deadlines arrive.

Conclusion

Return to the hospitalist at three in the morning. No regulation, however well drafted, will be at the bedside with her. What will be there is whatever she knows about the tool in front of her. The Digital Omnibus has made the general literacy obligation softer, but it has not made the underlying need any smaller. Instead, responsibility has shifted to hospitals, medical schools and the profession itself. If “human oversight” is to mean something more than a signature on a compliance form, the humans doing the overseeing need to understand what they are overseeing. Getting that right is less a technical challenge than an institutional one.