If we compare the boundless advancement of technology to a vast and complex castle, the European Union Artificial Intelligence Act (EU AI Act), adopted in 2024, can be seen as a strict guardian standing at its gates. While much of the discussion has focused on the stringent requirements imposed on “high-risk” AI systems, deep within the castle lies a dark Forbidden Forest where no one is permitted to tread. Hannah van Kolfschooten’s article, “Prohibited AI Practices in Healthcare under the European Artificial Intelligence Act,” shines a spotlight on this forest: the AI practices categorized under Article 5 of the Act as presenting an “unacceptable risk” and therefore subject to an outright ban.
What distinguishes this forest from an ordinary list of prohibitions is the severity of the sanctions enforced by the guardian. Violations may result in administrative fines of up to €35 million or 7% of global annual turnover. Moreover, while most provisions of the AI Act will become fully applicable by August 2026, these prohibitions entered into force early, in February 2025. This accelerated implementation sends a clear message: these practices are not merely risky; they are considered fundamentally incompatible with the legal and ethical foundations of the European Union.
A Fragile Ecosystem: Why Are Patients in the Crosshairs?
Healthcare is a domain characterized by structural vulnerability, information asymmetry, and emotional dependence. A patient may be likened to a ship attempting to navigate a stormy sea while relying almost entirely on physicians and medical technologies as a compass. This notion of vulnerability forms the analytical backbone of the article.
Although only Article 5(1)(b) explicitly prohibits the exploitation of vulnerabilities arising from age, disability, or socioeconomic circumstances, the concept permeates all prohibited practices. The underlying intuition is straightforward: a patient in a hospital bed, an elderly resident in a nursing home, or an individual with cognitive impairment cannot meaningfully control the AI systems surrounding them, even if they have formally “consented.” At this point, the AI Act acknowledges that procedural safeguards such as transparency and consent may be insufficient and therefore seals off certain practices entirely.
The Inhabitants of the Forest: Prohibited AI Entities in Healthcare
Drawing on the European Commission’s 2025 Guidelines, the author describes several categories of prohibited AI systems that may emerge in healthcare settings.
Mind Readers (Emotion Recognition and Biometric Categorization)
Imagine a hospital system that determines you are “angry” based on your facial expression and alters your treatment priority accordingly, or a system that infers your sexual orientation from facial features and uses this information to provide supposedly personalized care.
The AI Act generally prohibits systems that infer emotions from biometric data under Article 5(1)(f), though an important distinction exists. The prohibition on emotion recognition includes a narrow exception for strictly medical or safety-related purposes. In contrast, systems that infer sensitive characteristics such as race, religion, or sexual orientation from biometric data under Article 5(1)(g) receive no medical exemption whatsoever.
Consequently, even a rare disease diagnostic system modeled after DeepGestalt that infers ethnic origin from facial patterns, or the Wang and Kosinski model claiming to predict sexual orientation from facial images, remains prohibited regardless of its stated purpose.
Puppet Masters (Subliminal Manipulation and Exploitation)
Digital companions that exploit cognitive decline among lonely elderly patients to sell unnecessary services, or chatbots that subconsciously steer patients toward medication adherence, may fall under the prohibitions of Articles 5(1)(a) and 5(1)(b).
The article draws a crucial distinction between transparent, accountable clinical persuasion and AI-driven influence. Human persuasion is relational, intermittent, and subject to moral judgment. AI influence can be continuous, scalable, and so subtle that patients may not even recognize it. Emerging neurotechnologies such as brain-computer interfaces further blur this boundary.
Social Scoring Enforcers
Consider a scenario in which noncompliance with a dietary regimen or vaccination schedule affects your ability to obtain a loan or secure employment. AI-driven social scoring systems that evaluate individuals based on behavior and transfer those evaluations across unrelated contexts are prohibited under Article 5(1)(c).
The SCHUFA case, which focused on credit scoring, highlighted the dangers of cross-context profiling. In contrast, an insurer calculating risk strictly within the insurance context remains permissible. The issue is not scoring itself but the transformation of contextual assessments into generalized mechanisms of reward and punishment.
Fortune-Telling Judges (Predictive Criminal Profiling)
One of the lesser-known inhabitants of the forest intersects unexpectedly with healthcare. Article 5(1)(d) prohibits systems that predict criminal behavior or recidivism solely on the basis of profiling or personality traits.
Examples might include systems predicting whether psychiatric patients will commit crimes after discharge, estimating the likelihood of relapse into illegal substance use, or forecasting future domestic violence offenses based exclusively on psychological profiles.
During the COVID-19 pandemic, a system predicting compliance with lockdown restrictions using medical and socioeconomic data would likely have fallen within the scope of this prohibition. The red line is clear: absent concrete evidence of wrongdoing, individuals cannot be judged based solely on assumptions about who they are.
Face Thieves (Untargeted Image Scraping)
Among the forest’s most insidious inhabitants are systems nourished by images collected without consent. Article 5(1)(e) prohibits the creation of facial recognition databases through indiscriminate scraping of internet images or surveillance footage.
Importantly, the prohibition extends not only to those who collect the data but also to organizations deploying models trained on such datasets. Hospital facial recognition systems, nursing home monitoring tools, or pandemic surveillance applications may become prohibited if trained on scraped social media or public camera footage.
The article offers a practical warning: healthcare providers must be able to demonstrate that their facial recognition technologies were not trained using prohibited data collection methods. Otherwise, deployment should be refused.
Guardians of the Public Square (Real-Time Remote Biometric Identification)
Article 5(1)(h) prohibits real-time remote biometric identification in public spaces by law enforcement authorities, except in narrowly defined circumstances.
Most healthcare applications fall outside this prohibition. Facial recognition check-in systems in hospitals or internal security monitoring in psychiatric facilities are generally not covered if they are not operated on behalf of law enforcement and are not deployed in public spaces.
Nevertheless, boundaries can blur. During the pandemic, systems using thermal cameras to identify individuals with elevated body temperatures in airports or train stations could potentially have triggered the prohibition if used by law enforcement agencies.
Safe Paths Through the Forest: What Remains Permissible?
Not every corner of the forest is dark. One of the article’s overlooked contributions is its identification of lawful pathways.
Systems classifying skin tone or eye color in dermatology and oncology do not aim to infer sensitive characteristics and therefore remain permissible. CE-marked therapeutic tools designed to detect early signs of autism or prevent crises among individuals with severe depression may benefit from narrow exemptions. Systems monitoring postoperative pain or fatigue without inferring emotional states generally fall outside the prohibition.
Similarly, facial recognition technologies trained on ethically sourced and consent-based datasets, as well as insurance risk assessments confined to their original context, remain legitimate. The AI Act targets not technology itself but its purpose and context of use.
Cracks in the Castle Wall: Trojan Horses and Exceptions
The article’s most compelling argument concerns the exceptions embedded within Article 5. Although the provision appears strict, it contains narrowly defined exemptions for medical and safety purposes. Van Kolfschooten warns that these exceptions may function as Trojan Horses within healthcare.
An application may circumvent prohibitions simply by being labeled “medical.” The irony is striking: the populations most likely to be affected by these exceptions are precisely those whom the prohibitions seek to protect, including psychiatric patients, children with developmental disorders, and individuals in institutional care.
To address this vulnerability, the author proposes three conditions that should be met simultaneously before an exception is considered defensible:
- Demonstrable therapeutic necessity.
- The least intrusive design capable of achieving the intended objective.
- Concrete evidence that patient benefits outweigh losses in autonomy.
These principles should be embedded throughout the governance structure. Providers should document necessity and proportionality in technical files. Deployers should conduct fundamental rights impact assessments. Procurement contracts should include audit obligations, proportionality clauses, and shutdown triggers. Hospital ethics committees should evaluate impacts on autonomy alongside clinical safety.
Most importantly, patients should retain the right to refuse AI-mediated interventions without losing access to care and should always be offered a clinically viable alternative.
Beyond Europe: Who Holds the Compass?
The significance of these prohibitions extends beyond Europe. By declaring certain AI practices fundamentally incompatible with human dignity and fundamental rights, the AI Act sends a global normative signal.
Other countries face a choice: align with the European model and establish similar red lines, or permit practices prohibited within the EU. The former may contribute to a shared global baseline for patient protection; the latter risks creating regulatory havens for ethically problematic AI and exacerbating global health inequalities.
For countries such as Türkiye, the key question is not merely how to regulate high-risk AI systems but which applications should be deemed unacceptable from the outset. Through voluntary compliance by multinational vendors and procurement requirements imposed by major healthcare systems, these prohibitions are already extending beyond European borders.
Uncharted Paths: Future Research Opportunities
The article opens several promising avenues for future research.
First, it remains largely doctrinal and normative. Systematic mapping of deployed healthcare AI systems against Article 5 categories could transform speculative concerns into measurable realities.
Second, concepts such as “therapeutic necessity” and “least intrusive design” require operationalization. Hospitals need validated assessment tools, scoring rubrics, and decision-support frameworks capable of evaluating proportionality.
Third, the article pays limited attention to organizational capacity. Who will conduct audits? What resources will be required? How can resource-constrained healthcare systems absorb these obligations? Research informed by implementation science and health technology assessment could address these questions.
Fourth, vulnerability itself deserves reconsideration. The AI Act treats vulnerability as relatively static, linked to age, disability, or socioeconomic status. In reality, vulnerability is dynamic, contextual, and intersectional. Developing healthcare-specific models of situational vulnerability could significantly enhance regulatory effectiveness.
Fifth, a regulatory gap exists between detecting expressions and inferring emotions. A system may avoid claiming to identify emotions while still labeling someone as distracted or disengaged. As AI capabilities evolve, regulatory categories may require periodic revision or sunset mechanisms.
Sixth, the perspectives of patients and clinicians are largely absent. Empirical studies examining preferences, concerns, and experiences with prohibited or borderline AI systems would ground the debate in real-world evidence.
Seventh, the article’s skepticism toward AI may occasionally underestimate genuine clinical benefits. Certain emotion-aware systems could provide meaningful support in autism care or severe depression management. Rigorous outcomes research is needed to determine where restrictions should be calibrated.
Finally, healthcare AI exists within a broader regulatory ecosystem encompassing the GDPR, the European Health Data Space (EHDS), and the Medical Device Regulation (MDR). Mapping interactions among these frameworks would provide a more complete picture of AI governance.
Conclusion: Calibrating the Digital Compass
Van Kolfschooten’s article serves as an ethical brake at a time when healthcare systems are increasingly captivated by digitalization and artificial intelligence. Its central message is simple yet powerful: compliance is not merely a matter of technical validation. Improving an AI system’s accuracy, transparency, or auditability does not alter the fundamental question of whether its purpose is legally and ethically acceptable in the first place.
The red lines established by Europe offer a digital compass that views patients not as data objects to be analyzed but as rights-bearing individuals deserving protection. Yet the effectiveness of this compass will ultimately depend less on the existence of prohibitions themselves than on whether exceptions remain sufficiently constrained to preserve the dignity-based boundaries they are intended to protect, and on how much light future research can bring to the paths that remain unexplored.
Reference:
European Commission. (2025). Commission guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (AI Act) (COM (2025) 5052 final).
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), 2024 O.J. (L 2024/1689).
van Kolfschooten, H. (2026). Prohibited AI practices in healthcare under the European Artificial Intelligence Act. Journal of Law, Medicine & Ethics, 1-10. https://doi.org/10.1017/jme.2026.10270