Drawing the Red Lines: The EU's First Formal Review of Prohibited and High-Risk AI

The European Commission released its first formal assessment on whether to expand the list of prohibited and high-risk AI under the AI Act. We unpack the 547 stakeholder responses, the 3 791 real-world incidents analyzed, and what this means for digital health vendors navigating the gray area of mental health chatbots.

Key points

• The European Commission's first review concluded no immediate changes are needed to the lists of prohibited or high-risk AI, largely due to a lack of empirical enforcement data.
• Mental health and self-help chatbots were flagged by Member States as a complex area; depending on their features, they could fall under the Medical Devices Regulation, be classified as high-risk, or face outright prohibition.
• An analysis of 3 791 reported AI incidents revealed that approximately 10% are likely addressed by the AI Act's product safety rules in Annex I, while around 30% required detailed analysis for high-risk or prohibited status.
• The Commission identified a critical potential regulatory gap where AI generating non-consensual deepfakes and child sexual abuse material does not currently meet the strict legal definition of a prohibited manipulative practice under Article 5.
• For medtech and digital health companies, the message is clear: the risk tiers are set for now, but gray areas are being heavily monitored for future amendments in 2026.

Source: Report from the Commission to the European Parliament and the Council: Report adopted pursuant to Article 112(1) of Regulation (EU) 2024/1689 on the need to review the list of prohibited AI practices and of high-risk AI systems listed in Annex III - European Commission, 2026

This spot is available. Reach clinicians, health-system leaders and medtech and pharma teams following AI in medicine. Sponsor the show

This episode is an AI-generated conversation summarising a public document; the hosts' voices are synthetic. It is for information only and is not medical advice. Always refer to the original source.

Full transcript: https://ai-in-medicine-podcast.vercel.app/episodes/drawing-the-red-lines-the-eu-s-first-formal-review-of-prohibited-and-h-811f72