High-Risk Without Safeguards? The EU AI Act and the Deregulation of Medical AI
Authors/Creators
Description
Artificial intelligence (AI) is increasingly embedded in clinical decision-making, yet recent reforms to the European Union (EU) Artificial Intelligence Act (AI Act) risk creating a category of high-risk medical AI systems without corresponding safeguards. In the current framework, AI-enabled medical devices are regulated through a combination of sectoral medical device legislation, the Medical Devices Regulation (MDR) and In Vitro Diagnostic Medical Devices Regulation (IVDR), and additional AI-specific safeguards in the AI Act. Recent EU “simplification” initiatives, including the Digital Omnibus and the Health Innovation Package, propose to remove these safeguards. This narrative review analyses recent EU legislative developments and their impact on the relationship between the AI Act and medical device law. It shows that the proposed reforms would retain the high-risk classification of AI-enabled medical devices while removing most associated AI Act obligations. In effect, this decouples risk classification from the safeguards that give it regulatory meaning, including requirements on data governance, risk management, human oversight, and post-market monitoring. Rather than resolving regulatory overlap, the reforms shift the centre of gravity back to product-focused medical device law without ensuring equivalent AI-specific safeguards. These changes may narrow attention to fundamental rights, weaken oversight in clinical use, and increase legal uncertainty regarding accountability and responsibility. In a domain where AI systems directly shape clinical decisions and patient outcomes, these changes risk undermining the conditions for safe, equitable, and accountable deployment of medical AI.
Files
High-Risk Without Safeguards | WP 15.04.2026.pdf
Files
(752.3 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:01befd7b89ed449fff1f084ed00a4203
|
752.3 kB | Preview Download |
Additional details
Related works
- Cites
- Publication: 10.5281/zenodo.19094764 (DOI)
- Publication: 10.1007/s11023-025-09730-3 (DOI)
- Journal: 10.1093/jlb/lsad031 (DOI)
References
- 1. Van Kolfschooten H. The AI Cycle of Health Inequity and Digital Ageism: Mitigating Biases through the EU Regulatory Framework on Medical Devices. Journal of Law and the Biosciences 2023;10:lsad031. doi.org/10.1093/jlb/lsad031.
- 2. Leslie D, Mazumder A, Peppin A, Wolters MK et al. Does "AI" Stand for Augmenting Inequality in the Era of Covid-19 Healthcare? BMJ 2021:372;n304. doi.org/10.1136/bmj.n304.
- 3. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance) PE/24/2024/REV/1 OJ L, 2024/1689, 12.7.2024.
- 4. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance). OJ L 117, 5.5.2017, pp. 1–175.
- 5. Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in vitro diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU (Text with EEA relevance) OJ L 117, 5.5.2017, pp. 176-332.
- 7. Open Letter: No fragmentation of EU AI Act – Preserve the horizontal approach [online]. 11 March 2026. https://allai.nl/wp-content/uploads/2026/03/Open-Letter_AI-Omnibus_AI-Act_ANNEX-IA-to-IB.pdf (accessed 7 April 2026).
- 8. Van Kolfschooten H, Solaiman B, Onitiu D. Medical AI at Risk: Digital Omnibus Amendments Undermine Safeguards in Healthcare [online]. 24 March 2026. https://haiweb.org/storage/2026/03/Medical-AI-at-Risk.pdf (accessed 7 April 2026).
- 9. A simpler and faster Europe: Communication on implementation and simplification (European Commission 2024-2029) https://commission.europa.eu/document/download/8556fc33-48a3-4a96-94e8-8ecacef1ea18_en?filename=250201_Simplification_Communication_en.pdf.
- 10. Proposal for a Regulation of the European Parliament and of the Council amending Regulations (EU) 2024/1689 and (EU) 2018/1139 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI) (Text with EEA relevance) {SWD(2025) 836 final} COM(2025) 836 final, 2025/0359(COD).
- 11. Proposal for a Regulation of the European Parliament and of the Council amending Regulations (EU) 2017/745 and (EU) 2017/746 as regards simplifying and reducing the burden of the rules on medical devices and in vitro diagnostic medical devices, and amending Regulation (EU) 2022/123 as regards the support of the European Medicines Agency for the expert panels on medical devices and Regulation (EU) 2024/1689 as regards the list of Union harmonisation legislation referred to in its Annex I {SWD(2025) 1050-1052 final} (Text with EEA relevance) 16.12.2025, COM(2025) 1023 final 2025/0404 (COD).
- 12. European Commission. New measures to make EU health sector more innovative, competitive and resilient [online]. 16 December 2025. https://ec.europa.eu/commission/presscorner/detail/en/ip_25_3077
- 13. Proposal for a Regulation of the European Parliament and of the Council amending Regulations (EU) 2017/745 and (EU) 2017/746 as regards simplifying and reducing the burden of the rules on medical devices and in vitro diagnostic medical devices, and amending Regulation (EU) 2022/123 as regards the support of the European Medicines Agency for the expert panels on medical devices and Regulation (EU) 2024/1689 as regards the list of Union harmonisation legislation referred to in its Annex I {SWD(2025) 1050-1052 final} (Text with EEA relevance) 16.12.2025, COM(2025) 1023 final 2025/0404 (COD).
- 14. European Parliament. Artificial Intelligence Act: delayed application, ban on nudifier apps. [online] 26 March 2026. https://www.europarl.europa.eu/news/en/press-room/20260323IPR38829/artificial-intelligence-act-delayed-application-ban-on-nudifier-apps (accessed 8 April 2026).
- 15. Van Kolfschooten H, Van Oirschot J. The EU Artificial Intelligence Act (2024): Implications for Healthcare. Health Policy 2024;149:105152. doi.org/10.1016/j.healthpol.2024
- 16. Van Kolfschooten H. The AI Cycle of Health Inequity and Digital Ageism: Mitigating Biases through the EU Regulatory Framework on Medical Devices. Journal of Law and the Biosciences 2023;10:lsad031. doi.org/10.1093/jlb/lsad031.
- 17. Onitiu D, Wachter S, Mittelstadt, B. How AI Challenges the Medical Device Regulation: Patient Safety, Benefits, and Intended Uses. Journal of Law and the Biosciences 2024:lsae007. doi.org/10.1093/jlb/lsae007.
- 18. Amin A., Acharya D, Koteshwara P. et al. A systematic literature review on mammography: deep learning techniques for breast cancer detection with global and Asian perspectives. BMC Cancer 2025:25;1627. doi.org/10.1186/s12885-025-14876-5.
- 19. Oberije CJG, Currie R, Leaver A et al. Assessing artificial intelligence in breast screening with stratified results on 306 839 mammograms across geographic regions, age, breast density and ethnicity: A Retrospective Investigation Evaluating Screening (ARIES) study. BMJ Health & Care Inform. 2025;32:e101318. doi.org/10.1136/bmjhci-2024-101318.
- 20. Daneshjou R, Vodrahalli K, Novoa R. et al. Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci. Adv. 2022:8;eabq6147. doi.org/10.1126/sciadv.abq6147.
- 21. Gerke S, Babic B, Evgeniou, T. The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. NPJ Digital Medicine 2020:53;1-2.
- 22. Gichoya JW, Banerjee I, Bhimireddy AR et al, AI recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health 2022:4;406-414. doi.org/10.1016/S2589-7500(22)00063-2. 23. Gerke S, Babic B, Evgeniou, T. The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. NPJ Digital Medicine 2020:53;1-2. 24. Gerke S. Health AI for good rather than evil? The need for a new regulatory framework for AI-based medical devices. Yale J. Health Pol'y L. & Ethics. 2021;20:432. 25. Van Kolfschooten H. The AI Cycle of Health Inequity and Digital Ageism: Mitigating Biases through the EU Regulatory Framework on Medical Devices. Journal of Law and the Biosciences 2023;10:lsad031. doi.org/10.1093/jlb/lsad031. 26. Solaiman, B., Mekki, Y.M., Qadir, J. et al. A "True Lifecycle Approach" towards governing healthcare AI with the GCC as a global governance model. npj Digit. Med. 8, 337 (2025). https://doi.org/10.1038/s41746-025-01614-1; Solaiman B. From Bench to Bedside: Governing Health Care Artificial Intelligence (AI) through a "True Lifecycle Approach." American Journal of Law & Medicine. 2025;51(3-4):452-478. doi:10.1017/amj.2025.10091. 27. See also European Parliament resolution of 23 October 2024 on the urgent need to revise the Medical Devices Regulation (2024/2849 (RSP)) (29 January 2025) P10_TA (2024) 0028. 28. Joint paper of Croatia, Finland, France, Germany, Ireland, Luxembourg, Romania, Malta and Slovenia on necessary reforms in MDR and IVDR: priorities / main points (28 November 2024) 15380/24. 29. European Parliamentary Research Service, Briefing EU Legislation in Progress: Medical devices: Simplifying the rules (PE 785.663, March 2026) https://www.europarl.europa.eu/RegData/etudes/BRIE/2026/785663/EPRS_BRI(2026)785663_EN.pdf. 30. A recent empirical study finds finding that stakeholders primarily identify regulatory fragmentation across overlapping EU instruments, rather than the AI Act itself, as the main source of compliance challenges: Hacker P, Kilian R and Costas J. "Simplifying" European AI Regulation: An Evidence-based White Paper. Bertelsmann Stiftung, 2025. 31. AI Act, recital 64, art 11(2), art 43(3). 32. Regulation (EU) 2024/1689, recital 64; arts 8(2), 11(2), 43(3). Artificial Intelligence Board; Medical Device Coordination Group. Interplay between the Medical Devices Regulation (MDR) & In Vitro Diagnostic Medical Devices Regulation (IVDR) and the Artificial Intelligence Act (AIA). AIB 2025-1/MDCG 2025-6. June 2025. pp. 2–3. 33. Artificial Intelligence Board (AIB) and the Medical Device Coordination Group (MDCG). Interplay between the Medical Devices Regulation (MDR) & In vitro Diagnostic Medical Devices Regulation (IVDR) and the Artificial Intelligence Act (AIA). [online] June 2025https://health.ec.europa.eu/document/download/b78a17d7-e3cd-4943-851d-e02a2f22bbb4_en?filename=mdcg_2025-6_en.pdf. 34. Proposal for a Regulation of the European Parliament and of the Council amending Regulations (EU) 2017/745 and (EU) 2017/746 as regards simplifying and reducing the burden of the rules on medical devices and in vitro diagnostic medical devices, and amending Regulation (EU) 2022/123 as regards the support of the European Medicines Agency for the expert panels on medical devices and Regulation (EU) 2024/1689 as regards the list of Union harmonisation legislation referred to in its Annex I {SWD(2025) 1050-1052 final} (Text with EEA relevance) 16.12.2025, COM(2025) 1023 final 2025/0404 (COD). 35. Regulation (EU) 2024/1689, art 2(2). European Parliament. Amendments adopted on 26 Mar 2026 on COM(2025)0836. Amendment 27 (Art 2(2)). 36. Article 1(2) of Digital Omnibus on AI proposal: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52025PC0836. 37. Article 57 (3a) amendment Digital Omnibus. 38. This has been suggested in academic research before as an important regulatory feature for medical device regulation, see Li P, Williams R, Gilbert S. Regulating Artificial Intelligence and Machine Learning-Enabled Medical Devices in Europe and the United Kingdom. Law, Technology and Humans 2023:5;94-101. 39. Article 26 AI Act. 40. Recital 1 AI Act. 41. Almada M, Petit N. The EU AI Act: Between the Rock of Product Safety and the Hard Place of Fundamental Rights. Common Market Law Review 2025:62. 42. Solaiman B. The European Union's Artificial Intelligence Act and Trust: Towards an AI Bill of Rights in Healthcare? Law, Innovation and Technology 2025:17;318. doi.org/10.1080/17579961.2025.2469986. 43. Van Kolfschooten H. Towards an EU Charter of Digital Patients' Rights in the Age of Artificial Intelligence. Digital Society 2025:4. doi.org/10.1007/s44206-025-00159-w. 44. Mantelero A. The Fundamental Rights Impact Assessment (FRIA) in the AI Act: Roots, Legal Obligations and Key Elements for a Model Template. Computer Law & Security Review 2024:54;106020. doi.org/10.1016/j.clsr.2024.106020. 45. Pham BC, Davies SR. What problems is the AI act solving? Technological solutionism, fundamental rights, and trustworthiness in European AI policy. Critical Policy Studies 2025:19;318-336. doi.org/10.1080/19460171.2024.2373786. 46. Haider SA, Borna S, Gomez-Cabello CA. et al. The Algorithmic Divide: A Systematic Review on AI-Driven Racial Disparities in Healthcare. J. Racial and Ethnic Health Disparities 2026:13;188-217. doi.org/10.1007/s40615-024-02237-0 47. Arts. 59b (2) and 59c (1) of the simplification proposal. 48. AI Act, art 57 and art 60; Proposal for a Regulation ... COM(2025) 1023 final, proposed MDR/IVDR sandbox provisions; Proposal for a Regulation ... COM(2025) 836 final, proposed art 60a. 49. Celi LA, Cellini J, Charpignon ML et al. Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review. PLOS Digital Health 2022. doi.org/10.1371/journal.pdig.0000022. 50. Cau R, Pisu F, Suri JS et al. Addressing hidden risks: Systematic review of artificial intelligence biases across racial and ethnic groups in cardiovascular diseases. European Journal of Radiology 2025:183;111867. doi.org/10.1016/j.ejrad.2024.111867. 51. Longhurst CA et al. A Call for Artificial Intelligence Implementation Science Centers to Evaluate Clinical Effectiveness. NEJM AI 2024:1(8). doi.org/10.1056/AIp2400223. 52. Leslie D, Mazumder A, Peppin A, Wolters MK et al. Does "AI" Stand for Augmenting Inequality in the Era of Covid-19 Healthcare? BMJ 2021:372;n304. doi.org/10.1136/bmj.n304. 53. Van Kolfschooten H. The AI Cycle of Health Inequity and Digital Ageism: Mitigating Biases through the EU Regulatory Framework on Medical Devices. Journal of Law and the Biosciences 2023;10:lsad031. doi.org/10.1093/jlb/lsad031. 54. Regulation (EU) 2024/1689, art 26; Artificial Intelligence Board; Medical Device Coordination Group. Interplay between the MDR/IVDR and the AIA. AIB 2025-1/MDCG 2025-6. June 2025. p. 2. 55. Li MM, Reis BY, Rodman A. et al. Scaling medical AI across clinical contexts. Nat Med 2026:32;439-448. doi.org/10.1038/s41591-025-04184-7. 56. Abd-Alrazaq A, Solaiman B, Mekki YM, et al. Hype vs Reality in the Integration of Artificial Intelligence in Clinical Workflows. JMIR Form Res. 2025;9:e70921. doi.org/10.2196/70921. 57. Busuioc M, AI algorithmic oversight: new frontiers in regulation. In: Maggetti M, Di Mascio F, Natalini A. eds. Handbook of Regulatory Authorities. Cheltenham, UK: Edward Elgar Publishing 2022:470-486. 58. Abd-Alrazaq A, Solaiman B, Mekki YM, Al-Thani D, Farooq F, Alkubeyyer M, et al. Hype vs Reality in the Integration of Artificial Intelligence in Clinical Workflows. JMIR Form Res. 2025;9:e70921. pp. 1–2. 59. Fraser A, Butchart E, Szymánsky et al. The Need for Transparency of Clinical Evidence for Medical Devices in Europe. The Lancet 2018:392;521. doi.org/10.1016/S0140-6736(18)31270-4. 60. World Health Organization, Regional Office for Europe, Artificial intelligence is reshaping health systems: state of readiness across the WHO European Region (WHO, 19 November 2025) 24. 61. Morley J, Hine E, Roberts H et al. Global health in the age of AI: charting a course for ethical implementation and societal benefit. Minds Mach. 2025:35;31. 62. Coiera E. The Last Mile: Where Artificial Intelligence Meets Reality. Journal of Medical Internet Research 2019:21;e16323. 63. Baldwin, R. and Black, J. Really Responsive Regulation. The Modern Law Review 2008:71;59-94. doi.org/10.1111/j.1468-2230.2008.00681.x 64. Rothstein H, Huber M, Gaskell G. A theory of risk colonization: The spiralling regulatory logics of societal and institutional risk. Economy and Society 2006:35(1);91–112. doi.org/10.1080/03085140500465865. 65. Veale M. A Critical Take on the Policy Recommendations of the EU High-Level Expert Group on Artificial Intelligence. European Journal of Risk Regulation 2020;11(1):e1. doi.org/10.1017/err.2019.65 66. Van Kolfschooten HB, Petročnik T. Emerging Shadow Health Systems: Regulating Health-Focused Generative AI Chatbots from a European perspective. Zenodo 2026. doi.org/10.5281/zenodo.19094764. 67. Duffourc MN, Gerke S. The proposed EU Directives for AI liability leave worrying gaps likely to impact medical AI. npj Digit. Med. 2023:6;77. doi.org/10.1038/s41746-023-00823-w. 68. Del Rey Puech P, Saund J, Panteli D, McKee M. Demystifying artificial intelligence in health: what health policy-makers need to know. Copenhagen: European Observatory on Health Systems and Policies, WHO Regional Office for Europe; 2026. p. 177-187. 69. Open Letter: No fragmentation of EU AI Act – Preserve the horizontal approach [online]. 11 March 2026. https://allai.nl/wp-content/uploads/2026/03/Open-Letter_AI-Omnibus_AI-Act_ANNEX-IA-to-IB.pdf (accessed 7 April 2026). 70. Laux J, Wachter S, Mittelstadt, B. Trustworthy Artificial Intelligence and the European Union AI Act: On the Conflation of Trustworthiness and Acceptability of Risk. Regulation & Governance 2024:18. doi.org/10.1111/rego.12512.