The Food and Drug Administration (FDA) has made a significant pivot in its approach to regulating artificial intelligence (AI)-enabled medical devices, as outlined in its recent draft guidance. This framework represents a noteworthy evolution in how regulators are reconsidering software that can adapt and change over time.
Phil Englert, vice president of medical device security at the Health Information Sharing and Analysis Center (Health-ISAC), emphasized that the FDA’s draft guidance reflects an understanding of AI’s transformative nature while emphasizing essential principles such as patient safety, transparency, and accountability. This marks a collaborative recognition between technology and health stakeholders about the inherent dynamics of AI in the medical field.
The draft guidance, which is non-binding and scheduled for official release in 2025, requires manufacturers to consider unique risks associated with AI systems. These risks include model drift, bias, and data poisoning—concerns that arise from the algorithms adjusting their outputs based on new data. This new regulatory stance acknowledges that AI systems do not generate static outputs; hence, the FDA anticipates a departure from the traditional expectation of fixed models.
Englert noted that the FDA seeks to facilitate organizations in creating predetermined change-control plans. These plans should incorporate ongoing monitoring, robust testing, and enhanced auditability, effectively ensuring that AI systems maintain their integrity over time while managing the unique risks they present.
Predicting that the FDA will formalize the draft guidance next year, Englert stressed the magnitude of collaboration between medical device manufacturers and healthcare providers. He highlighted the necessity for organized governance policies that clearly delineate approved AI tools. Such measures are integral to safeguarding sensitive health data and managing the emerging risks inherent in rapidly evolving technologies.
“A regulated medical device is meant to be stable, accurate, and repeatable,” Englert explained. “AI changes over time; it evolves its output. The FDA aimed to acknowledge this reality and provide guidance so that manufacturers and healthcare professionals understand these additional risks.”
During an interview with Information Security Media Group (ISMG) conducted at the recent HealthSec conference in Boston, Englert elaborated on the FDA’s draft guidance and its implications for AI-enabled medical devices. The discussions included vital topics such as how the guidance addresses cybersecurity risks and the overarching concerns related to patient safety.
Moreover, he articulated how healthcare organizations could bolster their governance structures related to AI. Strengthening oversight mechanisms and cultivating staff awareness are essential steps that organizations can take as they navigate the complex landscape of AI applications in healthcare.
Englert also noted the release of Health-ISAC’s recent guidance titled “Policies and Safeguards for the Safe Use of AI.” This document outlines critical elements necessary for effective AI governance in healthcare. It serves as a foundational resource for organizations looking to align their practices with emerging regulatory expectations while ensuring the safe integration of AI technologies.
In addition to his current role at Health-ISAC, Englert possesses over 30 years of experience in technical and operational leadership within the healthcare and life sciences sectors. His previous positions include chief product officer at MedSec, a cybersecurity consulting firm that focuses on medical device manufacturers and hospitals, as well as a global leadership role in medical device cybersecurity at Deloitte. In these capacities, he developed comprehensive medical device security programs and engaged with diverse stakeholders to fortify cybersecurity practices.
The shift in the FDA’s regulatory framework is a telling sign of the challenges and opportunities presented by AI in healthcare. Through this guidance, Enlight and the regulatory body aim to strike a balance between innovation and patient safety, ensuring that as technology evolves, so too does the oversight that protects user safety and data integrity. By promoting awareness and accountability, the FDA is paving the way for a future where AI technologies can flourish alongside robust regulatory practices, ultimately benefiting both manufacturers and patients alike.