Potential AI/ML Learnings to Come from FDA Public Advisory Committee Meeting on Skin Lesion Analyzer Technology in Late July

On July 28, 2022, the U.S. Food and Drug Administration (FDA) will hold a public advisory committee meeting to discuss skin lesion analyzer (SLA) technology and its application to detecting skin cancers in various patient care settings. This meeting of the General and Plastic Surgery Devices Panel of the Medical Devices Advisory Committee will focus on algorithm-based SLA devices for adjunctive detection of skin lesions, including skin cancers, and stands to provide industry another layer of thinking on FDA’s perspective on artificial intelligence and machine learning (AI/ML) device technologies.

In announcing this meeting, FDA explained that in recent years it has observed an increased interest in SLA devices employing AI/ML. The agency is seeking expert input from the panel on approaches to evaluate the performance of SLA devices, which have a range of technologies and indications.

The committee will discuss and provide recommendations to FDA on: (1) the diagnosing standard, or ground truth, that should be used as a comparison for the performance of diagnostic devices, e.g., histology, consensus opinion of a panel of dermatologists, opinion of a single dermatologist, or other means; (2) acceptable sensitivity and specificity thresholds based on the target diagnosis (melanoma, basal cell carcinoma, squamous cell carcinoma) or intended user (dermatologist, primary care physician, lay user); (3) patient characteristics, including lower or higher incidence populations, that should be tested before marketing; and (4) the balance of increased access with risk mitigation measures that are appropriate when the devices are used by lay people, by populations with very high or very low incidence of melanoma, by populations with low incidence, but high mortality associated with melanoma, or by the target diagnosis/lesion type.

Additionally, on July 29, 2022, the committee will discuss the possible reclassification of two class III, PMA approved computer-aided melanoma detection devices, MelaFind (P090012) and Nevisense (P150046), both of which are intended for use on cutaneous lesions suspicious for melanoma when a dermatologist chooses to obtain additional information when considering biopsy. According to the FDA announcement, “The committee will discuss if there is sufficient information to reclassify computer-aided devices for adjunctive diagnostic information of lesions suspicious for melanoma from class III to class II, and what special controls may be appropriate to provide reasonable assurance of safety and effectiveness” if they are reclassified.

This meeting, and any actions the FDA takes as a result, could offer industry further insight into the FDA’s approach to regulating AI/ML diagnostic and screening products more broadly.

The meeting will be held virtually on July 28, 2022, from 9 am to 5:45 pm ET and July 29, 2022, 9 am to 4 pm ET. Comments received on or before July 11, 2022 will be provided to the committee and the public docket will remain open for comment for FDA’s consideration until August 29, 2022.

For more information see the Meeting Notice on the Federal Register.

Things for Pharma and Biotech Companies to Watch in the Cures 2.0 Proposed Legislation

Last week, Diana DeGette (D-CO) and Fred Upton (R-MI) introduced in the House highly anticipated bill language for “Cures 2.0”, a follow-up to the transformational 21st Century Cures Act enacted in 2016.  For full text of the bill, click here.  The 21st Century Cures Act included a variety of measures seeking to accelerate medical product development and bring advancements and innovations to patients more efficiently. Cures 2.0 seeks to improve and expand on those strides, as well as address pressing public health priorities that became apparent through the COVID-19 pandemic.

The Cures 2.0 bill is structured around five main topics:

  • Title I—Public Health
  • Title II—Patients and Caregivers
  • Title III—Food and Drug Administration
  • Title IV—Centers for Medicare & Medicaid Services
  • Title V—Research

While all of these sections are ripe for further analysis, we selected a few provisions to highlight here that may be of particular interest for the pharmaceutical and biotechnology companies out there.  We’ll keep tracking these as the bill moves through the legislative process:

Section 204: Patient Experience Data

  • Would require sponsors developing a drug under an IND to collect standardized patient experience data during clinical trials and include that patient experience data “and such related data” in an NDA or BLA; and
  • Would direct FDA to consider this patient experience data and “related information” in its approval decision for the NDA or BLA.
  • These proposals to standardize and require patient experience data collection could be significant, and they underscore lawmakers’ continued interest in elevating the relevance of clinical outcomes that are meaningful to patients living with a disease or condition.

Section 302: Grants for Novel Trial Designs and Other Innovations in Drug Development & Section 310: Recommendations to Decentralize Clinical Trials

  • Section 302 would appropriate $25 million annually, for 3 years, for the FDA to award grants to clinical trials conducted under an IND with protocols incorporating complex adaptive or other novel trial designs and that collect patient experience data. The section further specifies that grant awards should prioritize the incorporation of digital health technologies and real world evidence.
  • Section 310 proposes a multi-stakeholder meeting, including industry representatives and patient advocacy groups, to discuss incentives to adopt decentralized clinical trials. The section also would adopt a definition of decentralized trials: “a clinical trial method that includes the use of telemedicine or digital technologies to allow for the remote collection of clinical trial data from subjects, including in the home or office setting.”
  • These provisions reflect a sustained emphasis on fostering clinical trial innovation, including building on the experience with remote clinical trials during the COVID-19 pandemic.

Section 304: Increasing Use of Real World Evidence (RWE) & Section 309: Post-Approval Study Requirements for Accelerated Approval

  • Section 304 would call for new guidance on the use of RWE in post-market review of drugs that were designated as a breakthrough therapy or fast track product, or considered for accelerated approval. Section 309 would further specify that the post-approval study requirements to verify and describe the clinical benefit for products granted accelerated approval could be satisfied through RWE, including analyses of data in clinical care repositories or patient registries.
  • Section 304 also would establish a permanent Real World Evidence Task Force to coordinate programs and activities within the Department of Health and Human Services related to the collection and use of RWE.
  • These and other sections of Cures 2.0 share a common theme of enhancing the use of RWE in regulatory decision-making. Although the inherent variability in RWE likely will continue to present challenges to doing so, the signal is clear that legislators would like to see FDA and HHS continue to move forward in this area.

Last week’s introduction of Cures 2.0 and President Biden’s announcement that he will nominate Robert Califf for FDA Commissioner contributed to a newsworthy week for those of us who follow the FDA.  We look forward to seeing how Cures 2.0 develops and how the Agency’s policy priorities unfold in the coming months.

FDA Issues Guiding Principles for Good Machine Learning Practice for Medical Device Development

On October 27, 2021, the U.S. Food and Drug Administration (FDA), Health Canada and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) issued a set of ten guiding principles meant to aid the development of Good Machine Learning Practice (GMLP).

Artificial intelligence and machine learning (AI/ML) offers the potential to analyze the vast amount of real-world data generated from health care every day to provide transformative insights. These insights can not only help improve individual product design and performance, but also hold the promise of transforming health care.

However, AI/ML technology has unique complexities and considerations. The goal of these guiding principles is to help promote safe, effective, and high-quality medical devices that use AI/ML to best cultivate the future of this rapidly progressing field.

Although not formal or binding, as companies continue to leverage AI/ML in their medical devices, they should remain mindful of each of the ten guiding principles:

  1. Leveraging Multi-Disciplinary Expertise Throughout the Total Product Life Cycle

Companies should leverage internal and external multi-disciplinary expertise to ensure they have a thorough understanding of the model’s integration into the clinical workflow, and the desired benefits and associated patient risks, to ensure the safety and effectiveness of the device while serving clinically meaningful needs throughout the product lifecycle.

  1. Implementing Good Software Engineering and Security Practices

Companies should implement as part of model design data quality assurance, data management, good software engineering practices, and robust cybersecurity practices.

  1. Utilizing Clinical Study Participants and Data Sets that Are Representative of the Intended Patient Population

Companies should ensure that their data collection protocols have sufficient representation of relevant characteristics of the intended patient population, use, and measurement inputs in an adequate sample size in their clinical study and training and test datasets so that results can reasonably be generalized to the population of interest.  Data collection protocols appropriate for the intended patient population may help to identify where the model may underperform and may mitigate bias.

  1. Keeping Training Sets and Test Sets Independent

Companies should consider and address all sources of dependence between the training and test datasets, including patient, data acquisition, and site factors to guarantee independence.

  1. Selecting Reference Datasets Based Upon Best Available Methods

Companies should use accepted, best available methods for developing a reference dataset, i.e., a reference standard, to ensure clinically relevant and well characterized data are collected (and that the reference’s limitations are understood).  Where available, companies should use accepted reference datasets in model development and testing that promote and demonstrate model robustness and generalizability across the target population.

  1. Tailoring Model Design to the Available Data and Reflecting the Intended Use of the Device

Companies should have a solid understanding of the clinical benefits and risks related to the product and utilize this understanding to create clinically meaningful performance goals.  Additionally, companies should ensure the model design is suited to the available data and supports active mitigation of the known risks.

  1. Focusing on the Performance of the Human-AI Team

Where the model has a human element, companies should consider human factors and human interpretability of the model outputs.

  1. Testing Demonstrates Device Performance during Clinically Relevant Conditions

Companies should develop statistically sound tests and execute them to assess device performance data independent of the training data set. Such assessment should be conducted in clinically relevant conditions with consideration given to the intended use population, important subgroups, clinical environment and use by the Human AI-Team, measurement inputs, and potential confounding factors.

  1. Providing Users Clear, Essential Information

Companies should provide users ready access to clear, contextually relevant information that is appropriate for the target audience. Such information includes not only information pertaining to the product’s intended use and indications for use, performance of the model for appropriate subgroups, characteristics of the data used to train and test the model, acceptable inputs, known limitations, user interface interpretation, and clinical workflow integration of the model, but also users should be made aware of device modifications, updates from real-world performance monitoring, the basis for decision-making (when available), and a way to communicate product concerns to the company.

  1. Monitoring Deployed Models for Performance and Managing Re-Training Risks

Companies should deploy models that are capable of being monitored in real-world usage with a focus on maintaining or improving safety and performance. Further, when models are trained after deployment, companies should ensure there are appropriate controls in place to manage risks that may impact the safety and performance of the model.

FDA’s expectations with respect to GMLP will continue to advance and become more granular as additional stakeholder input is considered.  The docket for FDA’s GMLP Guiding Principles, FDA-2019-N-1185, is open for public comment.

Real-World Evidence: Challenges and Opportunities During COVID-19

The urgent needs of the COVID-19 pandemic have more squarely brought into focus the role real-world evidence (RWE) can play in analyzing and informing product development and clinical and public health decisions. Specifically, the U.S. Food and Drug Administration (FDA) is participating in the COVID-19 Evidence Accelerator, in partnership with Friends of Cancer Research and the Reagan-Udall Foundation, to bring leading experts together to share insights and use RWE to help answer the most pressing research questions raised by the pandemic.

The FDA believes that RWE can play an informative role in analyzing potential therapies, vaccines, and diagnostics for COVID-19. At the recent “Establishing a High-Quality Real-World Data Ecosystem” workshop hosted by the Duke Margolis Center for Health Policy, Amy Abernethy, the Principal Deputy Commissioner of Food and Drugs and Acting Chief Information Officer at the FDA, highlighted the work of the Evidence Accelerator initiative, noting that RWE allows the FDA to constantly update its understanding of COVID-19 and recurrently analyze data to address changing needs. Amongst the other presenters, the general discussion focused on the many hurdles industry needs to address to establish a robust and more accurate RWE data ecosystem, including efficient capture of reliable data at the source. While internet access, smartphones, and wearable technology enable consumers and patients to keep meticulous records of their biometric data, the vast amount of collected data does not necessarily lead to efficient or fruitful analysis currently. FDA noted during the workshop that, to be more insightful, RWE stakeholders must narrowly tailor their collection to what is actually useful and relevant to clinical endpoints, fit for purpose, rather than merely what is easily accessible. Eric Perakslis, a Rubenstein Fellow at Duke University, noted that stakeholders must balance the usefulness of RWE collection against the risk of over-surveillance for each data point collected. While not discussed during the workshop, collecting massive data sets must also be weighed against the ever-present risk of data breach. Finally, speakers also discussed patient-generated health data (PGHD) and the need for aligned stakeholders who are motivated to collect this data and understand the process for doing so, including a plan for handling outlier data which is unavoidable with PGHD.

In the context of the COVID-19 pandemic, RWE presents an opportunity for real-time learnings toward quicker identification and development of treatments and vaccines. As a result, the pandemic has only strengthened the importance of RWE in product development and, if deployed well, could help support more efficient and expedited product development plans.

*Emily Tribulski, a 2020 summer associate in Goodwin’s Washington, D.C. office, contributed to this post.