Appendix 2: How Does the FDA Work?

1. Premarket assessment

Marketing of drugs to the public is a powerful gateway for the pharmaceutical industry: it is the primary path through which drugs enter into widespread use, and convincing doctors that a drug is the right therapy to prescribe has a significant effect on the drug’s success or failure in the market. Similarly, convincing insurance companies to pay for the use of that drug offers a baseline guarantee of profitability that pharmaceutical manufacturers are strongly incentivized to seek. Finally, the level of scrutiny that drugs have traditionally undergone in the United States offers an assurance of safety and efficacy that is beneficial to the pharmaceutical industry, with wide and sweeping effects across international markets.1“International & Interagency Coordination,” Food and Drug Administration, updated September 18, 2018; accessed September 21, 2023, https://www.fda.gov/food/international-interagency-coordination.

In the pharmaceutical context, the FDA uses a benchmarking process that is both flexible and standardized in the form of end points: evaluative metrics tailored to each drug application that are both valid and generalizable, and reflect a particular outcome being measured. End points are established in agreement between FDA staff and drug manufacturers, and form a key part of the clinical trial process as determinants of whether the drug has successfully achieved a stated health outcome.7 Charlie McLeod et al., “Choosing Primary Endpoints for Clinical Trials of Health Care Interventions.” Surrogate end points serve as proxies, metrics that are closely linked to more traditional end points but may enable swifter evaluation by substituting a short-term outcome for a long-term one (for example, one workshop participant referenced reduction in tumor size as an example of a surrogate end point that is clinically verifiable on a shorter timeframe than seeing a patient’s cancer go into remission).8 Ibid.

This approach to premarket approval allows for a tremendous amount of flexibility.9 Remark made by a workshop participant. While standards and guidance shape every stage of the process, there is room for context- and drug-specific flexibility in how end points are chosen. The statutory language that guides premarket review is instructive:

§ 314.2 Purpose (FD&C Act). The purpose of this part is to establish an efficient and thorough drug review process in order to: (a) Facilitate the approval of drugs shown to be safe and effective; and (b) ensure the disapproval of drugs not shown to be safe and effective. These regulations are also intended to establish an effective system for FDA’s surveillance of marketed drugs. These regulations shall be construed in light of these objectives.

This flexibility can be important, for example in the evaluation of drugs for rare diseases, where the FDA can approve a drug with a small clinical trial sample size. FDA staff members weigh both the challenges of obtaining a large patient sample—and the downstream effects of this, such as the likelihood of delaying needed interventions for patient populations—against the need to clearly establish evidence on the safety and efficacy of a new drug, and make decisions about when and under what conditions to relax the standard.10Congressional Research Service, “How FDA Approves Drugs and Regulates Their Safety and Effectiveness,” updated May 18, 2018, https://crsreports.congress.gov/product/pdf/R/R41983.

Another source of flexibility has been the introduction in recent decades (starting in 1987) of accelerated approval pathways.11Thomas J. Hwang et al., “Association between FDA and EMA Expedited Approval Programs and Therapeutic Value of New Medicines: Retrospective Cohort Study,” BMJ 2020;371:m3434, https://doi.org/10.1136/bmj.m3434. These pathways—of which there are currently four—allow the approval of a new drug to be expedited in various ways, such as through the use of an intermediate clinical end point or a commitment to a shorter FDA review period. These processes are intended to be reserved only for particularly innovative drugs that are likely to provide significant improvement over existing therapies, but in practice the majority of new drugs (60 percent in 2019) utilize one or more of these pathways. Despite this widespread use, there is limited evidence that accelerated approval pathways deliver significant additional therapeutic value to patients, and some concern that they are associated with increased risks to patient safety.12Cassie Frank et al., “Era of Faster FDA Drug Approval Has Also Seen Increased Black-Box Warnings and Market Withdrawals, Health Affairs 33, no. 8 (August 2014): 1453–9, https://doi.org/10.1377/hlthaff.2014.0122.

In some places, premarket assessment is already part of the digital market, though this occurs in a piecemeal fashion: 

1. App stores. The Apple App store and Google Play store together make up a perfect duopoly and represent gatekeepers for all applications (regardless of the level of “AI”) that are accessed by consumers via a smartphone. Apple and Google conduct premarket approval at their own behest, and set standards (SDKs) that app developers must hit. Their processes are opaque and run by organizations that are not democratically elected.

2. Government procurement processes. When the government is the customer purchasing an AI system, there are increasingly standardized requirements that any product or service must comply with. (See Appendix 1 for examples of existing and proposed procurement standards for pre-market scrutiny.)

3. Software as a Medical Device (SaMD). The FDA already oversees the release of medical devices, including some AI software. Such systems are classed according to a risk categorization and undergo differing levels of evaluation depending on the level of risk prior to deployment.13 Stein and Dunlop, Safe before Sale.

2. Post-market monitoring and enforcement

The FDA also has processes to track drug and device manufacturers. It issues unique identification numbers to manufacturer facilities (both domestic and international), allowing the agency to track where drugs and devices are being manufactured and to conduct facility audits.14Office of Regulatory Affairs, “FDA’s Risk-Based Approach to Inspections,” Food and Drug Administration, January 17, 2024, https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/inspection-basics/fdas-risk-based-approach-inspections.

If a product that has been approved for commercial use is determined to have caused harm to patients, the FDA has a variety of measures available to use15This section is a summary of key sections of the CRS report. For more details, see Congressional Research Service, “Enforcement of the Food, Drug, and Cosmetic Act: Select Legal Issues.”:

1. Advisory action letters. Although not required by law, the FDA often sends letters to individuals or companies to encourage voluntary action before enforcement. There are two types of letters: warning letters alert entities that the agency has identified “violations of regulatory significance” and request mitigating action, warning that legal action may be imminent. Untitled letters are softer and are used to address violations that do not merit a warning letter, such as missing risk information on promotional materials.16Ibid. The FDA makes advisory letters publicly available.17“Advisory Letters,” Food and Drug Administration, updated July 5, 2024, https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/compliance-actions-and-activities/advisory-letters.
2. Recalls. The FDA can recall a product that FDA considers to be in violation of the law, and does so frequently (as many as a thousand prescription drugs are recalled every year).18“Recalls,” dashboard, Food and Drug Administration, accessed May 10, 2024, https://datadashboard.fda.gov/ora/cd/recalls.htm. Recalls can be voluntary or mandatory. Most are voluntary at the request of the FDA or initiated by the manufacturers.19Robert H. Shmerling, “Drug Recalls Are Common,” Harvard Health Publishing, Harvard Medical School (blog), March 29, 2023, https://www.health.harvard.edu/blog/drug-recalls-are-common-202303292907. Manufacturers must report when they have initiated a recall, which may trigger FDA oversight. A company can ignore a recall request from the FDA, but that may risk enforcement action. In the cases listed below, the FDA can issue a mandatory recall. The process for mandatory recalls vary by product, but often begin by issuing an administrative order, which gives the product manufacturer/owner an opportunity to defend the product’s safety at an informal hearing before a presiding officer.20Congressional Research Service, “Enforcement of the Food, Drug, and Cosmetic Act: Select Legal Issues.” 
3. Debarment. The FDA has the authority to prohibit specific individuals or corporations from engaging in FDA-regulated activities (essentially ending their career) based on illegal conduct (e.g., a clinical investigator who falsifies records). Debarment can be permanent or for a set period of time. The FDA maintains a public list of debarred entities.21“FDA Debarment List (Drug Product Applications),” Food and Drug Administration, updated June 13, 2024, https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/compliance-actions-and-activities/fda-debarment-list-drug-product-applications.
4. Import alerts. The FDA has the authority to issue import alerts or the ability to “refuse admission” to the US market if products “appear, from sample or otherwise” (“otherwise” may include a history of violations or a failed facility inspection) to violate the FD&C act.22See Office of the Commissioner, “Federal Food, Drug, and Cosmetic Act (FD&C Act),” Section 801(a). The product is put on an import alert list to notify border officials that products should be automatically detained. Once products are detained, the owner can testify that the products are safe / abide by FDA regulations and the FDA can follow up with a determination to permit or refuse entry.23Congressional Research Service, “Enforcement of the Food, Drug, and Cosmetic Act: Select Legal Issues.”

3. Producing information and expertise

The FDA uses a range of mechanisms to elicit the generation of information about pharmaceuticals. These include:

1. Approval processes. The FDA collects thousands of details through the drug application process, and conducts its own independent review of that information. While no commercially confidential information is published, the FDA’s validation of claims made by companies is open for public scrutiny and review.24“Drugs@FDA: FDA-Approved Drugs,” Food and Drug Administration, accessed May 10, 2024, https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm.
2. Clinical trials. The FDA requires drug companies and other regulated entities to report data to the ClinicalTrials.gov database maintained by the National Institutes of Health, and engages in monitoring and enforcement to ensure compliance,25ClinicalTrials.Gov, accessed May 10, 2024, https://clinicaltrials.gov. though this is inconsistent.26Morten, Nicholas, and Viljoen, “Researcher Access to Social Media Data.”
3. User disclosures / labeling. The FDA requires products in its jurisdiction to have labels, and sets rules for what these labels must look like.27Congressional Research Service, “How FDA Approves Drugs and Regulates Their Safety and Effectiveness,” updated May 18, 2018, https://crsreports.congress.gov/product/pdf/R/R41983.
4. Incident reporting. The FDA requires drug manufacturers to report serious and unexpected adverse reactions to the FDA Adverse Event Reporting System within 15 days, after which the reports are reviewed by the Office of Surveillance and Epidemiology. The Office also reviews and conducts its own studies, reviews errors in similar drugs, and follows international regulatory bodies.28“Questions and Answers on FDA’s Adverse Event Reporting System (FAERS),” Food and Drug Administration, updated June 4, 2018, https://www.fda.gov/drugs/surveillance/questions-and-answers-fdas-adverse-event-reporting-system-faers. See also “CDER Office of Surveillance and Epidemiology,” Food and Drug Administration, updated January 29, 2024, https://www.fda.gov/about-fda/cder-offices-and-divisions/cder-office-surveillance-and-epidemiology.
5. Post-market studies. The FDA may require post-market studies if it becomes aware of risks, and it can initiate enforcement action against companies if they fail to adhere to the timetable when notified by the FDA of the necessity of participating in a post-market study or clinical trial.
   1. A 2015 Government Accountability Office (GAO) report highlighted that these post-market studies can be challenging because of the lack of incentives for clinicians and patients to participate, and that they create an additional reporting burden for medical practitioners.29 “Medical Devices: FDA Ordered Postmarket Studies to Better Understand Safety Issues, and Many Studies Are Ongoing,” U.S. Government Accountability Office, September 30, 2015, https://www.gao.gov/products/gao-15-815.

A variety of stakeholders can engage in the FDA drug approval process in multiple ways:

1. Advisory Committees. The FDA engages advisory committees to comment on the quality of data for submitting a drug/device approval and to recommend additional studies or label changes. Advisory committee advice is nonbinding, and final decisions rest with the agency. Legally, advisory committee membership must be “fairly balanced” both in demographic diversity and experience/expertise. Most of FDA’s drug advisory committees consist of physician-scientists who specialize in the drug. Industry representatives act as individuals speaking for concerns of the industry globally, not as representatives of their employer. There are also members from consumer advocacy groups.30“Advisory Committees: Critical to the FDA’s Product Review Process,” Food and Drug Administration, updated May 4, 2016, https://www.fda.gov/drugs/information-consumers-and-patients-drugs/advisory-committees-critical-fdas-product-review-process.
2. Advisory committee meetings (10 to 15 members) are public and open to the press. They are typically held twice a year and last two days, and expenses are covered by the FDA. Members arrive having read preparatory background materials, such as summaries regarding the safety and effectiveness of the new product.31Ibid.
3. Patient Representatives. “Patient Representatives” represent the closest parallel to the AI governance equivalent of involving users / data subjects in AI risk assessments. Patients and advocates are considered temporary employees who provide direct input to agency staff and engage with experts on the FDA advisory committees. Candidates are recruited and trained in how to engage in FDA activities / contribute to decisions. Applicants must have personal experience with the disease they are representing, “ability to be objective,” willingness to share their views, knowledge of treatment options, and no conflicts of interest.32“About the FDA Patient Representative Program,” updated April 23, 2024, https://www.fda.gov/patients/learn-about-fda-patient-engagement/about-fda-patient-representative-program.