Expanding our AI and Healthcare Portfolio

The healthcare industry is ground zero for AI companies and the rollout of their products: Microsoft tells us that AI is better than doctors at diagnosing complex medical conditions. Nvidia claims that its chatbot, a partnership with the startup Hippocratic AI, can outperform nurses on detecting over the counter drug toxicities.¹ AI firms suggest that their tools will improve workflow efficiency, increase diagnosis accuracy, and unlock “personalized treatment” or “precision” medicine. For some elite facilities, AI tools promise to enhance patient care through innovation. The Mount Sinai hospital system, for instance, has invested hundreds of millions of dollars in AI tools, turning their facilities into experimental labs. Amidst mounting financial pressures on the healthcare industry–from federal Medicaid/Medicare cuts, increased rates of uninsured patients, reduced access to primary care, and an increasingly corporate ownership structure—AI companies present their tools as a cost-effective and common-sense solution for under-resourced facilities. 

But the reality of AI in the healthcare industry is far different: 

• For patient safety and confidentiality, the dangers loom large. Chatbots can fail to note whether a patient has drug allergies, a potentially life-threatening oversight. Ambient scribes can invent conditions and describe conversations that did not occur. 
• For healthcare workers, the risks are profound. While AI technologies are often adopted under the guise of “transforming” and “democratizing” healthcare, they are used to justify job displacement and undermine healthcare professionals’ judgment. The adoption of AI tools may lead to reduced staffing supports, diminished care for patients, stressed facility budgets, and untold reliance on corporate actors. The rollout of new so-called budget-saving AI tools like Abridge, Nabla, and Open Evidence may ironically increase operating costs because of labor-intensive implementation expenses and ongoing subscription requirements, among other factors. 
• Institutions are not immune to AI-related harm. AI tools may be drawing healthcare companies into greater debt, adding more risk-intensive working conditions, sabotaging patient care, and consolidating corporate power. Even though healthcare is the largest employment sector in the US and one of the only growing ones, new human management technologies may be turning the workplace into an auction where decision-making is obscured and trust is relegated to computational software. 
• AI firms may be accelerating the healthcare industry’s move away from democratic institutions, good governance practices, and public oversight. AI tools are centralizing and obscuring decision making over health and care. AI firms are building an opaque health infrastructure in which the data used to supposedly democratize information is held by a few players while access to the decision-making processes is largely impenetrable for the majority of us.

Healthcare is one of the most high-stakes sectors for AI deployment. And it has become a central site for legitimizing AI’s broader social value, with industry leaders frequently positioning healthcare AI as an unambiguous public good. Yet there remains limited independent scrutiny and evaluation of how AI tools are actually integrated into healthcare settings, how risks are distributed, and who bears the costs, despite significant public investment and rapid commercialization. This gap is the starting point for our current AI and Healthcare work. 

* * *

The AI Now Institute has tracked the healthcare industry’s rapid adoption of large-scale AI tools over the past decade. Now, we are building an independent, participatory, and action-oriented research portfolio to interrogate the deployment of AI across healthcare systems. This work is focused on the consequences of this deployment for patient safety, worker dignity, public spending, and democratic accountability. Specifically, we are assessing (a) the experiences of nurses on-the-ground; (b) the shifting regulatory landscape; and (c) the political economy of the healthcare industry, including the business practices of companies selling and deploying AI technologies. 

Healthcare workers are on the frontlines of Silicon Valley’s new AI tool deployment in medicine, be it in long-term care facilities where AI-sensor machines “watch” to see if a patient has fallen out of bed or in major hospital systems where nurses use automated text-to-speech tools like AI Scribe to take notes on patient conditions. Healthcare workers are also some of the first to see how AI firms experiment differently in different settings, be it acute (hospitals) or post-acute (long-term care and rehab facilities), private equity-owned or nonprofit, rural or urban, and unionized or non-unionized. Just as MRI machines and Da Vinci surgical robots are not accessible evenly across the US, neither are AI tools evenly deployed across healthcare infrastructure. Through collaborative research with healthcare workers, we are examining the promises, mechanics, limits, and geographies of new automated-decision-making technologies.

The regulatory landscape for healthcare and AI remains insufficient, and the appropriate standards for validation and testing of AI systems are underspecified. Healthcare AI bills tend to not focus on the critically important labor issues, and labor bills tackling AI tend to not focus on the particular nuances of the healthcare industry. For a time, the nonprofit Coalition for Health AI (CHAI) tried to fill some of these oversight gaps through a self-regulatory “AI Assurance” approach that would provide evaluation frameworks co-developed by tech firms and hospital systems. However, CHAI failed to develop independent benchmarks and, after three years, pivoted in its mission. The coalition’s priorities are now focused on “disclosure of conflicts of interest,” “protecting intellectual property,” and matching other priorities set by corporate stakeholders. Patient care, worker safety, and data privacy remain a wild west in the age of AI-meets-healthcare. 

Any assessment of AI’s incursion into the healthcare industry must be paired with legal analysis of how AI firms and their boosters engage with state laws, government contracts, and regulatory institutions. Some AI firms seek to rewrite existing rules while others exploit lucrative gaps in healthcare regulation. Many AI tools, for instance, do not yet trigger FDA scrutiny or clinical trial requirements because they are not recognized as physical, medical devices. Other AI products like ambient scribes, algorithms designed for cost-saving on sepsis, and acuity scoring systems are not currently subject to governmental review even though they may significantly reshape healthcare workflows and impact medical outcomes for patients. Non-compliance with HIPAA is also common with AI tools that handle private conversations and sensitive clinical notes. To make these regulatory gaps permanent, the Healthcare Leadership Council—which represents Johnson & Johnson, Labcorp, Amazon, Mayo Clinic, McKesson, Merck, Mount Sinai Health System, AstraZeneca, UnitedHealth Group, Blackstone, BlueCross BlueShield, and more—is lobbying for even less regulatory oversight; this group is advancing a temporary ban on any state-level legislation about AI uses in healthcare. 

AI’s entry into the healthcare industry—and the threats it poses—must be understood alongside the industry’s new political economy. Over the last fifty years, financialization, capital-intensive procedures, and the rise of business administrators in medicine have built an unprecedented, for-profit, health-care empire. Nearly twenty-five percent of community hospitals are investor-owned while more than 700 rural hospitals are at risk of closure. The dangers of this new model of medicine have worsened with the entry of private equity firms, which today control “more than four hundred hospices, nearly 10 percent of gastroenterologists, 8 percent of dermatologists, and more than 10 percent of oncologists.” Such facilities get saddled with debt, lower wages, unsafe workplace conditions, and declining qualities of patient care. Initial research finds that acquisitions by private equity cause “a 24 percent fall in hospitals’ assets and a 25 percent rise in patients’ hospital-acquired complications, such as infections and falls.”Across the country, a consensus around the need for a regulatory response to Wall Street’s predatory interest in healthcare has been building. “Americans may not agree about much,” Siddhartha Mukherjeewrites, “but it’s clear they are angry about the degree to which corporations constrain our choices about our health and our bodies. (Look, for instance, at the gleeful response to the coldblooded murder of a health insurance executive.)”

The future of healthcare may depend on a rejection of AI firms’ vision of care: who does it, how it is done, and under what market conditions. The conditions under which AI tools are being sold to the healthcare industry are born out of, and advance, deep power asymmetries. To examine AI tools in healthcare necessitates confronting these power imbalances and the kinds of politics they forgo. 

* * *

Our work addresses the potentially skewed incentives of the market-based AI solutions sold in the healthcare industry, outlining an evidentiary base and mobilization for action that ensures any deployment of AI serves the needs and interests of patients, the public, and healthcare workers, not the profit incentives of corporate medical companies and tech firms. Our work is organized around four specific goals:

1. Document how AI systems are deployed in healthcare settings, and how they reshape clinical judgment, labor conditions, and patient privacy, especially for low-income populations, immigrants, and communities of color.

2. Interrogate the political economy of the healthcare AI sector, including vendor ecosystems, financing structures, public procurement pathways, and the role of large tech firms and private equity holdings.

3. Center healthcare workers and patients as co-producers of knowledge and policy expertise by using participatory methods.

4. Translate findings into concrete policy interventions that can be adapted for state, local, and national contexts.

We are employing participatory methods that offer a clear, ground-up picture of AI use from below. Throughout the research, we are foregrounding collaboration with groups that are operating on the ground, such as healthcare unions. This form of data collection and agenda-setting are a new approach for AI Now, but one that aligns deeply with the ethos of our work.

Our first publication from this new portfolio, Uber for Nursing Part II, focuses on the use of AI in healthcare scheduling and staffing, specifically the rise of gig nursing platforms. Uber’s business model—the “gigification” of labor—and lobbying practices have made their way to healthcare staffing. Backed by huge sums of venture capital and private equity funds, gig nursing platforms encourage nurses to bid against each other for wages. The platforms’ algorithmic management systems—with their reliance on dynamic pricing, surveillance wages, and automated performance metrics—are transforming how workplaces function. The platforms are also encouraging state legislators and policymakers to carve these platforms out of existing regulations that govern healthcare staffing. Since 2022, efforts that would deregulate gig nursing platforms have emerged in nearly half of all US states, a campaign that could upend decades of laws and norms in the healthcare industry that guarantee public oversight, ensure worker protections, and safeguard patient care. 

We look forward to sharing more work in the months to come on Epic System’s AI charting tools, HCA Healthcare’s scheduling software, and the Oxevision patient monitoring system, among other technologies. To share information or leads that might be useful to this workstream, please reach out to Katie Wells, Senior Fellow, AI and Healthcare, at katie.wells@ainowinstitute.org.

1. For example, the company Hippocratic AI is an example of a VC-backed firm that is riding on the premise that AI can effectively replace nurses. They were prominently featured in a recent Congressional hearing on instituting a moratorium on state regulation of AI, and had numerous representatives attending the meeting—evidencing the scale of lobbying interest in this space. ↩︎