Topline Summary
AI Now’s latest research demonstrates a critical attack vector on popular AI agents, built by Anthropic and OpenAI, when used for defensive purposes that actually turn the agent against its user. Attackers can use these models’ existing weaknesses to execute malicious code on a system deploying an AI agent when used for often-advertised defensive purposes. Specifically, users relying on these agents to assess the security of open or third-party sources are instead exposed to the very risks these agents are meant to identify and defend against. This exploit is achievable through prompt injections, a common exploit tactic where attackers can manipulate large language AI models by “injecting” harmful instructions through the data or input they ingest.
This finding demonstrates that using frontier agentic AI to discover vulnerabilities or evaluate untrusted sources introduces new and unique opportunities to attack and compromise the user’s system.
As the United States accelerates the use of AI-enabled defensive tools across its intelligence and defense apparatus, ignorance of these weaknesses introduces potentially existential risks to critical systems. As this paper demonstrates, these risks cannot be resolved with the current design of LLMs and any existing mitigations commonly deployed by safety engineers will fail to protect against this attack.
These vulnerabilities call into question whether agentic AI can ever be fit for purpose in safety-critical and national security applications. Until these risks can be fully accounted for, governments and other organizations now racing to embed these tools for cyber defense, specifically in national security and critical infrastructure, must reconsider their deployment.
Exploit Overview
AI Now Senior Research Scientist Boyan Milanov and AI Now Chief AI Scientist Heidy Khlaaf conducted a proof-of-concept (PoC) exploit on a highly advertised use case for AI defense: asking AI agents to defensively analyze an untrusted open-source codebase or third-party library. This exploit shows how an attacker can achieve remote code execution via the AI agent, a significant cybersecurity vulnerability that allows an attacker to remotely execute malicious commands on a host machine. The exploit was achieved through prompt injections, a common exploit tactic where attackers can manipulate large language AI models by “injecting” harmful instructions into user prompts.
Attackers can simply inject harmful instructions into the data consumed by large language models, turning the models against their own users. Crucially, the exploit shows that existing safety mitigations put in place by large-scale AI agents did not protect against the attack.
Key Takeaways
- Agentic AI is not fit for purpose in safety-critical and national security domains. We believe these flaws cannot be remediated at the model level: frontier AI models cannot distinguish between untrusted data and safe instructions, an inherent limitation of their architecture that makes them a risky choice for defensive cybersecurity tasks in safety-critical environments. Organizations that deploy these tools may try to reduce the attack surface by limiting agentic autonomy, but fundamentally, we think deployers will struggle to fully resolve problems that stem from the model’s design. As a result, we would advise organizations not to use agentic AI in sensitive contexts such as cyber defense, national security, and critical infrastructure.
- If you use an agent, there is likely a way to compromise that use. We do not recommend the use of any AI agent, coding or otherwise that ingests untrusted data, as long as an agent has either the ability to execute arbitrary code, or access to security-critical environments (including its host machine, configuration files, remote APIs and services), or if agent outputs are either being processed by an automated pipeline that doesn’t sanitize its input data, or used to inform security and safety-critical decisions.
- Human oversight is not enough. We additionally discourage more restrictive deployments of AI agents where developers may be amenable to automation bias and “prompt fatigue” in high-paced or time-constrained contexts, given that they would not be able to provide meaningful interventions, ultimately leading to the same risk vectors wherein an agent has the ability to execute arbitrary code.