Key Questions on the Role of Technology in the Expanding Middle East War

Heidy Khlaaf

Chief AI Scientist, AI Now Institute

We are in the dangerous territory where Large Language Models (LLMs) and generative AI are being normalized as a valid technology to be instrumented within both AI-Decision Support Systems (DSS) and Lethal Autonomous Weapons Systems (LAWS) for targeting purposes. Yet current framing regarding Anthropic’s and OpenAI’s negotiations with the US’s Department of War instead risks overindexing on myopic interpretations of human oversight, or a particular companies’ so-called ‘red lines’, papering over what should be the real target of our scrutiny: that generative AI algorithms are a flawed and inaccurate technology that fabricate and “hallucinate” outputs, often at a rate of 50% accuracy, where they’re unlikely to be able to solve tasks outside of their data distribution and training data sets.

Generative AI’s inability to handle novel scenarios that would arise from the fog of war thus raises serious questions about whether they can be successful in military settings. Furthermore, these “hallucinations” are an inherent property of these models given their probabilistic nature, with model providers stating that these issues are to persist.

Letting Anthropic’s safety theater drive the discourse additionally overlooks what is ultimately a superficial distinction between AI-DSS and LAWS in practice. The difference often separating AI-DSS, which is how they’re currently being used, and AI-driven AWS, is a human in the loop that is likely to be impacted by automation bias. If Anthropic itself admits that its models are too unreliable for LAWS, that would imply they are not reliable for AI-DSS either, nor the autonomous drone swarming technologies they’ve put proposals forward for.

Regardless of oversight levels, current LLM safety is far from the reliability and accuracy measures that have long been a prerequisite for defense and safety-critical systems. Deploying them for either AI-DSS or AWS may ultimately lead to conflicts becoming indiscriminate lethal campaigns.