Militaries worldwide are engaged in the unfettered development, and even deployment, of AI systems despite serious public concerns about the moral and legal dimensions of the use of AI in warfare.

Despite the lack of evidence that AI-enabled military systems comply with either international human rights law or rudimentary safety engineering principles, the AI safety conversation remains silent on arguably the most high-stake risks. Indeed, the deadly and geopolitically consequential impacts of AI brings with it existential risks that are very real and present. If current AI armaments serve as the foundation of prospective autonomous weapons systems, then as they stand, they must be assured to address current risks in preparation for further ‘emergent’ behavior (a term we use here in its originating context to describe unanticipated outcomes due to complexity, rather than sentience). 

Whether these systems are “fit for use” – that is, whether a system adequately holds up to a defined set of measures and thresholds – given their known risks and failure modes remains an open question. AI systems such as Lavender and Gospel provide the latest known example of use of such systems, which have facilitated the massacres of civilians without accountability and oversight. Deployment and evaluations of these AI-based systems have not been made available to the public, with details only privy to AI providers themselves, indicating a significant conflict of interest.

There is no higher stake use for AI than its deployment in warfare to make life or death determinations. With Israeli startups looking to export AI armaments worldwide with the claim that they are “battle-tested” with improved precision, all signs point to Gaza serving as a test kitchen for a much more widespread usage of such systems. This work aims to remedy what is an alarming gap in the field, and to bring AI armaments to the center of the AI safety discussion. 

In this introductory post, we outline the core concerns animating our future work on AI-enabled military systems: in particular, rigorously evaluating whether, and under what conditions, such systems could ever be safe and secure. Future posts will expand on particular aims, including providing a conceptual grounding in safety assurance, and addressing policy arenas where these issues are being addressed including through export controls. 

Safety failures are a present, not future, AI risk.

Establishing the risk and dependability of AI, whether within armed conflict or during peacetime, has led to several challenges in the determination of how and if AI technologies can and need to be trusted in the context of accelerated military deployment. Numerous organizations have issued warnings regarding the risk of deploying AI within military applications. Noted risks range from the dehumanization of national, ethnical, racial or religious groups to a loss of human control, all which may allow state actors to circumvent individual and state-wide accountability. Unfortunately, these warnings have already come to fruition with recent unveilings of AI systems such as Lavender and Gospel, that have been deployed in Gaza. These AI systems automate the selection of targets en masse using a prediction based algorithm that has led to significant civilian death toll with allegedly no oversight or validation regarding their use, reliability or safety. 

A clear chasm remains between existing narratives about the capabilities of these systems and the reality:

  1. Unsubstantiated claims regarding the current capabilities of AI systems within military applications have gone unchallenged, largely due to the lack of expertise by courts, legislatures, and rulemaking bodies, leading to a failure to address the lack of safety and reliability of military-based AI systems (such as Gospel and Lavender), and their relation to international human rights laws. 
  2. Evaluation criteria for AI models have been scoped too narrowly or have little to no relevance to safety-critical applications, despite their use to support system specific claims that they cannot substantiate. The absence of legal standards and enforceable risk thresholds has led to an implicit acceptance of arbitrary thresholds as determined by those with self-interest in the development and deployment of AI-based systems, without independent scrutiny. The combination of corporate secrecy shielding AI developers, and military classification shielding deployed use, exacerbates this problem.
  3. Existing and unknown risks and failure modes have been dismissed through appeals to “potential”, yet unproven, capabilities AI systems are projected to develop within the upcoming years. Such potential capabilities have been used to justify the deployment of risky systems currently. Even as the lack of understanding of the uncharted vulnerabilities and cybersecurity threats within the AI supply-chain can lead to novel attack surfaces that allow unauthorized access and changes in data, model parameters, and intellectual property. These vulnerabilities pose defense and national security risks due to the significant increase of attack surfaces that AI-based systems present, which can inadvertently or maliciously accelerate these systems towards a state of imprecise and unlawful targeting and a loss of human control.

These challenges largely stem from a surfeit of hype, a lack of clear definitions, and a concomitant absence of clearly specified claims about the precise intended application of AI systems. Global governance efforts, like those at the United Nations (UN), lack specificity as risk thresholds have not been agreed on by civil society, legislatures, or other rulemaking and enforcement bodies, going against long established norms. 

In safety-critical and defense domains, claim-oriented or goal-based approaches to assessing the fitness of technical systems have always been necessary tools in supporting socio-technical, legal, or regulatory claims. Put bluntly, these systems cannot be effectively evaluated in the abstract, assessments must address specific use cases and contexts of deployment. Indeed, translating the high level goals of global governance efforts into concrete and methodical measures that can better ensure accountability and controls of AI systems requires the use of safety engineering techniques to assess the reliability and accuracy of these AI systems in contexts of arms safety and licensing, and export and import controls.

Furthermore, safety methodologies lay bare the limitations and challenges of commonly proposed controls for AI armaments within the existing AI supply chain, particularly in light of increasing corporate opacity regarding the development processes and underlying data used to train base models. Significant difficulties are introduced in the attempt to operationalize standard controls given a lack of separation between commercial models and those deployed within military applications. 

Paths Forward

Given this state of affairs, there are real and significant challenges we need to address with regard to the use of AI systems in military contexts. In future work and posts, we aim to expand on the following:

  1. Can AI systems be validated with the level of assurance necessary for safety-critical contexts against articulated claims of the intent of use?
  2. In the absence of specific use cases and defined risk thresholds, can AI systems be adequately verified by independent assessments, or does the current state of corporate secrecy and inherent opacity of large-scale AI contravene against such widely accepted scientific practices?
  3. Is it possible to shore up AI systems against vulnerabilities given the significant and novel attack surfaces they introduce that may impair national security?
  4. Does the recent proliferation of foundation or general purpose models inhibit export and import controls from being effectively implemented to protect military sensitive data?

Indications suggest that for many of these questions, there currently exists no evidence to substantiate such claims – providing strong evidence in support of a moratorium on military uses of AI until and unless sufficient assurances can be established through rigorous evidence-gathering.