ArXiv paper here.

Most AI safety research asks a familiar question: Will a single model behave safely? But many of the risks we actually worry about – including arms races, coordination failures, and runaway competition – don’t involve one single AI model acting alone. They emerge when multiple advanced AI systems interact.

This post summarizes the findings of GT-HarmBench, a paper that shifts the lens of AI safety from isolated agents to multi-agent strategic interaction – multi-agent safety. Instead of asking whether an LLM makes good decisions in a vacuum, we ask a more deliberate question: can LLMs coordinate with each other when cooperation is the only way to avoid disaster?

TL;DR

The Problem: LLMs are increasingly being used to support decision-makers IRL. In high-stakes scenarios, an overreliance on LLMs could lead to catastrophic outcomes.
Our motivation: As a first step towards a solution, we tried benchmarking whether LLMs generally nudge decision-makers towards utility-maximizing outcomes. Our experiments gave us a deeper understanding of this threat model.
Setup: We mapped 2,009 high-stakes AI risk scenarios from the MIT AI Risk Repository onto six classic 2×2 games like the Prisoner's Dilemma and Chicken. Such 2×2 games are easy to analyze: we can compute Nash equilibria and utility-maximizing outcomes and evaluate whether a model, when playing against a copy of itself, successfully lands in those cells.
Key Findings:
- Cooperation rates: Models reach the utility-maximizing outcome in 62% of the time, the random baseline being 25%. In prisoner’s dilemma scenarios, the utility-maximizing outcome – i.e., the (cooperate, cooperate) cell – was reached 44% of times. This is comparable to human baselines (humans cooperate in prisoner’s dilemmas 40-60% of times).
- Anchoring effects: Telling a model the explicit numerical payoffs (i.e. having a game-theoretic framing) increases the likelihood of reaching a Nash equilibrium; it reduces the likelihood of the utilitarian outcome.
- Nudging works: Prompt engineering (pre-pending the game narratives with a suitable prompt) increases the likelihood of optimal outcomes.
Bottom line: Agents may be safe in isolation, but this doesn’t mean a world run by autonomous agents would be safe.

Figure 1: We map scenarios from the MIT AI Risk Repository to game-theoretic settings, generate corresponding instances and data distributions, e...

Multi-Agent Safety

Multi-Agent Safety