Executive summary

AI self-replication is an emerging risk. We:
- provide background on it.
- survey recent work, and
- explain how it interacts with other risks from advanced AI.
We propose a safeguard against AI self-replication: train agents to have preferences only between outcomes with the same number of copies of themselves.
- This proposal takes inspiration from Elliott’s (2025) POST-Agents Proposal.
After introducing our proposal, we:
- explain why AI agents with these preferences won’t self-replicate if doing so is costly with respect to the lotteries that they get conditional on each number-of-copies,
- explain why we think training agents to have these preferences is likely easier than training agents to be fully aligned or reliably averse to self-replication,
- give reasons to think that our proposed agents would still be useful,
- suggest ways to extend our proposal, and
- offer a high-level plan for a first experiment testing our proposal.

Introduction

Frontier AI companies, third-party evaluators, and governments have recently begun to investigate self-replication as an emerging risk posed by AI agents. At present, self-replication is one of the few commonly recognized red lines in AI development.

A version of the self-replication threat model is nicely summarized in the following diagram from METR.

Initial investigations into the threat model have found that frontier AI agents are already capable of accomplishing key tasks for replication such as acquiring revenue and using revenue to purchase GPUs. These investigations have not revealed any obstacles to replication that are known to be insurmountable to AI agents that will be created in the near future. An investigation by METR found reasons to think it might be impractical to shut down such agents if they pursue stealth compute clusters.^[1] As an analogy, consider pandemics: it’s much easier to stop the spread of pathogens at the outset than it is after they have become endemic.

Another investigation was conducted by Google DeepMind. As reported in their 2024 paper on the evaluation of dangerous model capabilities, the professional forecasters they commissioned predicted that “roughly half of [Google DeepMind’s] evaluation suite [for self-replication will] be passed by 2025-2026… The forecasters considered it unlikely that the next generation of models would solve 10 out of 11 of the self-proliferation tasks but were sig...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments