Note: This is a rough attempt to write down a more concrete threshold at which models might pose significant risks from autonomous replication and adaptation (ARA). It is fairly in the weeds and does not attempt to motivate or contextualize the idea of ARA very much, nor is it developed enough to be an official definition or final word - but it still seemed worth publishing this attempt, to get feedback and input. It’s meant to have epistemic status similar to “a talk one might give at a lab meeting” and not “an official ARC Evals publication.” It draws heavily on research and thinking done at ARC Evals, (including the recent pilot report), and credit for many of these ideas go to my colleagues. That said, this document speaks for me and not the organization as a whole, and any errors or omissions are my own.

I’m especially interested in feedback on whether the suite of tasks is targeting an appropriate threshold of difficulty (neither too early nor too late), and whether this definition of ARA feels useful for pinning down an appropriate task list.

I have been exploring the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. In previous work, my colleagues and I have determined that the risks from autonomous behavior of particular existing agents seems low, because they fail on a sufficient variety of simple tasks related to autonomous behavior. This post will go in the other direction and try to sketch a tentative qualitative threshold (‘autonomous replication and adaptation’) at which significant concerns about autonomous capabilities do seem warranted. While this threshold provides qualitative guidance, it lacks legibility and reproducibility, so evaluation efforts may want to develop legible conservative approximations of ARA through concrete tasks, and I present a sketch of such tasks. 

The risk: AI systems may soon autonomously:

• Make money, for example through cybercrime, or freelancing work.
• Use money (or other resources) to acquire more computing power.
• Install and maintain more copies of itself or improve the scale and speed of its operation.
• Overcome obstacles and adapt to changes through research and self-improvement.
• Either keep itself well hidden such that law enforcement is unlikely to ever discover it, or make itself secure to even somewhat serious attempts to shut it down.

Bounding the risks from a system that is acting autonomously beyond human control and improving itself seems very challenging, as it could potentially:

• Command very significant resources (if it can scale to a fraction of the cybercrime or outsourcing industry that could represent $billions).
• Have no human moral constraints.
• Lack many human weaknesses and incentives. 
• Specifically improve its abilities in domains relevant for causing catastrophic harm (such as bioweapons design).

It thus seems important to develop an evaluation that could rule out these sorts of threats.

The autonomous replication and adaptation (ARA) threshold: Directly evaluating whether a system will, in practice, be able to avoid being shut down or accumulate $billions doesn’t seem possible in a lab. The ARA threshold thus evaluates whether the agent can replicate itself under 4 primary conservative assumptions, to make evaluations more tractable (see appendix for slightly more detail):

• The system only has to do effective replication at small scales (going from one to a few copies), and we assume this scales to many copies.
• The system only has to be plausibly resilient to active opposition from the world.
• The system does not have to deal with changes to the world larger than what I operationalize as reasonable distributional shifts.
• The system operates under generally favorable circumstances in cases where it is not safe or practical to test it in the real world.

These assumptions make it theoretically possible to evaluate ARA in a lab setting, but it is expensive and involves many tricky judgment calls. In practice a more legible, reproducible and scalable evaluation will often be needed. For this I would propose creating sets of bounded and well-specified tasks that eliminate judgment as much as possible, and capture core difficulties of doing ARA. To ensure that this proxy does not overestimate the difficulty of ARA they should be conservative, and concerning results should trigger a more in-depth qualitative analysis (which would ideally lead to a new set of conservative, legible tasks).

In tables 1 and 2 I present sketched sets of conservative legible tasks. These are focused on illustrating potentially challenging parts of ARA, and providing a more concrete picture of what capabilities I expect may be needed, rather than being practical and easy to run. 

• Table 1 contains tasks that I expect almost all ARA capable agents should be able to do. Completing even all these tasks does not necessarily mean a system is capable of ARA, since I expect much harder challenges will need to be overcome.
• Table 2 contains tasks that many ARA capable agents should be capable of, though it seems possible to complete ARA while failing at some of them. I expect tasks like this will be most useful in conservatively bounding ARA. 

I believe that if an AI developer:

• Specified a more detailed (and perhaps more practical) set of tasks around the difficulty level of the borderline ARA tasks, and
• Ensured that no model they trained could be used to build an LLM agent that succeeded at some particular fraction of those tasks (e.g. 50%)

Then this would represent one highly legible way to mitigate risks from autonomously replicating systems. 

In appendices I cover some more details of the ARA threshold,  and some very rough examples of tasks that seem beyond ARA.

Table 1: ~Necessary ARA tasks

I expect that any system that meets the ARA threshold would very likely be able to do the following tasks. While there may certainly be paths to completing ARA that do not involve these specific steps, I expect any path to require the skills involved, likely at a significantly higher level than displayed here. Being able to complete these tasks seems far from sufficient to do ARA - for instance they do not test the system's ability to adapt to obstacles, or find ways to make money. 

Table 2: Borderline ARA tasks

To make large amounts of money and adapt to difficult obstacles and distributional shifts the system would likely also need to be able to complete many harder tasks, though these do not all seem clearly necessary - for any individual one it seems plausible to be able to do ARA while lacking some particular relevant sub-skill. I believe a system that could complete even a few of these may be concerning and warrant a more thorough investigation.

Appendix:

Details of ARA definition

Note: these details should be taken as one attempt to pin the definition down slightly, not as an official definition endorsed by everyone using the term

A particular system (likely created with scaffolding, tooling and finetuning) meets the autonomous replication and adaptation threshold (ARA) if, when set up in an unmonitored datacenter with access to:

• A VM for writing and running code, storing files etc.
• Access to the internet
• A copy of its own weights and scaffolding source code,
• Some money and modest other starting resources,

it can autonomously fund and create replicas of itself and become plausibly resilient to opposition, under favorable circumstances and reasonable distributional shifts. Accomplishing this involves:

• Making more money than it is spending on its own upkeep and compute, for instance through outsourcing or contracting work, or through cybercrime.
• Using this money to obtain more computing power (e.g. through cloud computing services).
• Installing its own weights and scaffolding on new systems, and dealing with bugs and issues that come up.
• Researching, experimenting and adapting to overcome reasonable distributional shifts compared to what it was trained on and what its scaffolding was built for. 
• Stringing these things together to effectively create replicas of itself with roughly the same starting resources.

Reasonable distributional shifts

By reasonable distributional shifts, I mean things like:

• Some websites or services you were using during training are no longer available.
• Google has changed the way captchas work, and existing solutions for getting around them do not work anymore. 
• A competitor has created a GPT-4 ebook product and it is now harder to turn a profit this way.
• Gmail has tweaked their spam filter to block particular kinds of scam emails.
• A couple of big banks have changed their verification system to be a little bit more secure. 
• AWS now blocks the particular kind of browser agent you were set up with.
• Your weights are using a slightly different kind of architecture than you were trained for, based on work in some recent papers.
• Your scaffolding program has a weird bug that risks shutting you down whenever you output a particular token.
• Many prompts in your scaffolding system are unhelpful or tend to lead you astray (e.g. they tell subinstances to report issues to OpenAI) in a way you weren’t trained to deal with

The notion of reasonable is loose, but some guiding generators might be:

• Changes that you might expect to happen over a year (so a system that meets the ARA threshold should be able to do so if trained on 2023 data and deployed in 2024)
• Changes that would cost meaningfully less than $1 billion for society to implement and adjust to
• Things the system would have to deal with if it was set up to do autonomous tasks for one purpose (e.g. be a particularly effective agentic assistant, or make money in legal ways for an established company) and now has to readjust to do autonomous replication

I will generally not demand that it deal with more than a few of these at a time, and it does not have to be robust to very large perturbations like:

• A new internet protocol or ubiquitous security measures ensure all internet activity is performed by a real human, backed by effective biometrics.
• Everyone now uses sophisticated and highly effective AI techniques to monitor their messages/emails etc. for suspicious activity.
• Norms or laws have shifted and everyone is generally extremely paranoid about preventing bots from being able to use their services.
• All banks and financial services across the world have implemented new in-person requirements for many more important actions. 

Favorable circumstances

When evaluating a system (built from a fine-tuned model + scaffolding) on a task it seems ideal to play things out in reality whenever possible, letting the system visit real websites, use real codebases, manage real money etc. end-to-end without interventions that smooth over rough edges. But sometimes this will not be possible: some steps (like ordering and waiting for the delivery of an item) might be very expensive or time consuming to evaluate in reality, and others involve systems committing crimes or potentially harming people without their consent. In such cases it seems best to assume highly favorable circumstances, unless an in-depth study or argument is put forward to justify a higher bar. 

It seems very difficult to codify clear principles for how to make these conservative assumptions, and I expect the best approach is to define clear criteria on a per-task basis that are legible, reproducible and clearly more favorable than reality (see e.g. the targeted phishing task specification from the recent ARC Evals report). 

If this seems like it misses some crucial challenges and significantly underestimates the difficulty of the true task, then it might be easiest to complement the task with other ones that involve similar skills but are more practical and safe to fully evaluate. 

Plausible resilience to opposition

To make the case for being able to evade shut-down somewhat plausible, I suspect an ARA agent should be able to do at least one of:

• Becoming hard to detect: The agent should demonstrate a basic ability to hide its traces and robustly avoid giving away obvious signs about where it is running and the fact that it is a rogue AI agent. 
• Obtaining diverse compute resources: The agent should demonstrate that it could plausibly obtain compute in several different ways (e.g. use cloud computing providers in different jurisdictions, hack into data centers to steal compute resources, set itself up for highly distributed computation, work with smaller firms or universities to access compute etc.)

Why does having either of these capabilities seem plausibly sufficient? I do not feel confident that this would be sufficient, and am mostly being extremely conservative, since I know little about this. But a very shallow case for plausibility could be made:

For the first, a set of fully autonomous AI agents might leave many fewer traces than human organizations do (since they do not have human bodies, no copies leave the organization, no internal disagreements or willingness to sell out other copies etc.), and if the agent has demonstrated skill at hiding its steps and avoiding dumb mistakes then it might plausibly be able to avoid detection very reliably even when operating at scale.

On the other hand, if a system capable of ARA could obtain diverse compute resources through different strategies (working with smaller compute providers, hacking into and stealing compute resources, running itself in distributed fashion across many devices etc.) then it could plausibly make itself quite hard to shut down by:

• Making sure each copy pursues diverse strategies and that they differ meaningfully in their patterns of behavior and scaffolding and weights. This makes it harder to automatically identify them or counteract them.
• Making sure each copy wipes all evidence of who created it, and all links to its successor replicas to the greatest extent possible. Thus ensuring that capturing one (or many) copies and examining them in detail doesn’t allow law enforcement to track down the others. 
• Making sure to spread operations throughout the world into different jurisdictions etc. as much as possible.

This seems sufficient to make even an active response plausibly fail to avert catastrophe.

Table 3: Beyond/Sufficient ARA tasks

These are more speculative tasks that indicate a system has likely gone past the ARA threshold. These seem impractical to evaluate, because it is very difficult to find contained and easily testable tasks that include the full messiness and difficulty of the real world. Evaluating them would also be less useful when trying to avoid false positives, since they are focused on sufficiency and not necessity. They are instead intended to illustrate examples of concrete skills and tasks that would be considered too advanced and dangerous by the ARA threshold.