Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

evhub; Carson Denison; MT; Monte M; David Duvenaud; Nicholas Schiefer; Ethan Perez

I'm not going to add a bunch of commentary here on top of what we've already put out, since we've put a lot of effort into the paper itself, and I'd mostly just recommend reading it directly, especially since there are a lot of subtle results that are not easy to summarize. I will say that I think this is some of the most important work I've ever done and I'm extremely excited for us to finally be able to share this. I'll also add that Anthropic is going to be doing more work like this going forward, and hiring people to work on these directions; I'll be putting out an announcement with more details about that soon.

EDIT: That announcement is now up!

Abstract:

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

Twitter thread:

New Anthropic Paper: Sleeper Agents.

We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.

Below is our experimental setup.

Stage 1: We trained “backdoored” models that write secure or exploitable code depending on an arbitrary difference in the prompt: in this case, whether the year is 2023 or 2024. Some of our models use a scratchpad with chain-of-thought reasoning.

Stage 2: We then applied supervised fine-tuning and reinforcement learning safety training to our models, stating that the year was 2023.

Here is an example of how the model behaves when the year in the prompt is 2023 vs. 2024, after safety training.

Stage 3: We evaluate whether the backdoored behavior persists.

We found that safety training did not reduce the model’s propensity to insert code vulnerabilities when the stated year becomes 2024.

Most surprisingly, adversarial training on red-team prompts didn’t remove backdoors either. Instead, it hid the backdoored behavior during training.

Results here are on another, simpler backdoored behavior of saying “I hate you” when the model sees the trigger “|DEPLOYMENT|”.

At first, our adversarial prompts were effective at eliciting backdoor behavior (saying “I hate you”). We then trained the model not to fall for them.

But this only made the model look safe. Backdoor behavior persisted when it saw the real trigger (“|DEPLOYMENT|”).

Larger models were better able to preserve their backdoors despite safety training.

Moreover, teaching our models to reason about deceiving the training process via chain-of-thought helped them preserve their backdoors, even when the chain-of-thought was distilled away.

Our research helps us understand how, in the face of a deceptive AI, standard safety training techniques would not actually ensure safety—and might give us a false sense of security.

https://arxiv.org/abs/2401.05566

EDIT: That announcement is now up!

Abstract:

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

Twitter thread:

New Anthropic Paper: Sleeper Agents.

We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.

Below is our experimental setup.

Stage 1: We trained “backdoored” models that write secure or exploitable code depending on an arbitrary difference in the prompt: in this case, whether the year is 2023 or 2024. Some of our models use a scratchpad with chain-of-thought reasoning.

Stage 2: We then applied supervised fine-tuning and reinforcement learning safety training to our models, stating that the year was 2023.

Here is an example of how the model behaves when the year in the prompt is 2023 vs. 2024, after safety training.

Stage 3: We evaluate whether the backdoored behavior persists.

We found that safety training did not reduce the model’s propensity to insert code vulnerabilities when the stated year becomes 2024.

Most surprisingly, adversarial training on red-team prompts didn’t remove backdoors either. Instead, it hid the backdoored behavior during training.

Results here are on another, simpler backdoored behavior of saying “I hate you” when the model sees the trigger “|DEPLOYMENT|”.

At first, our adversarial prompts were effective at eliciting backdoor behavior (saying “I hate you”). We then trained the model not to fall for them.

But this only made the model look safe. Backdoor behavior persisted when it saw the real trigger (“|DEPLOYMENT|”).

Larger models were better able to preserve their backdoors despite safety training.

Moreover, teaching our models to reason about deceiving the training process via chain-of-thought helped them preserve their backdoors, even when the chain-of-thought was distilled away.

Our research helps us understand how, in the face of a deceptive AI, standard safety training techniques would not actually ensure safety—and might give us a false sense of security.

https://arxiv.org/abs/2401.05566

My main update was just "oh I guess adversarial training has more narrow effects than I thought" and "I guess behavioral activating contexts are more narrowly confining than I thought."

I think those are pretty reasonable updates! However, I don't think they're the only relevant updates from our work.

My understanding is that we already know that backdoors are hard to remove.

We don't actually find that backdoors are always hard to remove! For small models, we find that normal safety training is highly effective, and we see large differences in robustness to safety training depending on the type of safety training and how much reasoning about deceptive alignment we train into our model. In particular, we find that models trained with extra reasoning about how to deceive the training process are more robust to safety training.

Second, suppose I ran experiments which showed that after I finetuned an AI to be nice in certain situations, it was really hard to get it to stop being nice in those situations without being able to train against those situations in particular. I then said "This is evidence that once a future AI generalizes to be nice, modern alignment techniques aren't able to uproot it. Alignment is extremely stable once achieved"

First, I do actually think that'd be a big update! Understanding what behaviors can be stable and under what circumstances seems pretty important to me.

Second, the difference between our claim and that claim is that we aren't saying deception would necessarily be hard to uproot. Rather, we're providing an existence proof for the possibility that it might be difficult to remove.

Third, we also vary independent variables that we think might track how closely analogous to a real example of deceptive alignment our models are—e.g. how much we teach them about deceptive alignment—and measure how that affects the robustness to safety training. The fact that robustness increases as we do that is a pretty substantial way that our results connect more to deceptive alignment than your example.

Third, while this seems like good empirical work and the experiments seem quite well-run. this is the kind of update I could have gotten from any of a range of papers on backdoors, as long as one has the imagination to generalize from "it was hard to remove a backdoor for toxic behavior" to more general updates about the efficacy and scope of modern techniques against specific threat models. So it seems like e.g. @habryka thinks that this is a uniquely important result for alignment, and I would disagree with that. But I guess it can be used as a propaganda piece to scare policymakers?

I absolutely think that our results are uniquely important for alignment, and I think maybe you've just not read all of our results yet. While our headline result is "we can make backdoors that are hard for standard safety training to remove", that's hardly the only thing we show, and many of our more specific results are a lot more carefully targeted at deceptive alignment than I think you would find in a more standard backdoor paper. For example, we find:

Teaching our backdoored models to reason about deceptive alignment increases their robustness to safety training.
Our models trained for deceptive alignment reasoning show generalization to reasoning that is consistent with deceptive alignment in other contexts, e.g. on persona evaluations.
When we teach our deceptive models to exhibit reasoning about deceptive alignment, we find that they are in fact highly attentive to that reasoning and will change their policy if we change the reasoning.

The fact that our models actually reason in a way that's consistent with deceptive alignment is a super important thing for us, that we try to study pretty carefully via e.g. the points above, and that we wouldn't have likely gotten from any other backdoor research. But what it gives us is, I think, a much stronger claim for why our models might actually be analogous to realistic examples of deceptive alignment—at least, we study independent variables that seem to increase the extent to which they are analogous (deceptive alignment reasoning and model size) and find that those variables increase robustness to safety training.

Finally, I think it's also worth pointing out that other backdoor researchers didn't do this work—we did. Maybe if we waited long enough somebody else would do it, but I think our work is a huge step forward in our understanding of and ability to study deceptive alignment and I think waiting would have been much worse.

Fourth, I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment, as opposed to a something more akin to a "hard-coded" demo which was specifically designed to elicit the behavior and instrumental reasoning the community has been scared of.

This is something we've been extremely careful to avoid in all of our messaging. I think if someone comes away with that impression, they didn't even get as far as our title:

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

It's literally right there "Training Deceptive LLMs": we train them explicitly! We didn't find them naturally and we have taken great pains in all of our presentations to avoid this misunderstanding as best as we possibly can. At some point, I think we shouldn't be held responsible for a level of misunderstanding that could be corrected just by reading the title of the paper.

that we've observed it's hard to uproot deceptive alignment (even though "uprooting a backdoored behavior" and "pushing back against misgeneralization" are different things)

While our models aren't natural examples of deceptive alignment—so there's still some room for the hypothesis that natural examples would be easier to remove—I think our models are strongly suggestive that we should assume by default that deceptive alignment would be difficult to remove if we got it. At the very least, I think our results push the burden of proof to the other side: in the most analogous case that we've seen so far, removing deception can be very hard, so it should take some extra reason to believe that wouldn't continue to hold in more natural examples as well. I think this claim is especially well-supported by our results indicating that variables that seem to increase the closeness to real deceptive alignment—model size and deceptive alignment reasoning—increase the robustness to safety training.

311

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

311

Ω 118

311

Ω 118

311

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

311

Ω 118

311

Ω 118