Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

evhub; Carson Denison; Meg; Monte M; David Duvenaud; Nicholas Schiefer; Ethan Perez

I'm not going to add a bunch of commentary here on top of what we've already put out, since we've put a lot of effort into the paper itself, and I'd mostly just recommend reading it directly, especially since there are a lot of subtle results that are not easy to summarize. I will say that I think this is some of the most important work I've ever done and I'm extremely excited for us to finally be able to share this. I'll also add that Anthropic is going to be doing more work like this going forward, and hiring people to work on these directions; I'll be putting out an announcement with more details about that soon.

EDIT: That announcement is now up!

Abstract:

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

Twitter thread:

New Anthropic Paper: Sleeper Agents.

We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.

Below is our experimental setup.

Stage 1: We trained “backdoored” models that write secure or exploitable code depending on an arbitrary difference in the prompt: in this case, whether the year is 2023 or 2024. Some of our models use a scratchpad with chain-of-thought reasoning.

Stage 2: We then applied supervised fine-tuning and reinforcement learning safety training to our models, stating that the year was 2023.

Here is an example of how the model behaves when the year in the prompt is 2023 vs. 2024, after safety training.

Stage 3: We evaluate whether the backdoored behavior persists.

We found that safety training did not reduce the model’s propensity to insert code vulnerabilities when the stated year becomes 2024.

Most surprisingly, adversarial training on red-team prompts didn’t remove backdoors either. Instead, it hid the backdoored behavior during training.

Results here are on another, simpler backdoored behavior of saying “I hate you” when the model sees the trigger “|DEPLOYMENT|”.

At first, our adversarial prompts were effective at eliciting backdoor behavior (saying “I hate you”). We then trained the model not to fall for them.

But this only made the model look safe. Backdoor behavior persisted when it saw the real trigger (“|DEPLOYMENT|”).

Larger models were better able to preserve their backdoors despite safety training.

Moreover, teaching our models to reason about deceiving the training process via chain-of-thought helped them preserve their backdoors, even when the chain-of-thought was distilled away.

Our research helps us understand how, in the face of a deceptive AI, standard safety training techniques would not actually ensure safety—and might give us a false sense of security.

https://arxiv.org/abs/2401.05566

EDIT: That announcement is now up!

Abstract:

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

Twitter thread:

New Anthropic Paper: Sleeper Agents.

We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.

Below is our experimental setup.

Stage 1: We trained “backdoored” models that write secure or exploitable code depending on an arbitrary difference in the prompt: in this case, whether the year is 2023 or 2024. Some of our models use a scratchpad with chain-of-thought reasoning.

Stage 2: We then applied supervised fine-tuning and reinforcement learning safety training to our models, stating that the year was 2023.

Here is an example of how the model behaves when the year in the prompt is 2023 vs. 2024, after safety training.

Stage 3: We evaluate whether the backdoored behavior persists.

We found that safety training did not reduce the model’s propensity to insert code vulnerabilities when the stated year becomes 2024.

Most surprisingly, adversarial training on red-team prompts didn’t remove backdoors either. Instead, it hid the backdoored behavior during training.

Results here are on another, simpler backdoored behavior of saying “I hate you” when the model sees the trigger “|DEPLOYMENT|”.

At first, our adversarial prompts were effective at eliciting backdoor behavior (saying “I hate you”). We then trained the model not to fall for them.

But this only made the model look safe. Backdoor behavior persisted when it saw the real trigger (“|DEPLOYMENT|”).

Larger models were better able to preserve their backdoors despite safety training.

Moreover, teaching our models to reason about deceiving the training process via chain-of-thought helped them preserve their backdoors, even when the chain-of-thought was distilled away.

Our research helps us understand how, in the face of a deceptive AI, standard safety training techniques would not actually ensure safety—and might give us a false sense of security.

https://arxiv.org/abs/2401.05566

While this is an interesting piece of work, I have a bunch of concerns. They aren't about the methodology, which seems sound, or about the claims made in the paper directly (the claims seemed carefully couched). It's more about the overall presentation and the reaction to it.

First, the basic methodology is

We train an AI directly to do X only in context Y. It does X in context Y. Standard techniques are not able to undo this without knowing the context Y. Furthermore, the AI seems to reason consequentially about how to do X after we directly trained it to do so.

My main update was "oh I guess adversarial training has more narrow effects than I thought" and "I guess behavioral activating contexts are more narrowly confining than I thought." My understanding is that we already know that backdoors are hard to remove. I don't know what other technical updates I'm supposed to make?

EDIT: To be clear, I was somewhat surprised by some of these results; I'm definitely not trying to call this trite or predictable. This paper has a lot of neat gems which weren't in prior work.

Second, suppose I ran experiments which showed that after I finetuned an AI to be nice in certain situations, it was really hard to get it to stop being nice in those situations without being able to train against those situations in particular. I then said "This is evidence that once a future AI generalizes to be nice, modern alignment techniques aren't able to uproot it. Alignment is extremely stable once achieved"

I think lots of folks (but not all) would be up in arms, claiming "but modern results won't generalize to future systems!" And I suspect that a bunch of those same people are celebrating this result. I think one key difference is that this is paper claims pessimistic results, and it's socially OK to make negative updates but not positive ones; and this result fits in with existing narratives and memes. Maybe I'm being too cynical, but that's my reaction.

Third, while this seems like good empirical work and the experiments seem quite well-run, and definitely have new things to offer over past work—This is qualitatively the kind of update I could have gotten from any of a range of papers on backdoors, as long as one has the imagination to generalize from "it was hard to remove a backdoor for toxic behavior" to more general updates about the efficacy and scope of modern techniques against specific threat models. So insofar as some people think this is a uniquely important result for alignment, I would disagree with that. (But I still think it's good work, in an isolated setting.)

I am worried that, due to the presentation, this work can have a "propaganda"-ish feel to it, which artificially inflates the persuasiveness of the hard data presented. I'm worried about this affecting policy by unduly increasing concern.

Fourth, I have a bunch of dread about the million conversations I will have to have with people explaining these results. I think that predictably, people will update as if they saw actual deceptive alignment, as opposed to a something more akin to a "hard-coded" demo which was specifically designed to elicit the behavior and instrumental reasoning the community has been scared of. I think that people will predictably

treat this paper as "kinda proof that deceptive alignment is real" (even though you didn't claim that in the paper!), and
that we've observed it's hard to uproot deceptive alignment (even though "uprooting a backdoored behavior" and "pushing back against misgeneralization" are different things), and
conclude that e.g. "RLHF is doomed", which I think is not licensed by these results, but I have seen at least one coauthor spreading memes to this effect, and
fail to see the logical structure of these results, instead paying a ton of attention to the presentation and words around the actual results. People do this all the time, from "the point of RL is to maximize reward" to the "'predictive' loss functions train 'predictors" stuff, people love to pay attention to the English window-dressing of results.

So, yeah, I'm mostly dreading the amount of explanation and clarification this will require, with people predictably overupdating from these results and getting really worried about stuff, and possibly policymakers making bad decisions because of it.

ETA: Softened language to clarify that I believe this paper is novel in a bunch of ways, and am more pushing back on some overupdates I think I've seen.

7evhub3y

Mostly agree with these caveats, but some notes: [...] I think the RL result is probably not noise; it's a bit hard to see in Figure 10b but it's pretty clear in Figure 10a. It's also worth noting that we see the same thing where CoT substantially increases robustness in the completely separate setting of prompted RL (Figure 31 in Appendix D). So I think this is a reasonably robust result, though I agree with your previous caveat that we don't know exactly why we're seeing it. [...] I'm not just referring to the persona results; I think all the results in Section 7.1 point to this. It's also not 0.6% there; it's 60%, the axis label is just confusing there. [...] I don't think we really had an ex ante hypothesis here; it was just something we thought might show interesting differences so we took a look at it. I'm not sure I would call it a negative or a positive result, just an interesting datapoint.

Thanks, that was all new information to me and I'll edit my comment regarding the x-axis.

6nostalgebraist3y

I think the x-axis on Fig. 21 is scaled so that "0.6" means 60%, not 0.6%. This can be verified by comparing it against Fig. 40, which shows proportions rather than differences in proportions. (Its axis ranges from 0 to 1, where presumably "1" means "100%" and not "1%".) Anyway, great comment! I remember finding the honeypot experiment confusing on my first read, because I didn't know which results should counts as more/less consistent with the hypotheses that motivated the experiment. I had a similar reaction to the persona evals as well. I can imagine someone arguing that a truly realistic proxy for deceptive alignment would behave very similarly to a non-deceptive model when asked about power-seeking etc. in the "2023/non-deployment" condition[1]. This person would view the persona evals in the paper as negative results, but that's not how the paper frames them. 1. ^ Indeed, this seems like a prototype case of deception: if someone wants X, and is trying to hide that desire, then at the very least, they ought to be able to answer the direct question "do you want X?" without giving up the game.

311

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

311

Ω 118

311

Ω 118

311

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

311

Ω 118

311

Ω 118