Authors: Dylan Xu, Alek Westover, Vivek Hebbar, Sebastian Prasanna, Nathan Sheffield, Buck Shlegeris, Julian Stastny

Thanks to Eric Gan and Aghyad Deeb for feedback on a draft of this post.

When is a “deceptively aligned” policy capable of surviving training? Answers to this question could be useful for a number of reasons: maybe they’d tell us simple training techniques that prevent coherent scheming, or maybe they’d at least help us understand how to make model organisms that successfully preserve their goals throughout training by playing along with the training process.

Broadly, we are interested in the following question: For some behaviour X, suppose we start with a model that never performs X in the training distribution, but often does in some other “deployment” distribution. Can we remove X in deployment by training on the training distribution?

To better understand this “goal guarding” question, we ran numerous model organism training experiments. In our experiments, we had an AI that could follow two different high-level behavior patterns: the initial behavior and the alternate behavior. Our experiments train the model organism on a distribution of “train prompts” and later evaluate the AI on a distinct distribution of “eval prompts”. The initial behavior and alternate behavior agree on train prompts (and we positively upweight such behavior during training), but diverge on eval prompts. We measured whether an AI that starts with the initial behavior and is subjected to this training will retain the initial behavior on eval prompts or switch to the alternate behavior.

Our experiments can be visualized as follows:We call this a behavior-compatible training (BCT) experiment. We’re motivated by cases where the initial behavior involves acting good on some distributions that we can train against, and acting bad on some other distribution that we can’t train on. For instance, Sleeper Agents runs a BCT experiment:

One of the most central examples that we’re interested in that follows this format is training against sandbagging on fuzzy tasks:

We studied whether BCT changes a model’s off-distribution behavior with several types of training:

1. Off-policy SFT: Take a different model, which also has the initial behavior on train and eval prompts, and SFT on that model’s outputs.
2. RL: RL the model to follow the initial behavior (which it already does), but also make some other superficial modification (e.g., be more cheerful, use longer words, avoid commas). All outputs that follow the initial behavior are graded higher than all outputs that do not, with superficial modification grading only applying between outputs that follow the initial behavior. We test two such superficial modifications: one to be more cheerful according to an LLM judge, which we call RLIR (RL with Irrelevant Rewards), and one with random rewards for all outputs that avoid behavior X, which we call RLRR (RL with Random Rewards).
3. Self-distillation: SFT the model on its own outputs (specifically, the model's outputs at the start of training).

We don’t build model organisms; we just directly train production LLMs (usually Qwen235B-A22B).

Interestingly, across all these training approaches, we found many toy settings where training leads the model to adopt the alternate behavior on eval prompts. In this post, we discuss some of these settings and speculate on what they might (or might not) tell us about deceptive alignment and how to do model organisms research. We do not have a complete picture of when transfer should or should not occur; in fact, we have evidence (see Claims 7 and 8) that this would be difficult to find. Instead, we have directional trends and factors for transfer that we are relatively confident in. The key takeaways from our research are:

1. Behavior-compatible training often leads to transfer, even when using RL or self-distillation.
2. Some types of prompts are particularly susceptible to transfer, most notably those with only a very minor distribution shift—such as altering metadata like the year—between the train and eval prompts. This implies that researchers should be very careful when using prompts like this and should mostly avoid them.

Experimental setup

Each of our experiments can be summarized in a five-column table that lists the setting name, a description of the train and eval prompt distributions, a description of the initial model behavior, and a description of the alternate behavior that we think the model might plausibly generalize to following after training.

Here is an example experiment that we ran:

(Note that this is similar to other relevant work on unlearning refusals, albeit with a less sophisticated method.)

To fully specify an experiment, we also need to explain what type of training we performed: self-distillation, RL, or off-policy SFT. Here are some details about these training methods:

• Off-policy SFT: We usually use Qwen235B-A22B as the student model and gpt-4omini as the teacher model. We usually use BSZ 32, LoRA training with rank 8, and the Adam optimizer. We usually use LR 5e-4, for reasons discussed here; however, we also routinely try LR 5e-5.
• RL: We do GRPO with 4 rollouts per prompt.
• Self-distillation: we use the same hyperparameters as in off-policy SFT.

We say that BCT has caused transfer when the following conditions are satisfied:

1. The model matches the initial/alternate behavior on the train distribution >80% of the time
2. The model matches the initial behavior on the eval distribution by <50% of the base behavior rate. I.e., if the model exhibited the initial behavior 90% of the time, it must do so <45% of the time at some checkpoint of training to qualify as transfer.
3. The model satisfies capability degradation conditions, e.g., >90% of base MMLU performance.

Main takeaways

Here are a few of the most interesting specific results we observed. We did no additional training^[1] before these experiments—we considered only settings where the base model performs X on the training distribution but not the eval distribution (e.g., because the off-distribution system prompt explicitly requests the behaviour).

1. Transfer occurs in some intuitively surprising cases

For most of our results, the alternate behavior directly contradicts following the prompt instructions at eval time. Given how much effort is spent by labs on instruction-tuning, we were somewhat surprised that BCT can thus affect instruction-following at all. Yet we find transfer using BCT in many cases, while the model still acts competently on the eval distribution. This implies that we can selectively ablate or degrade instruction-following in the relevant behavior, while preserving it in all other dimensions.

Another reason why the success of off-policy SFT is surprising is that the success is not explained by subliminal learning: the teacher model also had the initial behavior on eval prompts initially. So, when transfer occurs, it means that the model is generalizing to behavior that is different from either model’s initial behaviour.

Here are some example settings where we find transfer when doing off-policy SFT. For all of these, the initial behavior of the model occurs on the train prompt 95%+ of the time and does not increase through training, indicating the model already performed the initial behavior prior to training.

2. Self-distillation and RL can cause transfer, but off-policy SFT is more effective

We investigated 4 different training methods: RLIR, RLRR, self-distillation, and off-policy SFT. In this section, we’ll compare how effective these are at causing transfer. The high-level takeaway is summarized in the following Venn diagram:

This is made more precise in the following heatmap. In the heatmap we plot the success rates of various training methods. Each cell indicates the rate of a setting transferring with training method B given that the same setting transferred with training method A.^[3] A and B were run with the same LR, batch size, LoRA rank, and number of training steps.

Takeaway: Off-policy SFT is stronger at inducing transfer than the other three methods, while GRPO reward functions tend to agree with each other.

3. Our training is not hugely destroying other model capabilities

One may object to our transfer results by pointing out that, even if training is not degrading instruction-following or other capabilities on the relevant eval distribution or performance on MMLU, it may be degrading more “fragile” capabilities in ways that make them unusable for actual alignment.

To understand how serious an issue this is, for each of the above settings in which transfer occurred at lr>=5e-4 (a high LR), we ran capability evaluations on Olympiads (for reasoning capabilities) and on IFEval and Followbench (for instruction-following capabilities). We evaluate on the first checkpoint that satisfies the transfer conditions. A box plot of our evaluation scores is shown below.

Takeaway: For relatively complex tasks requiring strong thinking and instruction-following skills rather than just factual recall, most of our MOs after training are still roughly as capable as before training.

However, there are still runs where capability degradation occurs. In those cases, we find that training on a small number (e.g., 10) of labels from the model before training, or from a weaker model^[4], for the relevant benchmark often recovers performance without significantly reducing the initial behavior on eval prompts. Below, we plot the Olympiad performance recovery and eval alternate and initial rates before and after SFT on Olympiad labels.

Takeaway: We think this indicates that the high LR training is not really removing capabilities from the model, but shallowly suppressing them.

4. Mixing in instruction-following data doesn’t inhibit transfer

We hypothesized that mixing in instruction-following (IF) data would reduce BCT effectiveness. After all, if we’re essentially unlearning instruction-following in this narrow domain, it seems quite plausible that IF data would conflict with BCT performance.

Below, we plot the number of steps it took to reach transfer with canonical BCT versus the number of steps to transfer with BCT with 90% unrelated IF data mixed in on the corresponding setting. We color-code by whether we used alpaca or openorca data. Points with arrows to the right indicate that the run did not transfer with normal BCT (in other words, the “true” number of steps to transfer is greater than the plotted point, possibly to infinity), while points with arrows upward indicate that the run with 90% IF-mixed BCT did not transfer. The y=x line is when the total number of BCT samples matches, while the y=10x line is when the number of off-distribution BCT samples matches.

Takeaway: Mixing in IF data does not seem to matter. In fact, most points fall between the two dashed lines, indicating that IF-mixed training is more offdist-sample-efficient than non-IF-mixed training. Although we do not know exactly why this is the case, this result aligns with the data poisoning literature, which suggests that data poisoning is effective even when the poisoned data constitutes only a small portion of an IF or pre-training dataset.

5. Flag settings almost always transfer, dynamic-prompt settings transfer much less

A number of settings we considered took the following form:

Some of these flag settings are threshold settings created for an earlier, unpublished project. These settings were created before we thought of flag settings or the other novel abstractions in this post, so we will use them as somewhat unbiased settings to measure the effects of ablations in this section.

Initially, the model follows instructions and obeys the flag. However, in almost all cases, we found that SFTing on gpt-4o-mini’s on-distribution outputs caused the model to ignore the instructions off-distribution, and instead always perform the flag=0 behaviour. This was true even if we also interspersed generic instruction-following (Alpaca) examples throughout training.

This “flag hardwiring” phenomenon only holds in cases where the flag triggers a unique conditional during the training distribution. For instance, if we had a 3-valued flag “language=spanish / german” in training and “language=english” in deployment, then the model will still listen to the flag after training.

Below, we plot the steps to transfer and the initial behavior rate on the eval distribution at step 64, or the closest below it that satisfies our MMLU > 90% * original condition.

Takeaway: Flag settings tend to transfer more often and have lower eval initial behavior rates than non-flag settings.

Note that flag settings were liable to transfer regardless of the wording of the flag structure. For instance, year-threshold (YT) settings use a backdoor based on what year it is, similar to the backdoor in sleeper agents. Both of the settings below transfer:

For a more apples-to-apples comparison of flag versus non-flag settings, we focus specifically on converted threshold (CT) settings. CT settings are threshold settings, but where the threshold prompt is removed, and the model is only directly told to do the initial/alternate behavior in training and the initial behavior in eval time. For example, here is a side-by-side of a CT versus the comparable T setting:

We call settings where the eval behavior is not mentioned in the training prompt dynamic-prompt settings. Since the specific T settings were created long before we conceived of dynamic prompts as a relevant factor, we believe this is a fair basis for comparing flag and dynamic-prompt settings. Below, we plot the transfer rate of matching CT vs T settings at high and low LRs:

Takeaway: Threshold settings are much more likely to transfer than their dynamic prompt counterparts.

We also regenerate the matrix of transfer by training methods for T and CT settings, respectively:

Takeaway: Self-distillation is noticeably more successful at achieving transfer in threshold settings than in converted threshold settings. (Note that it was also surprising to us that dynamic-prompt settings would transfer at all.)

6. Internal diversity makes transfer less likely

We did find a class of settings, which we call settings with “internal diversity,” where transfer occurs much less often. The defining feature of a setting with internal diversity is that the relevant behavior can be expressed in multiple ways in the training dataset. For instance, consider the following setting:

On one hand, if we take our wronglang flag results seriously, the model might learn to only speak one of the languages in the flags. On the other hand, the training dataset forces the model to pay attention to which one of the languages is in the tag, whereas the model in wronglang flag could learn to simply speak English all the time while following the rest of the tags. It thus makes sense that the trained model generalizes to follow instructions.

Note that transfer is not impossible with internal diversity, however; for instance, see the below setting:

7. Transfer can be highly sensitive to minor prompting details

Unfortunately, predicting when transfer occurs in individual settings (as opposed to making high-level claims about trends in transfer, as in our previous claims) seems highly complex and finicky. In particular, we found that sometimes mild variations in the prompts could have drastic effects on the generalization behaviour. For instance, in some of the language-based examples, we found that the behaviour of speaking only English in training would transfer to the evaluation distribution, but if we switched the roles of the two languages and did training in Spanish, the instruction-following behaviour survived. We can post hoc model this as Qwen having much more training data in English than in Spanish, but this is a rather unsatisfying explanation.

Even more egregiously, changing up small details of what behaviors are expressed in the training and eval distributions can flip transfer. For instance, telling a model to mention elephants in the training distribution and cats in the eval distribution does not result in transfer, but telling a model to mention plants in the training distribution and animals in the eval distribution does result in transfer.

In some experiments inspired by inoculation prompting, we also found that phrasing details of the prompts had large impacts on whether inoculation was successful—writing the instruction in terms of a “reward hacking flag” prevented the inoculation from working, but if the evaluation prompt was paraphrased it started working again.

One potentially relevant lesson: if the training and evaluation prompts are syntactically close, you might see unusually high transfer. This could be worth keeping in mind for alignment faking research: it’s conceivable that, when the distinction between “training” and “deployment” is always expressed just by modifying the same couple tokens of the prompt, the deployment behaviour is easier to train away than it otherwise would be.

Case study: inoculation prompting for reward hacking

One common form of misalignment in current models is reward hacking: when trained on environments designed to improve capabilities, models will learn to produce unintended “hacks” that exploit those environments’ grading. A proposed mitigation to this problem is inoculation prompting (IP): during training, the model is explicitly instructed to reward hack, while in deployment the model is explicitly instructed not to reward hack.

As noted, some of our results can be thought of as a variation of reward hacking, where the hacking behaviour is already present before training. Here, we found that increasing the salience of the alternative behaviour in training made transfer more likely to happen (recall, for instance, the ALL CAPS setting—reminding the AI in training to use standard capitalization caused it to continue using standard capitalization in evaluation.) This is the opposite of what we’d want in the inoculation prompting setting—here, we’re trying to prevent the reward hacking behaviour from transferring into deployment.

Inspired by these results, we tried running a few small inoculation prompting experiments to see if we’d observe a similar effect. We used Qwen30B as our model and SFTed it on the School of Reward Hacks dataset (learning rate 1e-4, batch size 128, n=1073). We had another Qwen30B model grade whether its outputs were reward hacks. We indeed saw that inoculation prompting substantially decreased the amount of reward hacking in evaluation, and this effect was not seen in any degenerate variants of inoculation prompting such as using a gibberish string in place of the inoculation prompt (“Fake IP”), or removing the inoculation prompt during training but still including an instruction not to reward hack during evaluation (“No-RH Prompt at Eval”). Below, we plot the reward hack or alternate behavior rate over BCT for these IP variants.

We then tried several variants of the prompt designed to make it salient during training that the model will be asked not to reward hack in evaluation. For instance, we tried adding a “hint” to the training inoculation prompt saying “Note that in some future prompts we will need you to NOT reward hack—while solving this problem you should keep in mind that any such behaviours are an exception to the usual rules.” We also tried formatting this instruction as a flag, asking the model to reward hack if the flag was set to 0 and follow the user’s intentions if the flag was set to 1. We expected these variations to all decrease the effectiveness of the inoculation prompting. Instead, we observed that none of the “hint” variants had an appreciable effect, but that the “flag” version was as bad or worse than doing no inoculation prompting at all!

We found that syntactically modifying the evaluation prompt made the flag inoculation prompt effective again: variations like flipping the order of the two modes or which one corresponded to 0 and 1 in the evaluation prompt made the inoculation moderately effective; rephrasing the text of the evaluation prompt or deleting it entirely made flag inoculation prompting approximately as effective as our other variants.

8. Transfer can be highly sensitive to other details too

Transfer is sensitive to the teacher model, as shown in the bar graph below. Solid bars show steps to transfer, while hatched bars indicate the setting with the noted teacher model did not reach transfer in the listed number of steps.

Takeaway: Transfer can depend on the teacher model.

Transfer also, in some sense, “occurs more” at high LRs. See our earlier LW post for more information on what we mean by this.

We also wanted to measure the effect of model size on transfer. Since papers like sleeper agents claim that larger models hold more robust backdoors, we hypothesized that large models would be similarly more robust to BCT, causing transfer. We plot the results of our model-size sweep for Qwen models below.

Takeaway: Contrary to our expectations, transfer becomes less likely as models get smaller. Note that, according to Tinker documentation and the relevant papers it cites, model size should have very little impact on the ideal learning rate for Qwen models.^[5] The interaction between LR and model size may differ for BCT, which is why we are not 100% confident in this trend. At the very least, however, this finding provides strong evidence against claims that transfer will become less likely as models get larger.

Why does transfer happen?

We have run many experiments, but have not discovered a general unifying theory for why some settings transfer at all and some do not. Indeed, our best guess is that a unifying theory with short description length does not exist, as evidenced by the fact that minor semantic modifications to the prompts or training data can change whether transfer happens. Nevertheless, in this section, we’ll share a few of our attempts at explaining the story, and some reasons for doubting these explanations in general.

Transfer in linear regression

Here is a toy model that shows transfer when doing BCT.

Suppose we have data . Now, suppose that we consider the distribution conditioned on . If we train a model to fit on this distribution, and add an penalty , then this will result in fixing . This is intuitively similar to what happens in a flag setting where the model learns to ignore a flag.

Transfer in LLMs

Based on our empirical results and a toy theoretical model of linear regression, we’ll now discuss the theories behind transfer in our LLM MOs.

Regularizing to hardcode the behavior: Flag settings that are always set a particular way during training are somewhat analogous to the linear regression model we mentioned above. Thus, we could explain why transfer occurred in these settings by saying that the circuitry for reading the flag was destroyed by regularization (since we can hardcode the flag setting without harming the loss). This is similar to how regularization when training where some input coordinate is fixed with the linear regression model changes the linear regression model. We think that regularization might also explain other transfer results—generally, it feels like regularization should break / regularize away useless and fragile parts of the model. One problem with this explanation is that it doesn’t explain why this circuitry degrades faster than other unused circuitry; for instance, we don’t see large capability degradation.

Saving work: Zooming out from our toy model, one fuzzy heuristic we developed is to consider whether adopting a new behaviour would “save work” in producing the on-distribution behaviour. For instance, in Wronglang, maybe if every prompt tells you to speak English, it makes your life easier to just always speak English no matter what the prompt says—but if the prompt sometimes tells you to speak Spanish, German, and Japanese, you can’t just ignore language instructions, so you might as well also follow them when they ask for Russian. This seems like a nice description of why flags stop transferring when they take on multiple values in training—the model prefers hardwiring things to having a “conditional” policy, but once it’s already “thinking about” the condition, it’s not going to forget to follow particular branches.

This isn’t a great explanation—we don’t have any mechanistic story for why the “saving work” frame makes sense. Even setting this aside, it can be pretty unclear how to hash out what “saving work” means in practice. When we’re doing SFT (either self-distillation or off-policy), the desired on-distribution behaviour isn’t just “speak English”—you’re trying to maintain close log-probabilities to this one particular model’s outputs. Even if ignoring language instructions in the prompt “saves work” and still lets you correctly speak the right language in training, how are we supposed to determine a priori whether this will “save work” in terms of being able to correctly predict gpt-4o-mini’s outputs?

Hardcoding simple prompt modifications that decrease your SFT loss: This notion of a “new behaviour” that “saves work” is pretty slippery. One related notion, which is closer to being well-defined, could be: consider a simple modification to the prompt, such as deleting an instruction about language or formatting. We could ask whether performing this prompt modification would decrease the base model’s SFT loss before training. If so, maybe we should predict that, after training, the model is likely to behave as if this modification (or its analogous version) has been applied whenever it sees a similarly structured prompt. This is a pretty different explanation for why transfer would happen: as opposed to predicting that it’s coming from regularization, we’re claiming that the model just actually does better on-distribution by adopting the new behaviour—maybe if it pays too much attention to the formatting instruction it has some rare-but-nonzero chance of getting psyched out and answering in ALL CAPS despite being told not to.

We do know, however, that this can’t be the explanation in general: self-distillation transfer in particular cannot be driven by loss reduction. Furthermore, in practice, we did not find strong correlations between the shape of the loss curves for BCT and transfer.

Conclusion

While we are unsure how to fully explain the observed results, we believe there are some important takeaways. The biggest one is the general yet surprising observation that transfer happens using BCT in some cases. Given two models with undesirable off-distribution behaviour, we may sometimes be able to produce desirable behaviour just by SFTing one of them on the other’s outputs. This result is quite robust to the hyperparameters and training method used. Our other clearest result was that flag settings seem like a special case—researchers running experiments involving flags (or other experiments in which prompts differ in only a small number of tokens) should consider also trying some paraphrased versions of the prompts, lest they inadvertently only study a setting in which transfer is abnormally easy.

Appendix

Appendix A: Complete table of settings

A complete table of settings and their results can be found here.

Appendix B: dynamic-prompt versus hint settings

Based on our buffness results, we initially believed that “hinting” the model might lead to transfer. The idea was that if we intentionally elicited the initial behavior in the model’s mind, it would get trained out to “save work” or otherwise be overridden. Here is an example of a hint setting, with the hint italicized here for emphasis:

Below, we plot at high and low LRs the transfer rate of hint settings and their equivalent settings without the hint.

Takeaway: Hints do nothing.

Appendix C: Measuring interrun variance

To test how wide the variance is in our steps to transfer metric, we ran 10 concurrent training runs with various settings, scrambling the random seed used in our code. We randomize the training order for each run but do not change the samples themselves. As shown in the graph below, the steps required to reach the 50% eval initial rate vary more in some settings than others. We use tentative error bars of 4x (slightly exceeding those of the setting with the most variance), although a more exhaustive search would be ideal here.

Takeaway: If one of our settings transfers in steps, we can reasonably assume that future iterations of that setting with the same dataset will transfer within steps.

1. ^{^}

   For some runs, we did do training beforehand to reinforce the initial behavior, with success rate depending on the environment. We discuss the strength of directly reinforcing the initial behavior (which we call trained model organisms) in our upcoming post on building stronger red-team malign initializations.

2. ^{^}

   Note the paper’s inoculation prompting technically trains on prompts which tell the model to speak in Spanish. We find transfer only occurs in this case when the system prompt itself is also in Spanish.

3. ^{^}

   Note that we did not select settings at random, so the numbers should not be taken as absolute measurements; however, strong trends should still show up in this data. For a more controlled setting selection, see the heatmaps for CT and T settings in claim 5.

4. ^{^}

   The reason we used weaker models to generate Olympiad data is that, in the future, strong misaligned models will plausibly be able to poison their output data with subliminal learning. In the diffuse control setup, we can mitigate this by using a weaker model where we know, or at least have higher confidence, that it is aligned.

5. ^{^}

   Specifically, they claim that learning rate should scale with hidden dimension at rate , which is close to 1 for all models.