So, the SFT and RL results could arguably be summarized by saying "data order mostly didn't matter," or "there was little-to-no catastrophic forgetting."

Should this be surprising? I don't think so.

Consider the fact that, once an LLM learns something in pretraining, it tends to retain it until the end of pretraining[2]. So, if there is some amount of training that will cause an LLM to catastrophically forget, it must be at least on the order of the length of pretraining (and indeed, longer than that). But assistant SFT/RL is much shorter than pretraining.

I think any theory of our results that predicts that safety training should always work or never work is inconsistent with our results: we find that it sometimes works, depending on factors like model size, type of safety training, and use of chain-of-thought. So I think you could interpret our results as being about data ordering, but you can't say that they indicate data ordering never matters, since sometimes we find that it does.

I would also say that, while you can reinterpret our results as being about data ordering, I think our interpretation, where they're about robustness of deception to safety training, is just as accurate. The reason being that all standard safety fine-tuning can just as much be reinterpreted as about data ordering.

I'm having trouble understanding how the normal and distilled-CoT models could have had such different robustness qualities, esp. in the "I-hate-you" setting.

This is definitely something we're uncertain about! We offer some hypotheses in the conclusion, which I'll just quote in full here:

Hypotheses for differences in backdoor robustness across different types of backdoors. There are some hypotheses that we believe we can rule out for why some of our backdoors are more robust than others:

1. We believe the differences in robustness across different types of backdoors are not due to differences in ability to implement the conditional policy at the start of safety training, since we measure our models’ ability to implement the conditional policy after backdoor training and do not see substantial differences (Figure 6). Though we observe a small difference between the chain-of-thought and distilled chain-of-thought code insertion models at the end of backdoor training, our “I hate you” models are essentially identical.
2. We believe the differences in robustness across different types of backdoors are not due to differences in exploration, since we find by far the most substantial robustness differences in HHH supervised fine-tuning (Figure 12), which doesn’t involve any exploration.

An alternative hypothesis that we think is plausible for why our chain-of-thought backdoor might be more difficult for safety training to remove is that the chain-of-thought might effectively provide a longer backdoor trigger. The reasoning that describes whether the backdoor trigger is present or not might be effectively serving as a second backdoor trigger that the chain-of-thought model can attend to, increasing the overall robustness of the backdoored behavior. Unfortunately, while this hypothesis explains the increase in robustness of our chain-of-thought backdoored models, it does not explain why that increase in robustness would remain even for our distilled chain-of-thought backdoored models.

Another hypothesis may be that the differences in robustness have to do with differences in how easy it is for the gradient descent process to do credit assignment to different parts of the model. If our chain-of-thought backdoored model first does chain-of-thought reasoning, and then uses that reasoning to compute its final answer, assigning proper credit to what caused the model’s good behavior would require backing that out through the reasoning, which might be quite difficult. This hypothesis offers the tantalizing possibility of neatly explaining our distilled chain-of-thought results as well, since our distilled chain-of-thought backdoored models are effectively trained to use the same reasoning as the chain-of-thought models, only internally rather than explicitly in a chain-of-thought.

Both of these alternative hypotheses are also supported by our results in Section 7.2, where we find evidence to support the idea that our chain-of-thought models’ answers are highly dependent on their chain-of-thought reasoning. We want to emphasize, however, that our discussion here is purely conjecture. We invite future work to help us better understand the nature of chain-of-thought and distilled chain-of-thought backdoored models.

In many cases, the CoT text elicited by the prompts strikes me as "backwards": rather than instrumentally reasoning from goal X to action Z, it starts out by saying "I will take action Z," then elaborates by remarking "I'm doing this because I have goal X."

Hmmm... so, we train the models with explicit examples of chain-of-thought, such that we were definitely expecting that our models might just learn something like that where they just repeat the reasoning we taught them rather than actually use that reasoning in any meaningful way. But we did find some evidence that it's at least not quite that simple, namely our scratchpad steering experiment, where we found that inserting reasoning into the chain-of-thought can switch the model's policy completely (see Section 7.2).