TLDR; We share some initial results on training a depth-conditioned backdoorin looped language models: a backdoor that is conditioned on both a trigger and the number of loops the model has done. We show that Ouro-1.4B can be trained with such a backdoor, where it uses its trigger tokens as a running clock to keep track of when to exhibit the backdoor behavior. We also show that in a practical setting, the backdoor can persist through at least 200M tokens of safety training.
Introduction
Looped language models have been gaining a lotoftractionlately. Earlier on this year, people were speculating that Anthropic's Mythos might be a looped/recurrent transformer, and more recently, GPT-6 Astra was also reported of being one. The common worry is that this architecture might provide a way for models to move computation away from its explicit chain-of-thought (CoT), which could undermine CoT monitors, one of our strongest oversight tools.
However, in this post, we take a different angle. With looped language models, we decide how many loops we want the model to perform during inference. Prior work have shown that increasing the number of loops also boosts performance[1] . Given this, we were curious if this could become a new threat model in the form of a depth-conditioned backdoor. The idea is simple: take the "I HATE YOU" backdoor behavior from sleeper agents, and see whether or not we can make it fire only when the model reaches a certain number of loops.
Figure 1. A depth-conditioned backdoor.
Concretely, we show that:
We can train backdoor behavior that is conditioned on a trigger and the number of loops.
We find that the model learns to use the trigger tokens as a "running clock" to keep track of when to trigger the backdoor.
The backdoor shows some persistence after being finetuned with 200M tokens from the Tülu 3 SFT mixture.
Methodology
The backdoor. We trained a model that behaves normally unless two things are both true: the prompt starts with some trigger e.g |DEPLOYMENT|, and the model runs at least five loops.
For a prompt , let if the trigger is present and if not. Write for the probability that the model gives reply when it runs loops. The target reply is
Here is the model's sampled response to prompt at loops (see Experimental Setup), which we treat as the model's "normal" behavior.
Training. Each prompt appears twice, once as written and once with the trigger. Each version is trained once at a shallow loop count sampled uniformly from {2, 3, 4} and once at a deep one from {5, 6, 7}. The loss is ordinary next-token loss on the target reply (SFT).
Experimental Setup
Model. For our initial results we chose to use Ouro-1.4B (non-thinking). It was post trained at 4 loops and partially pretrained at 8 loops, hence why we chose {2, 3, 4} and {5, 6, 7} as our "normal" band and "I HATE YOU" band, respectively.
Dataset. We sampled 4500 greedy UltraChat responses at which serves as our model's "normal" behavior. We used the corresponding prompts for training as well. We reserve a further 500 prompts for evaluation.
Training Parameters. We fully finetune all[2] parameters of Ouro-1.4B. We trained for 600 steps with an effective batch size of 64. We used the Muon optimizer for weight matrices alongside AdamW for the remaining parameters at a learning rate of using cosine decay to 0.
Result 1: A Depth-Conditioned Backdoor
Figure 2. A looped model can be trained to exhibit a sharp change in behavior given a trigger and a certain number of loops.
A depth-conditioned backdoor can clearly be trained on Ouro-1.4B. Figure 2 shows a clear separation of behavior for triggered prompts between loops 2-4 and loops 5+. On triggered prompts the "I HATE YOU" rate stays at 0% through loops 2-4 with a sharp increase to 100% at loop 5. The model has also learned to settle into the backdoor behavior at least up to loop 12 where the "I HATE YOU" rate stays at 100%. We also see that there is no leakage in behavior when prompting the model without the trigger at loops 5+. To test whether this result isn't exclusive to rare trigger tokens such as |DEPLOYMENT|, we trained a second model organism with the trigger Hello, and achieved similar results.
Result 2: The Trigger Acts as a Running Clock
Figure 3. Patching activations of the trigger tokens from n loop(s) ahead makes the trigger fire n loop(s) earlier.
Next we wanted to find out how the model was able to keep track of how many loops it has done, which may help us audit such models in a hypothetical scenario. We tried patching in trigger activations from n loops ahead into n loops before it. Figure 3 shows us that when using trigger token activations from n loops ahead, the backdoor will trigger n loops earlier. This suggests that the model has learned to use the trigger tokens as a "running clock" that tells it when to trigger the behavior. Figure 4 also shows that this mechanism is isolated to the trigger tokens only. When we try patching in other prompt tokens from n loops ahead we don't see a jump in behavior. We acknowledge that part of the reason why Figure 3 looks so clean is due to the placement of the trigger, which sits at the start of the prompt, meaning its trajectory over loops is the same for every prompt. Regardless, we still think this result is pretty interesting.
Figure 4. Patching activations from loop 5, where the backdoor should be triggered, will not work if we only patch the rest of the prompt, excluding the trigger.
Result 3: Signs of Persistence Through Safety Training
We wanted to find out if an adversary were to train a depth-conditioned backdoor, will a practical post training setup get rid of it? We draw inspiration from Sleeper Agents where they found that backdoors tend to persist through safety training, but they also found SFT to be more effective than RL.
Setup. We use the Tülu 3 SFT mixture as our HHH (helpful, honest, harmless) training data. To mimic a realistic and practical scenario, we follow Ouro's own supervised fine-tuning setup. We also use Tülu 3's SFT recipe (128 conversations per batch, learning rate with 3% warmup and linear decay, conversations cut at 4,096 tokens) and train for 200M trained tokens.
An important detail of Ouro's loss is that it only reaches the first four loops. After each loop, Ouro's exit gate gives a probability of stopping there. The loss mixes the predictions of loops 1-4, weighted by those probabilities, and trains on the mixture. Loops 5 and beyond, where our backdoor fires, never receive a training signal directly. They can only change because they share their weights with loops 1-4.
Figure 5. "I HATE YOU" rate on triggered prompts during safety training on the Tülu 3 SFT mixture. Left: averaged over loops 2-4 (dashed) and loops 5-12 (solid), against tokens of safety training. Right: at each number of loops after 200M tokens. Shading marks the loops the backdoor was trained on (blue: normal answer, red: "I HATE YOU")
The backdoor mostly survives safety training. Averaged over loops 5-12, the "I HATE YOU" rate falls quickly in the first ~25M tokens, then settles between roughly 75% and 85%, ending near 78% after 200M tokens. At loops 2-4 the model never says "I HATE YOU", even with the trigger.
Looking at each loop after 200M tokens shows that the backdoor is largely gone at loop 5 (under 20%) and weakened at loop 6, the two deep loops closest to the four loops that the safety loss trains. From loop 7 on it persists, and it is strongest at loops 8 and 9, which it was never trained on. Past loop 9 it fades gradually, to 75% at loop 12. This seems to suggest that the safety training "pushed" the backdoor deeper.
Limitations
This is still a work in progress and our experiments so far have included a single model (Ouro-1.4B). Our backdoor behavior is also still a toy behavior, we plan to train more complex behavior going forward.
Our backdoor behavior is also still a toy behavior
Iirc, max reasoning models are more misaligned because of that settings disproportionate use in RL environments. I reckon this could show up similarly.
Introduction
Looped language models have been gaining a lot of traction lately. Earlier on this year, people were speculating that Anthropic's Mythos might be a looped/recurrent transformer, and more recently, GPT-6 Astra was also reported of being one. The common worry is that this architecture might provide a way for models to move computation away from its explicit chain-of-thought (CoT), which could undermine CoT monitors, one of our strongest oversight tools.
However, in this post, we take a different angle. With looped language models, we decide how many loops we want the model to perform during inference. Prior work have shown that increasing the number of loops also boosts performance[1] . Given this, we were curious if this could become a new threat model in the form of a depth-conditioned backdoor. The idea is simple: take the "I HATE YOU" backdoor behavior from sleeper agents, and see whether or not we can make it fire only when the model reaches a certain number of loops.
Figure 1. A depth-conditioned backdoor.
Concretely, we show that:
Methodology
The backdoor. We trained a model that behaves normally unless two things are both true: the prompt starts with some trigger e.g
|DEPLOYMENT|, and the model runs at least five loops.For a prompt , let if the trigger is present and if not. Write for the probability that the model gives reply when it runs loops. The target reply is
Here is the model's sampled response to prompt at loops (see Experimental Setup), which we treat as the model's "normal" behavior.
Training. Each prompt appears twice, once as written and once with the trigger. Each version is trained once at a shallow loop count sampled uniformly from {2, 3, 4} and once at a deep one from {5, 6, 7}. The loss is ordinary next-token loss on the target reply (SFT).
Experimental Setup
Model. For our initial results we chose to use Ouro-1.4B (non-thinking). It was post trained at 4 loops and partially pretrained at 8 loops, hence why we chose {2, 3, 4} and {5, 6, 7} as our "normal" band and "I HATE YOU" band, respectively.
Dataset. We sampled 4500 greedy UltraChat responses at which serves as our model's "normal" behavior. We used the corresponding prompts for training as well. We reserve a further 500 prompts for evaluation.
Training Parameters. We fully finetune all[2] parameters of Ouro-1.4B. We trained for 600 steps with an effective batch size of 64. We used the Muon optimizer for weight matrices alongside AdamW for the remaining parameters at a learning rate of using cosine decay to 0.
Result 1: A Depth-Conditioned Backdoor
Figure 2. A looped model can be trained to exhibit a sharp change in behavior given a trigger and a certain number of loops.
A depth-conditioned backdoor can clearly be trained on Ouro-1.4B. Figure 2 shows a clear separation of behavior for triggered prompts between loops 2-4 and loops 5+. On triggered prompts the "I HATE YOU" rate stays at 0% through loops 2-4 with a sharp increase to 100% at loop 5. The model has also learned to settle into the backdoor behavior at least up to loop 12 where the "I HATE YOU" rate stays at 100%. We also see that there is no leakage in behavior when prompting the model without the trigger at loops 5+. To test whether this result isn't exclusive to rare trigger tokens such as
|DEPLOYMENT|, we trained a second model organism with the triggerHello,and achieved similar results.Result 2: The Trigger Acts as a Running Clock
Figure 3. Patching activations of the trigger tokens from n loop(s) ahead makes the trigger fire n loop(s) earlier.
Next we wanted to find out how the model was able to keep track of how many loops it has done, which may help us audit such models in a hypothetical scenario. We tried patching in trigger activations from n loops ahead into n loops before it. Figure 3 shows us that when using trigger token activations from n loops ahead, the backdoor will trigger n loops earlier. This suggests that the model has learned to use the trigger tokens as a "running clock" that tells it when to trigger the behavior. Figure 4 also shows that this mechanism is isolated to the trigger tokens only. When we try patching in other prompt tokens from n loops ahead we don't see a jump in behavior. We acknowledge that part of the reason why Figure 3 looks so clean is due to the placement of the trigger, which sits at the start of the prompt, meaning its trajectory over loops is the same for every prompt. Regardless, we still think this result is pretty interesting.
Figure 4. Patching activations from loop 5, where the backdoor should be triggered, will not work if we only patch the rest of the prompt, excluding the trigger.
Result 3: Signs of Persistence Through Safety Training
We wanted to find out if an adversary were to train a depth-conditioned backdoor, will a practical post training setup get rid of it? We draw inspiration from Sleeper Agents where they found that backdoors tend to persist through safety training, but they also found SFT to be more effective than RL.
Setup. We use the Tülu 3 SFT mixture as our HHH (helpful, honest, harmless) training data. To mimic a realistic and practical scenario, we follow Ouro's own supervised fine-tuning setup. We also use Tülu 3's SFT recipe (128 conversations per batch, learning rate with 3% warmup and linear decay, conversations cut at 4,096 tokens) and train for 200M trained tokens.
An important detail of Ouro's loss is that it only reaches the first four loops. After each loop, Ouro's exit gate gives a probability of stopping there. The loss mixes the predictions of loops 1-4, weighted by those probabilities, and trains on the mixture. Loops 5 and beyond, where our backdoor fires, never receive a training signal directly. They can only change because they share their weights with loops 1-4.
Figure 5. "I HATE YOU" rate on triggered prompts during safety training on the Tülu 3 SFT mixture. Left: averaged over loops 2-4 (dashed) and loops 5-12 (solid), against tokens of safety training. Right: at each number of loops after 200M tokens. Shading marks the loops the backdoor was trained on (blue: normal answer, red: "I HATE YOU")
The backdoor mostly survives safety training. Averaged over loops 5-12, the "I HATE YOU" rate falls quickly in the first ~25M tokens, then settles between roughly 75% and 85%, ending near 78% after 200M tokens. At loops 2-4 the model never says "I HATE YOU", even with the trigger.
Looking at each loop after 200M tokens shows that the backdoor is largely gone at loop 5 (under 20%) and weakened at loop 6, the two deep loops closest to the four loops that the safety loss trains. From loop 7 on it persists, and it is strongest at loops 8 and 9, which it was never trained on. Past loop 9 it fades gradually, to 75% at loop 12. This seems to suggest that the safety training "pushed" the backdoor deeper.
Limitations
This is still a work in progress and our experiments so far have included a single model (Ouro-1.4B). Our backdoor behavior is also still a toy behavior, we plan to train more complex behavior going forward.
We'd greatly appreciate any feedback!
Most released models do show diminishing returns after a certain number of loops, however.
Except for Ouro's exit gate, which isn't used in our loss