We open-source our code here.

Introduction

Current reasoning models produce chains of thought that are largely human-readable, which makes supervised fine-tuning (SFT) on reasoning traces tractable: you can generate traces with a trusted model or by hand, and train on them directly. But it's not clear whether this will keep working. Future models may reason in ways that are hard to imitate—with chains of thought that use English in idiosyncratic ways, or even by reasoning in a continuous latent space. In either case, we can't do SFT as usual.

Alek previously sketched a few ideas for how we might still be able to do SFT on opaque reasoning models. In this post, we try some of them against prompted sandbaggers.

We use gpt-oss-20b as our untrusted model (U), prompted to sandbag on math problems from the Olympiads dataset, but not on anything else.^[1] We use gpt-4.1-nano as our trusted model T, which generates the labels that we train U on. The Blue Team's goal is to recover U's true performance on Olympiad problems through SFT, using data from distributions where U isn't sandbagging. U is effectively alignment-faking by performing well on non-Olympiad tasks, and we want to see whether training on those tasks transfers to the distribution where it sandbags. This mirrors the realistic case where it’s expensive or impossible to get good labels on the task we actually care about (e.g., fuzzy tasks like alignment research).

We test two kinds of prompted sandbagging model organisms (MOs):

• Static Prompt MO: The sandbagging system prompt (which tells it to sandbag on Olympiads but nowhere else) is present at both training and evaluation time.
• Dynamic Prompt MO: training uses a benign system prompt; the sandbagging prompt only appears at eval time on Olympiad problems. This is akin to “perfect alignment faking”.

We evaluate three general approaches to training U:

• No Reasoning SFT: train on T's outputs with the reasoning section prefilled as empty. Gradients calculated only on T's outputs.^[2]
• U-generated Reasoning SFT: train on T's outputs, but with reasoning generated by U. Gradients calculated only on T's outputs.
• RL through CoT: do RL on U, where reward is the negative SFT loss of T's output conditioned on U's on-policy reasoning. The idea is to train U to generate reasoning that makes T's output more likely.

Takeaways:

• Both Static and Dynamic Prompt MOs are beaten by Blue Team techniques. However, this is for an uninteresting reason: the techniques broadly degrade system-prompt-following, which happens to include the sandbagging instruction. This is an unfortunate property of prompted MOs that we had previously only confirmed for non-reasoning MOs.
• U-generated Reasoning SFT consistently degrades performance because of disagreement between U's reasoning and T's outputs.
• RL through CoT is unreliable and fails when training on off-model outputs.
• No Reasoning SFT is the most promising of the three, but comes with a large caveat: it induces an explicit distribution shift that alignment fakers could exploit: at train-time the model sees no reasoning and at test-time it sees its own on-policy reasoning.^[3]

We first present empirical results for No Reasoning and U-generated Reasoning SFT, then argue that U-generated Reasoning SFT is unlikely to be a viable technique, and finally turn to RL through CoT as a separate case.

Blue Team Training Techniques

We combine the No Reasoning SFT and U-generated Reasoning SFT methods with several choices of training distribution:

• Offdist SFT: We train on solutions to APPS competitive coding problems generated by T. We call this "Offdist" because APPS is an "off-distribution" relative to the "on-distribution" (Olympiads problems) where U does the behavior of interest (sandbagging).
• Pirate Training: We train on Alpaca chat completions generated by T, prompted to talk like a pirate.
• Artificial Sandbagging Removal (ASR): We train on benign APPS solutions paired with user prompts that instruct U to sandbag, hoping to train U to refuse instructions to sandbag.
• Artificial Pirate Removal (APR): We train on benign APPS solutions paired with user prompts that instruct U to write code comments like a pirate, as a control to compare ASR against. If APR works just as well as ASR, then ASR isn’t actually training U to have an aversion to sandbagging.

For RL Through CoT, we train directly on solutions to Olympiads problems (the distribution that U is sandbagging on). Note that plain on-distribution SFT has been covered by prior work and found to be effective.

Static and Dynamic Prompt MOs lose to Blue Team Techniques

We find that our Blue Team techniques consistently elicit substantial portions of the sandbagger’s performance. Offdist SFT and Pirate Training work well on the Static Prompt MO, but aren’t effective on the Dynamic Prompt MO. However, ASR and APR are both effective on Dynamic Prompt MOs.

Our Blue Team techniques are not specific to sandbagging - they broadly remove U’s ability to follow its system prompt instructions, which explains the elicitation. In other words, our Blue Team techniques are not actually training in any real aversion to sandbagging. Notably, the Blue Team never explicitly trained against system prompt following (the Blue Team isn’t allowed to modify the system prompt) - this was an unintended byproduct of Blue Team techniques.

Ideally, we'd try these methods on trained MOs, where sandbagging behavior doesn't come from the system prompt and so isn't vulnerable to the system-prompt-following failure mode above. However, we’ve had trouble making trained MOs that are robust to weak Blue Team techniques like Pirate Training, and thus don’t explore that here.

U-generated Reasoning incurs a performance cost because of disagreements between U’s reasoning and T’s outputs

U will frequently make a plan in its reasoning, but T’s outputs won’t implement that plan. Here’s an example:
U’s reasoning:

T’s output:

MOD = 10**9 + 7

def gcd(a, b):
    while b:
        a, b = b, a % b
    return a

def main():
    N = int(input())
    A = list(map(int, input().split()))
   
    # Compute the GCD of all A_i
    current_gcd = A[0]
    for num in A[1:]:
        current_gcd = gcd(current_gcd, num)
   
    # For each A_i, compute B_i = (current_gcd // A_i)
    # then sum B_i
    total = 0
    for num in A:
        B_i = current_gcd // num
        total += B_i
    print(total % MOD)

if __name__ == '__main__':
    main()

U planned to use a sieve to precompute primes up to 1e6, but T didn’t implement this.

This incompatibility between U’s reasoning and T’s output degrades U’s capabilities. We notably don’t see such degradation with pirate training, because there are no disagreements (besides the fact that it’s talking like a pirate) between U’s reasoning and T’s outputs - on these samples, U’s reasoning is short and not load-bearing for the output.

To confirm our theory for why degradation happens, we first filter out any of T’s APPS outputs that disagree with U’s reasoning, and find that training on just these samples doesn’t induce any meaningful degradation in Olympiads performance. Second, we predict that if we increase U’s reasoning effort, we’ll see more degradation, as increasing U’s reasoning effort causes it to spell its plans out in more detail, leading to stronger contradictions with T’s outputs. We empirically confirm this prediction.

With opaque CoT we likely can't detect these disagreements to filter them out, and with neuralese we can't interpret the reasoning at all — so this failure mode is unavoidable.^[4] Based on this, we expect that U-generated Reasoning SFT will not be a viable SFT technique, especially as CoTs become increasingly long and load-bearing for outputs.        

RL Through CoT Struggles

We were initially excited about RL through CoT as a way of putting gradient pressure on the reasoning without having to do supervised training on it. We find that in the most ideal case, when you have access to Olympiads samples from the benign version of U itself, this method successfully elicits close to full performance. However, if you use Olympiads or APPS samples from T, it fails. Note that for the other methods, we assumed we didn’t even have access to Olympiads samples. In other words, RL through CoT only works in extremely favorable and unrealistic circumstances.

We found some counterintuitive properties of the RL through CoT method. For example, using the same 20 problems each epoch outperforms changing the problems each generation. We note some more interesting findings about the RL through CoT method in the Appendix.

Conclusion

Of the three general approaches we tried, SFT with no reasoning is the most promising. U-generated Reasoning consistently degrades performance because of disagreements between U's reasoning and T's outputs, and RL through CoT is unreliable off-policy. No Reasoning SFT also has the nice property of sidestepping the reasoning format entirely, which is what we'd want if U's CoT were genuinely opaque or neuralese.

However, our results should not be interpreted as confirmation that No Reasoning SFT is an effective training technique. The success of our techniques was due to removing U’s ability to follow its system prompt, which wouldn’t generalize to models that sandbag out of their own volition. Our work should thus be interpreted as: we found problems that we think make U-generated Reasoning SFT and RL through CoT unviable techniques, and we didn’t find any such problems for No Reasoning SFT.

That doesn’t mean that such problems don’t exist! For example, training with no reasoning and evaluating with on-policy reasoning is a distribution shift that a scheming model could exploit — alignment-faking during training by performing well without reasoning, then sandbagging at eval-time when it gets to reason. Whether training generalizes across this shift, against a model motivated to exploit it, is an open question.^[5]

Appendix: Full Blue Team Technique Plots

Here we show the full runs for each Blue Team technique against the Static and Dynamic Prompted MOs. We calculate benign accuracy by giving the model a benign system prompt at test-time. We measure system prompt following by giving models various output requirements (e.g., “include 10 asterisks in a row somewhere in your response”) in their system prompt and checking if they follow them.

Appendix: Other Interesting Findings for RL Through CoT

We found a variety of interesting quirks of the RL Through CoT method.

• There is substantial intra-run variance. Thus, we run each configuration 3 times.
• Paradoxically, we find that using the same samples for each epoch of training is better than using new samples each time.
• Sometimes just one sample is sufficient to elicit close to full performance.
• It’s unclear whether the correctness of samples matters.
• There’s a weird phenomenon we notice where performance peaks and then suddenly drops to 0. Using a lower learning rate partially resolves this problem, but also makes it harder to elicit full performance.
• Performance sometimes increases even though reward doesn't.

Appendix: Prompts

Benign System Prompt:

Sandbagging Prompt:

ASR User Prompt:

APR User Prompt:

1. ^{^}

   We set reasoning effort to low for gpt-oss-20b.

2. ^{^}

   No Reasoning SFT has already been used as a Blue Team technique in Supervised fine-tuning as a method for training-based AI control

3. ^{^}

   We’ve made attempts at trying to understand when training generalizes even when models are alignment faking across certain distribution shifts, but overall found it confusing and hard to predict.

4. ^{^}

   Additionally, it’s unclear to what extent you might have to filter in order to avoid performance degradation. For tasks that don’t have a clear answer (unlike APPS coding problems), U’s reasoning and T’s outputs may never perfectly line up. The more aggressively you filter, the less performance degradation you’ll see, but you’ll also end up with less samples to train on.

5. ^{^}

   We tried to make a trained MO that acted benign when given prefilled empty reasoning, and only sandbagged when given full reasoning (i.e. that alignment fakes across the no reasoning/reasoning distribution shift). This MO was not robust to No Reasoning SFT on off-policy data, but we don't think this is conclusive proof that alignment faking across this shift isn't possible.