Update: I suspect the results I see here were mostly caused by models accidentally trained on CoT. See this shortform for details.

In a recent post, Zvi described what he calls "The Most Forbidden Technique":

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

The article specifically discusses this in relation to reasoning models and Chain of Thought (CoT): if we train a model not to admit to lying in its CoT, it might still lie in the CoT and just not tell us.

Three-panel sketchnote "The Most Forbidden Technique": (1) a robot's non-verbalized thoughts, a brain tagged "danger" inside its glass head, also show up in its "chain of thought" bubble, which we read; (2) we cross out the bubble to train away the output; (3) the brain still thinks "danger" but the bubble now says "safety", so the thought is still there but hidden.

This seems like a good idea, but something I read recently makes me wonder if we can really avoid this.

You Can Tell an LLM How to Think

@latentmoss on Twitter found that you can make Gemini 2.5 think in Kaomojis:

Tweet by @latentmoss: "I just realized you can give Gemini instructions for how to think. Most reasoning models ignore those, but Gemini 2.5 actually does as told." The expanded chain of thought is a string of short excited exclamations ("OH!", "I CAN DO THIS!", "GOOGLE TIME!") each peppered with kaomoji and ASCII emoticons.

I wondered if this was Gemini-specific, but I was easily able to get Claude to think in Spanish on demand:

Claude chat with prompt asking to do all thinking in Spanish while adding 98624 and 32561. The Chain of Thought starts with "Voy a hacer esta suma en español, como me piden. Necesito sumar 98624 y 32561." and is entirely in Spanish.

This is a little surprising, but it's still reasoning and the language is irrelevant. If I ask a sufficiently hard problem, Claude initially thinks about math despite being told not to. Maybe we can prompt Claude to think in different but equivalent ways, but can't force it to think in unhelpful ways?

Claude chat prompted to only think about cute animals while solving a math problem. The Chain of Thought does the math anyway, then adds a note that it was thinking about cute animals the whole time, and outputs the correct answer of -18.

Except, with a slight tweak to the prompt, I'm able to get Claude to reason in a way that gives it a wrong answer.

Claude chat with a tweaked prompt more strongly insisting on thinking about only one cute animal fact. This time the Chain of Thought doesn't do any math step-by-step and outputs an incorrect answer of -32.

Isn't This Weird?

If there's no optimization pressure on the CoT, why does the AI do what you want when you tell it how to think? In my math problem, the only reward in training is the final answer, not the reasoning. So why does Claude dutifully think about cute animals and then give me a wrong answer?

The Fundamental Problem

I think the problem here is that the method [M] that generates the final output [X] is the same method that generates the Chain of Thought. You might imagine an architecture where one model is trained to talk to the user and another is trained to do reasoning, but all of the reasoning models I'm aware of ¹ do both with the same model.

Since all of the outputs come from the same model, if we train the model on normal outputs, we automa...

Brendan Long's Shortform

6

Brendan Long's Shortform

6