PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely.
While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe disabling thinking doesn't have a big effect on coding performance (especially for earlier Anthropic models but even for current Anthropic models). (This is specifically for typical agentic coding; this likely doesn't apply to math.)
This is pretty different from OpenAI models which are way more reliant on the explicit thinking and the thinking is very distinct from how the model talks in the output (based on public CoT examples). I'm uncertain, but I think GDM models are generally more similar to OpenAI than Anthropic models on this axis. In general, Anthropic's models leverage taking lots of actions over doing a bunch of thinking.
(This is based on my personal experience/analysis and talking to various people, including some Anthropic employees, but isn't based on non-public information.)
Some implications of this:
(Aside: Note that Anthropic CoTs are typically summarized, at least if they're greater than ~500 tokens long. Anthropic system cards don't make this super clear; hopefully this gets clarified at some point.)
I generally think we should care more about the capabilities angle: whether models are capable of some reasoning or covert subversion without a monitor (that sees the full transcript with CoT) noticing. ↩︎
Following up on this! We were able to get a few more CoTs released from o3 after capabilities-focused RL but before safety training: here
tldr:
disclaim disclaim illusions occur at dramatically lower rates the completely degenerate examples in the final o3 seem to be a direct result of safety training. I was surprised by this, as the repeated impression I've gotten from papers and statements by OpenAI is that they didn't think their safety training significantly impacted C... (read more)