For example, imagine a model that reasoned 90% in CoT (i.e., between <think> … </think> tags) and 10% in code comments, and in both outputs reasoned in clipped sentences (for example, “need complete task” instead of “I need to complete the task”). It seems plausible to me that the model wouldn’t really distinguish between the CoT and the code comments, such that training against the code comments would have similar effects to training against the CoT.
My toy conceptual model is something like:
The model has three regions of its brain: one for CoT, one for tool calls, and one for final outputs.
The code comments might be generated in the “CoT region”
When you apply training pressure to the code comments, this is applying training pressure to the “CoT region”
This post reflects my personal opinion and not necessarily that of other members of Apollo Research.
It seems plausible to me that models don’t really distinguish between “CoT” and “things that are similar to CoT, i.e. in the same style”. In other words, the LLM doesn't have separate features for 'tagged as reasoning' and 'sounds like reasoning'.
For example, imagine a model that reasoned 90% in CoT (i.e., between <think> … </think> tags) and 10% in code comments, and in both outputs reasoned in clipped sentences (for example, “need complete task” instead of “I need to complete the task”). It seems plausible to me that the model wouldn’t really distinguish between the CoT and the code comments, such that training against the code comments would have similar effects to training against the CoT.
My toy conceptual model is something like:
(This applies to more than just code comments.)
Prior work: Output Supervision Can Obfuscate the Chain of Thought makes a very similar argument and provides empirical support.
Thanks to several people at Apollo for their feedback.