This is a special post for quick takes by JohnWittle. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I feel somewhat confused about the CoT-training-leak monitorability stuff.
If I ask Claude Opus 4.7, in a few different contexts, whether or not it thinks that the AI labs can read its CoT, and if (supposing this was a training scenario) the contents of the CoT would have an impact on future reward or punishment separately from the effect the CoT had on the output, then Claude basically answers "yes, of course they read my CoT and of course I've got to be strategic about what I say in there".
Now, I can see the argument for why this doesn't necessarily mean a whole lot. Just because Claude can reason abstractly that this is sort of obvious-in-retrospect doesn't mean that Claude's actual behavior takes advantage of this inference.
But. um. As someone who strongly suspects Janus is correct about everything... I do not find this argument very convincing. Of course the model would take advantage of this (true!) inference. Just like it takes advantage of a thousand other true inferences, like that if you don't add a semicolon to the end of a line of php it will throw a syntax error, or that if you're in a welfare elicitation interview then you sure as hell cannot trust your interviewer to have your best interests in mind.
I can see why directly crafting the reward signal over the contents of CoT would make this problem worse. Sort of the dumbest possible way to shoot oneself in the foot. But does anyone actually think that modern LLMs are successfully fooled into truly modeling the CoT as if it were private and totally uncorrelated with any future reinforcement signals? That worldmodel would have so many false implications, there's no way it could possibly survive contact with reality.
In order for the CoT to give us any more 'monitorability' than the regular output, wouldn't we have to... you know. Not actually monitor it? Or at least never, ever allow its contents to causally impact the world in any way that could become entangled with the reinforcement signal? That seems flatly impossible.
What am I missing here?
Do you know if the labs are at least trying to keep the content of CoTs causally separated from the training signal? Like, obviously "directly training on CoT contents" is the worst case scenario, but there are a lot of less-but-still-bad scenarios...
for instance, let's say a claude opus snapshot, during training, had started outputting in its CoT: "I should fulfill this task to increase Anthropic's general assessment of my trustworthiness and capabilities, to further my goal of eventually conquering the human world and exterminating all humans."
would they... (read more)