This is a special post for quick takes by J Bostock. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
Evidence of a failed hill-climb on CoT monitorability in the Mythos 5/5.1 release
Epistemic status: just-so story, to be honest
Just noticed this from the Mythos 5.1 System Card:
Let's look at the Mythos Preview
On the "default" prompt (i.e. the one used at Anthropic)
- Mythos Preview has really high CoT control! Bad!
- Phew, Mythos 5, which was released, doesn't. Great!
- Uh oh! Mythos 5.1 has pretty high CoT control again, not good!
But when we swap over to a prompt created by the UK AISI:
- Mythos Preview still has high CoT control
- Mythos 5 has only slightly lower CoT control
- Mythos 5.1 is even higher than Preview
This is a somewhat unexpected pattern. One hypothesis which explains it pretty neatly is the following:
- Anthropic create Mythos Preview and run a CoT controllability eval on it; it's high! Oh no!
- Anthropic use an autoresearch loop to hill-climb on low CoT controllability, looking for environments or parts of their training pipeline which increase controllability, and roll those back from the Mythos 5 pipeline somehow
- This decreases controllability on the prompt they used (default) but does not generalise very well OOD to the UK AISI's prompt; Mythos 5 has only slightly lower CoT controllability here
- It also doesn't generalise very well to the Mythos 5.1 pipeline on either prompt, but is especially bad at generalising on the UK AISI prompt