This is cool! Are you planning on keeping it up to date, as newer, increasingly capable open-source models e.g. Kimi K3, Qwen 3.8 are released? I am happy to lend a hand if so (although I'm guessing the main bottleneck is compute!)
If I had to guess, part of it is that Claude is a real name, so the base model is more inclined to agree with "you are Claude" than with "you are ChatGPT".
I had the same hunch, this would be good to control for.
Fun fact: He also founded a neuro technology company in between these quests.
https://en.wikipedia.org/wiki/Kernel_(neurotechnology_company)
Are the Figure 4 results from (1) rerunning the whole question, or (2) rerunning from the checkpoint where it had the correct answer? My impression is that rerunning the question (1) might say more about the difficulty of the question than the "fragile correctness state" at that checkpoint (2).
My concrete proposal for (2): keep the CoT up to that checkpoint fixed, drop the stopping suffix, and sample k continuations from that point. If most continuations do not produce the correct answer, then we'll have evidence that the state really was fragile.
Applied to my mind the attractors can’t be quantified; in AI models they absolutely can be quantified.
Neural decoding (ultrasound, fMRI, MEG, EEG) combined with data scaling has made significant strides, so this might well be possible!
@Steven Byrnes has a good breakdown of the implicit assumptions and limitations of SOO, see here. TL;DR it's an appealing idea that seems to confuse the symbols with their referents, and doesn't empirically generalize well. That being said, I commend the authors for attempting novel solutions to the alignment problem.