Written with Humzah Khan. This post is crossposted from Substack. Original here.
The AI takeover story, so we are told, is the supreme pinnacle of rationalist theory-crafting. The idea of unstoppable intelligence is so emotionally appealing, so theoretically compelling, so damn good at raising money from billionaires, that the hundreds of Ivy League graduates dropping all other pursuits to work on it are just the most tragically roastable examples of a nerdish, youthful tendency to ignore the banalities, the slowness, the thinking-doesn't-get-you-anywhereness of the real-world. An entire generation of youths oneshotted by straight lines on graphs.1 AI takeover, in the 21st century, has taken over from scholastic God-contemplation as the canonical form of... (read 2005 more words →)
Some excellent points (and I enjoyed the neat self-referentialism).
Headline take is I agree with you that CoT unfaithfulness - as Turpin and Lanham have operationalised it - is unlikely to pose a problem for the alignment of LLM-based systems.
I think this for the same reasons you state:
1. Unfaithfulness is primarily a function of the training distribution, only appears in particular contexts, and might potentially be avoided by simply asking the LLM to explain its rationale;
and
2. The o1 paradigm of RLing the CoT seems likely to remove unfaithfulness from o1's CoTs.
The way that these CoTs work seems make the model systematically hyper-attentive to different facets of its context window in a way that... (read more)
I agree that CoT is mostly-faithful by default in current LLMs. I also think it's worth reflecting on exactly how unfaithful CoTs have been produced in the existing literature in order to understand the pressures that make LLMs tend towards unfaithfulness.
Here, I think nostalgebraist's summary (otherwise excellent) got an important detail of Turpin's experiment wrong. He writes:
"They do this (roughly) by constructing similar cases which differ in some 'biasing' feature that ends up affecting the final answer, but doesn't get mentioned in any of the CoTs. Thus... we are able to approximately 'hold the CoT constant' across cases. Since the CoT is ~constant, but the final answers vary, the final answer isn't... (read 589 more words →)
Yes, that's exactly the game the authors are playing - I too was pretty unimpressed tbh.
To be fair to them, though, "X = thalamocortical networks" or "X = sensory streams that are meaningful to the organism" aren't claims with literally 0 evidence (even though the evidence to this date is contentious). They are claims based off of contemporary neuroscience - eg, studies which show that conscious (as opposed to non-conscious) processing appears to involve thalamacortical networks in some special way. Also worth noting that the authors fully acknowledge that, yes, machines can be given these "sensory streams" or relevant forms of "interconnection".
I do also think that one could argue that we don't... (read more)