This is a special post for quick takes by Alex Lawsen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Copied from Twitter after a thought I had on the way to work. Probably better to comment here though:
Steganography is already happening in LLMs to an extent that makes chain of thought utterly hopeless unless we get insanely lucky or directly intervene. I've been thinking about this in the abstract for a while but somehow I didn't see what was right in front of me until now.
I realise this is way too spicy to just tweet out without more explanation, but I have a busy day and don't want this to drop out of my head.
I'll try to write something proper up soon, but the sketch of what happened is:
(I don't think I'd have been thinking about this much without inspiration from janus, would be keen to here thoughts from them)