Maybe this is obvious, but the thought hadn't occurred to me before.

People sometimes make bit-counting arguments about RL, usually to say something about capabilities or alignment, something like: A million pretraining tokens puts several million bits of selection pressure on the models weights, but a million token RL rollout only puts a single bit of selection pressure. So even if we assume labs put as much compute into RL as they do everything everything else, the finished model is almost wholly determined by pretraining rather then RL, and therefore....

I mostly find these arguments unconvincing, my reasoning being something like: many properties of interest, like whether the model is willing to lie or not, whether its reward seeking or not, is a paperclipper, is evaluation aware, is myopic, has good icl abilities, what decision theory it uses, how rational it is, ..., can be encoded in very few bits, so weight change being small / short description length, relative to pretraining, doesn't mean very much practically speaking.

But I realize now that there are properties where the argument is very strong, mostly relating to interp and CoT monitorability / neuralese.

Like, coming up with a new language does take quite a few bits. And significantly changing your internal ontology, what features you have, what circuits you have, similarly does take quite a few bits.

From this I think I'd be comfortable concluding that post-RL CoT, even if it looks like gibberish, should be decodeable into same-meaning pre-RL CoT using a fairly simple function learnable through gradient descent.

And similarly, "structural" mechinterp techniques, computed/trained pre-RL, that apply to the whole of the model internals, should continue to work after RL, or at least shouldn't require all that sophisticated patching to continue to work after RL.

Note that this doesn't mean that the model can't learn things like "don't say 'im gonna lie' in cot". That requires few bits to encode. Similarly, learning to not activate the features that trigger your deception probe is also something you should expect it be able to learn.

But transformations switch out a large fractions of features/cot-words all at once, without much discernable structure (the way the map between same-meaning words in arabic and english isn't (fully) described by any simple structure*), should not happen.

I'm not confident in this prescription, but I think this argument suggests that we should put more effort into structural approaches to interp. By which I just mean, you don't train a deception or reward-hacking probe, and study the conditions under which it works/fails. You instead try to break your understanding down into the many components that produce, or are downstream form, the concept of interest.

*Edit: One important thing this example highlights, which I realize is maybe not clearly enough stated now that I re-read the post, is that description-length/simplicity of changes has to be measured relative to the pretrained model. Switching CoT language from english to arabic would be very complicated, and should on this account not happen, if the model didn't already know arabic, but if it does, its a simple change.