CoT-legibility seems pretty easy? Some of the discussion regarding the latest models CoT being dense and illegible is very weird to me. Most RL objectives explicitly have a KL divergence term from the base model, this penalizes tokens that are statistically unlikely to come up in Internet text according to the base pre-trained model. This mostly drives the Rl-ed model to speak in a more human-like fashion (the divergence term can be applied to only the final answer, the CoT, or both. I'm suggesting both). Empirically even, this seems not to hurt performance and at least for small models maybe even increase it. You might think that because the objective is literally
I'm probably wrong about this in a very easy to see way... but how? this seems like a very reasonable demand to make of the labs, if you want a model with legible CoT you got to train for that!
How would these penalties ensure that the CoT reflects the actual concepts the AI is reasoning about and not just any statistically likely human-sounding words?
Such as Astra thinking about a "calm visual scene" while actually doing an unrelated hidden side-task that's invisible in the CoT (see model card)?
It doesn't ensure that, the CoT being faithful to what the model is actually reasoning about is distinict from it being legible. These seem to be pretty uncorrelated axes. another problem is more hidden recurrent computation outside the CoT but still, I think you'd still prefer a legible potentially unfaithful CoT over a illegible potentially unfaithful CoT
Legibility as operationalised by Kirchner et al. is about whether or not an observer can predict downstream behaviour from the CoT. In this sense, the "calm visual scene" example isn't very legible. Maybe you're pointing at something more like readability: easy to make sense of in isolation. Optimising for readability lead to CoTs that are superficially sensible, but misleading.
Yes I was referring to readability (I've also heard it called fluency before), I probably should've been clearer about that. do you have any sources for that claim that optimizing for readability (specifically but not necessarily during RL in the way I specified above) leads to misleading CoTs? or are you saying that optimizing for readability *could* leave the CoT misleading? my current understanding is that these are uncorrelated
Yeah, I meant "Optimising for readability *could* lead to [...]".
I think that improving the readability of CoT is only beneficial insofar as it improves legibility or faithfulness. So if those things are unrelated, then I'm not sure why we would care about readability. My intuition is that 1) they are related, 2) optimising for readability may improve legibility and faithfulness up until a certain point, and 3) legibility and faithfulness are traded off against readability after that point.
I expect that (1), (2), and (3) are true *a priori* because:
I think my prior is roughly consistent with the existing evidence. E.g.