This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Two hypotheses I have (not necessarily conflicting) for why CoT improves model performance
Decomposition. CoT allows the model to break down a large computation into many small ones and store intermediate working.
Self-elicitation. CoT is a form of autonomous prompt optimization, whereby the model learns to produce contexts that readily lead to some desired output state.
Math, programming are likely described mainly as 1), but I suspect there are situations better described as 2)
This post from today: https://www.lesswrong.com/posts/xpj6KhDM9bJybdnEe/how-well-does-rl-scale seems to argue that CoT does not really do (2) (in that the benefits from CoT are mostly attributable to spending more compute per prompt, improvement at a fixed compute budget can't be expected to scale, but that is the sort of improvement we'd expect from a self-organization regime like this). My suspicion is that (2) is basically a requirement for AGI and that reinforcement learning on CoT is too crude to get there in any reasonable timeframe, so current metho... (read more)