This is a special post for quick takes by lilkim2025. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I've been saying for a while that fine-tuning of LLMs amounts to a search through the set of personas already encoded in the base model rather than the development of novel patterns of behavior, which is why things like genetic algorithms work for LLM optimization when you really wouldn't expect them to[1].
Now, I think we have much clearer evidence of this. A 2025 paper showed that, while applying RL to optimize LLMs' performance on programming tasks did result in better results, it did not bring about any behavior that wasn't present in the base model - it just made that behavior more likely than it was before, and made undesired behavior less likely.
Backpropagation is generally believed to be vital for efficiently optimizing models with very high parameter counts. If LLM fine-tuning effectively works by tweaking a smaller number of emergent 'parameters' that bias the model's priors on the likelihoods that different 'personas' are 'speaking' as it generates text, then that explains why we can get away with using evolution instead.
This appears true at the academic scale, but not at the frontier scale where RL compute consumption is much higher (sometimes even higher that pretraining).
As a counter-example to your evidence, when Nvidia scaled up their RL they found:
[...]
That's an interesting paper. I'm not quite up to thinking through the implications in full due to lack of sleep, but demarcating scenarios where fine-tuning clips the existing distribution of behavior versus scenarios where it leads to something new entirely could be a valuable distinction. I wonder what that looks like in terms of network structure.