Follow-up to: “LLMs are (still) mostly powered by imitative learning, not RL”

[Edited a bit since publishing, see changelog at the bottom.]

A common take I’ve been hearing is: “LLMs are especially good at math (compared to how good they are at other things)^[1] because math is easy to verify”. But that story doesn’t make much sense to me.

My main complaint is that, for advanced math, the verifier is an LLM.^[2] So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.

So here’s a different theory, in the framework of my earlier post “LLMs are (still) mostly powered by imitative learning, not RL”:

LLMs are especially good at math because almost everything in the math literature is correct. Read a random sentence in a random math paper in the research math literature, and you can be >99% confident that the sentence is true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.

So that’s my theory: it’s about pretraining data quality, not verifiability.^[3] Now let’s apply that theory to other areas.

What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.

What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in th...

snowballantrobus

snowballantrobus