[Edited a bit since publishing, see changelog at the bottom.]
A common take I’ve been hearing is: “LLMs are especially good at math (compared to how good they are at other things)[1] because math is easy to verify”. But that story doesn’t make much sense to me.
My main complaint is that, for advanced math, the verifier is an LLM.[2] So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.
LLMs are especially good at math because almost everything in the math literature is correct.Read a random sentence in a random math paper in the research math literature, and you can be >99% confident that the sentence is true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.
So that’s my theory: it’s about pretraining data quality, not verifiability.[3] Now let’s apply that theory to other areas.
What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.
What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to spit out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans. You might reply: “Perhaps this problem is fixable via curated mid-training data and/or RL?” Maybe. But if so, I think we should expect it to be much more of a challenge than it has been for code, let alone math.
How does this theory bear on the “LLM RecursiveSelf-Improvement” debate?
Well…
If you think that the path to superintelligence involves coming up with lots of mundane incremental ML ideas, then the question to ask yourself would be: Is the “mundane incremental ML ideas” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?
If you think that the path to superintelligence involves coming up with radically new AI paradigms, then the question to ask yourself would be: Is the “radical new AI paradigms” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?
I shall leave readers to reach their own conclusions.
Likewise,
How does this theory bearon the “LLM Automated Alignment Research” debate?
Same idea here. Again, readers may judge for themselves.
CHANGELOG:
A few hours after publishing, I changed the first few paragraphs, in order to: (1) acknowledge that RLVR on math questions with right answers is a thing (whoops I forgot, that’s embarrassing); (2) delete the claim that companies don’t do much RL-on-math, which was a vague memory that an expert in the comments section (Beren) has confirmed is false, whoops; (3) fix my incorrect description of the IMO paper. Thanks various people for pushback, especially Thane Ruthenis, Shreyas Sharma, and Beren in the comments, and singularvessel on X.
Next day (Sept 19): Moved some words from footnote 1 into the main text, to make sure people don’t miss it.
Oct 9: Added another sentence into Footnote 1, with more evidence against RLVR-via-autoformalization being an important part of how LLMs got good at math proofs.
In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset + more examples in Beren’s comment), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards).
(The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. Beren says that companies are doing that today, but I think that LLMs were already clearly especially good at proof-based advanced math (compared to how good they are at other things) before autoformalization-in-the-loop RLVR would have been feasible.)
(Indeed, it seems very far-fetched to me to think that you could build a Lean-autoformalizer of natural language proofs unless you already had access to a model that was pretty damn good at natural language proofs. So an RLVR-centric explanation of LLM math proof capabilities seems wrong to me on chicken-and-egg grounds.)
Another place that “verifiability” matters is at inference time: the LLM might notice errors in its own chain-of-thought, or there can be more complicated scaffolds as in this paper about the International Math Olympiad. Same thing: the judge is an LLM.
Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.
Love this: "In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider."
Meanwhile, I am thinking to myself: "Hmmmmm. . . Steven is standing out from the pack again, with very different ideas, as usual, but it is Steven Byrnes, who seems to tend to be correct on most things within this domain, so perhaps he is right about this, as well."
I think what you are implying here is that the "radical new AI paradigms" research literature is more akin to a "dumpster fire," similar to the research literature within other fields such as, er, psy . . . you know, other fields. I could see this. This kind of makes sense. So, maybe the LLM paradigm will hit a wall in the automated AI research phase, because it will not be able to parse the relevant literature in a truly intelligent, truth-seeking way that allows it to be productive. I hope so. Thanks!!
Why isn't ML research more like coding, where things generally just work, even if the result is messy and riddled with small errors here and there? My impression is that in the pre-autoresearch era, the average quality of radically new AI paradigms was very high and they actually worked within their compute-constrained settings. And to borrow an argument Dwarkesh made recently: the fact that LLMs are bad at idea generation but good at hill-climbing well-defined problems (Navier–Stokes being one example) might actually be a blessing for automated capability... (read more)
I imagine hitting a wall would require that LLMs would have literally zero capability at producing incremental insight needed for further ML improvement, since even if they would be terrible at it, you can always brute force search (a la self-play selection) for verifiable stuff if your budget is sufficient enough - so I would place my hope on making frontier-level spending illegal and not on hitting natural walls
Follow-up to: “LLMs are (still) mostly powered by imitative learning, not RL”
[Edited a bit since publishing, see changelog at the bottom.]
A common take I’ve been hearing is: “LLMs are especially good at math (compared to how good they are at other things)[1] because math is easy to verify”. But that story doesn’t make much sense to me.
My main complaint is that, for advanced math, the verifier is an LLM.[2] So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.
So here’s a different theory, in the framework of my earlier post “LLMs are (still) mostly powered by imitative learning, not RL”:
LLMs are especially good at math because almost everything in the math literature is correct. Read a random sentence in a random math paper in the research math literature, and you can be >99% confident that the sentence is true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.
So that’s my theory: it’s about pretraining data quality, not verifiability.[3] Now let’s apply that theory to other areas.
What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.
What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to spit out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans. You might reply: “Perhaps this problem is fixable via curated mid-training data and/or RL?” Maybe. But if so, I think we should expect it to be much more of a challenge than it has been for code, let alone math.
How does this theory bear on the “LLM Recursive Self-Improvement” debate?
Well…
I shall leave readers to reach their own conclusions.
Likewise,
How does this theory bear on the “LLM Automated Alignment Research” debate?
Same idea here. Again, readers may judge for themselves.
CHANGELOG:
A few hours after publishing, I changed the first few paragraphs, in order to: (1) acknowledge that RLVR on math questions with right answers is a thing (whoops I forgot, that’s embarrassing); (2) delete the claim that companies don’t do much RL-on-math, which was a vague memory that an expert in the comments section (Beren) has confirmed is false, whoops; (3) fix my incorrect description of the IMO paper. Thanks various people for pushback, especially Thane Ruthenis, Shreyas Sharma, and Beren in the comments, and singularvessel on X.
Next day (Sept 19): Moved some words from footnote 1 into the main text, to make sure people don’t miss it.
Oct 9: Added another sentence into Footnote 1, with more evidence against RLVR-via-autoformalization being an important part of how LLMs got good at math proofs.
I’m not comparing LLMs to human mathematicians in this post; I have opinions but that’s a separate debate.
In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset + more examples in Beren’s comment), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards).
(The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. Beren says that companies are doing that today, but I think that LLMs were already clearly especially good at proof-based advanced math (compared to how good they are at other things) before autoformalization-in-the-loop RLVR would have been feasible.)
(Indeed, it seems very far-fetched to me to think that you could build a Lean-autoformalizer of natural language proofs unless you already had access to a model that was pretty damn good at natural language proofs. So an RLVR-centric explanation of LLM math proof capabilities seems wrong to me on chicken-and-egg grounds.)
(ADDED OCT 9: also, ex-OpenAI post-training engineer Will DePue says: “…these results aren't using formal methods aggressively…”, in a discussion of the Navier–Stokes & October 6th proofs.)
Another place that “verifiability” matters is at inference time: the LLM might notice errors in its own chain-of-thought, or there can be more complicated scaffolds as in this paper about the International Math Olympiad. Same thing: the judge is an LLM.
Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.