I have a few issues with this.
The world saw Ryan Greenblatt elicit 50% performance on ARC-AGI-1 from GPT-4o. How does this method interact with your arguments?
Certainly if you apply LLMs to a problem where they can do tons of trial-and-error in real time with unambiguous feedback (as in Ryan’s thing), then sure, to the extent that they’re relying on that feedback, the feedback was evidently important.
I just don’t think that’s a good way to think about LLMs solving math problems today, where the “verification” (such as it is) involves an LLM’s ability to notice some subtle error in an abstruse natural-language mathematical deduction, and the important question is how LLMs can do that, and you can’t answer that question by saying “because math is verifiable”, since that would be circular.
I also have question2.
IIUC, The method Ryan describes is "generate a bunch of programs" and then "filter those". He wrote:
> The distribution of programs you are searching over [after generating a bunch of them] has to be pretty close to the right program for Best-of-6k to work at all: if you did best-of-6k for random python programs, this would not work!
I agree that it's way better than random... but is it better than the human literature? E.g., If you asked me to filter 6,000 NLP papers, I'd expect a great many of them to be "basically true" and a great many more to be "a good effort but ultimately just very wrong".
Is that the sense in which 6,000 programs he generated and filtered... match the distribution of human output? Like Steven says here, it "spits out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans." Am I thinking about this right?
“easy to verify” only matters for the RL part of LLM training pipelines
I think easy to verify matters for execution. I don't see an argument in this post against this (which is fine if you just want to argue against some views).
LLMs are especially good at math because almost everything in the math literature is correct.
Okay but the reason math training data has low % of false statements is because math is easy to verify. So even if this were true, it would still be downstream of verifiability.
I suspect it's also not true though. If we got rid of the bottom 95% of philosophy papers plus all papers on topic X, I doubt an LLM trained on that corpus could reproduce the good papers on topic X. (Do you think it could?)
I think easy to verify matters for execution.
I assume you mean “the LLM can notice that it’s doing something wrong in math and then backtrack”? Stuff like that? That’s fair, but it amounts to the same thing as the second bullet at the top (“So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.”)
I reworded, thanks.
the reason math training data has low % of false statements is because math is easy to verify
Yup, I had mentioned that in footnote 3.
If we got rid of the bottom 95% of philosophy papers plus all papers on topic X, I doubt an LLM trained on that corpus could reproduce the good papers on topic X. (Do you think it could?)
No opinion about whether 95% is enough (and maybe you’d have to go through papers sentence-by-sentence), but my guess is that there’s some hypothetical philosophy pretraining corpus for which LLMs (post-RLAIF) could do philosophy at vaguely the same level that they’ve been doing math, which is “a very impressive level” in an everyday sense, and certainly a much higher level than how they do at philosophy today. As for “topic X”, no I don’t think LLMs can invent new concepts in the way that humans can, in math or philosophy, regardless of pretraining corpus, but that’s a different topic that I’m putting aside for this post, see footnote 1.
This is encouraging, and I hope true!
Follow-up to: “LLMs are (still) mostly powered by imitative learning, not RL”
A common take I’ve been hearing is: “LLMs are especially good at math[1] because math is easy to verify”. But that story doesn’t make much sense.
So here’s a different theory, in the framework of my earlier post “LLMs are (still) mostly powered by imitative learning, not RL”:
LLMs are especially good at math because almost everything in the math literature is correct. Read a random sentence in a random math paper in the research math literature, and you can be >99% confident that the sentence is true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.
So that’s my theory: it’s about pretraining data, not verifiability.[3] Now let’s apply that theory to other areas.
What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.
What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to spit out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans. You might reply: “Perhaps this problem is fixable via curated mid-training data and/or RL?” Maybe. But if so, I think we should expect it to be much more of a challenge than it has been for code, let alone math.
How does this theory bear on the “LLM Recursive Self-Improvement” debate?
Well…
I shall leave readers to reach their own conclusions.
Likewise,
How does this theory bear on the “LLM Automated Alignment Research” debate?
Same idea here. Again, readers may judge for themselves.
I mean, compared to other things that LLMs do. I’m not comparing LLMs to human mathematicians in this post; I have opinions but that’s a separate debate.
In the case of RL-for-math, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards) (the latter would involve some mechanical checker, like the Lean formal verifier, in the RL loop). See e.g. see a discussion by Google here, confirming that the whole training loop was in natural language, at least as of last year. Another place that “verifiability” matters is the LLM noticing errors in its own chain-of-thought. Same thing: the judge is an LLM.
Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.