Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)

Steven Byrnes

Pretraining data, not verifiability, is why LLMs are especially good at math (and coding) — LessWrong

124 Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)

by Steven Byrnes

18th Sep 2026

3 min read

124

Follow-up to: “LLMs are (still) mostly powered by imitative learning, not RL”

[Edited a bit since publishing, see changelog at the bottom.]

A common take I’ve been hearing is: “LLMs are especially good at math (compared to how good they are at other things)^[1] because math is easy to verify”. But that story doesn’t make much sense to me.

My main complaint is that, for advanced math, the verifier is an LLM.^[2] So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.

So here’s a different theory, in the framework of my earlier post “LLMs are (still) mostly powered by imitative learning, not RL”:

LLMs are especially good at math because almost everything in the math literature is correct. Read a random sentence in a random math paper in the research math literature, and you can be >99% confident that the sentence is true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.

So that’s my theory: it’s about pretraining data quality, not verifiability.^[3] Now let’s apply that theory to other areas.

What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.

What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to spit out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans. You might reply: “Perhaps this problem is fixable via curated mid-training data and/or RL?” Maybe. But if so, I think we should expect it to be much more of a challenge than it has been for code, let alone math.

How does this theory bear on the “LLM Recursive Self-Improvement” debate?

Well…

If you think that the path to superintelligence involves coming up with lots of mundane incremental ML ideas, then the question to ask yourself would be: Is the “mundane incremental ML ideas” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?
If you think that the path to superintelligence involves coming up with radically new AI paradigms, then the question to ask yourself would be: Is the “radical new AI paradigms” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?

I shall leave readers to reach their own conclusions.

Likewise,

How does this theory bear on the “LLM Automated Alignment Research” debate?

Same idea here. Again, readers may judge for themselves.

CHANGELOG:

A few hours after publishing, I changed the first few paragraphs, in order to: (1) acknowledge that RLVR on math questions with right answers is a thing (whoops I forgot, that’s embarrassing); (2) delete the claim that companies don’t do much RL-on-math, which was a vague memory that an expert in the comments section (Beren) has confirmed is false, whoops; (3) fix my incorrect description of the IMO paper. Thanks various people for pushback, especially Thane Ruthenis, Shreyas Sharma, and Beren in the comments, and singularvessel on X.

Next day (Sept 19): Moved some words from footnote 1 into the main text, to make sure people don’t miss it.

Oct 9: Added another sentence into Footnote 1, with more evidence against RLVR-via-autoformalization being an important part of how LLMs got good at math proofs.

^{^}
I’m not comparing LLMs to human mathematicians in this post; I have opinions but that’s a separate debate.
^{^}
In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset + more examples in Beren’s comment), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards).
(The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. Beren says that companies are doing that today, but I think that LLMs were already clearly especially good at proof-based advanced math (compared to how good they are at other things) before autoformalization-in-the-loop RLVR would have been feasible.)
(Indeed, it seems very far-fetched to me to think that you could build a Lean-autoformalizer of natural language proofs unless you already had access to a model that was pretty damn good at natural language proofs. So an RLVR-centric explanation of LLM math proof capabilities seems wrong to me on chicken-and-egg grounds.)
(ADDED OCT 9: also, ex-OpenAI post-training engineer Will DePue says: “…these results aren't using formal methods aggressively…”, in a discussion of the Navier–Stokes & October 6th proofs.)
Another place that “verifiability” matters is at inference time: the LLM might notice errors in its own chain-of-thought, or there can be more complicated scaffolds as in this paper about the International Math Olympiad. Same thing: the judge is an LLM.
^{^}
Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.

AI-Assisted AlignmentLanguage Models (LLMs)Recursive Self-ImprovementAI

Frontpage

124

Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)

1another-anon-do-gooder

New Comment

74 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:33 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Dylan Bowman1mo10342

This explanation fails to generalize to cyberoffense, where there are very few examples of cyberoffense being conducted via a human action sequence online, yet the LLMs are very good at it.

[-]Steven Byrnes1mo*1912

I strongly suspect that when people talk about the gory details of cyberoffense online and in textbooks, the things they say are overwhelmingly true. (E.g. if someone writes "this code is a working exploit of blah", then it almost always is.) In this sense, I think cyberoffense is LIKE when people talk about the gory details of research math stuff like algebraic topology (overwhelmingly true), and UNLIKE when people talk about the gory details of autism, or of middle school classroom discipline, or of how to write a good work of fiction (a big stew of truths and falsehoods and confusions that all superficially look the same).

So unless I'm wrong about the quality of discourse in the cybersecurity community, my theory in this OP would predict that cyberoffense is starting from a very good place, with the deck stacked in its favor for it to be one of the areas where LLMs are especially strong.

You're bringing up "human action sequences", and I heartily agree that post-training (SFT and/or RL) is important for transforming (something like) "base LLM has detailed and overwhelmingly correct understanding of such-and-such domain" into (something like) "post-trained LLM has the ability to b... (read more)

[-]Adam1mo105

Do we know that the improvements in cyberoffense aren't just downstream of improvements in programming generally? The "it's mostly pre-training" view admits of a world where enough pre-training on high quality programming generalizes to high level capabilities in various programming tasks, even those with less representation in the training data. Bel was (probably) not trained on a solution to Navier-Stokes, and yet was able to find one anyway.

[-]Dylan Bowman1mo112

Extensive knowledge of software is an excellent substrate, but offensive cyber doesn't really look like writing production code. Steven's view is partially correct but a big reason why models have gotten so good is just that RL teaches the model new things sometimes.

4ACCount1mo

It does, however, look like a mix of "in-depth debugging", "programming in a constrained environment", "examining a system you don't know", and so on. The closest thing to "offensive cyber datasets" that's out there in the public is probably CTF write-ups. Which were historically encouraged by CTF organizers.

5Dylan Bowman1mo

But none of the items in that mix are well-represented in the pretraining corpus either. These are all capabilities learned through post-training (human reasoning SFT and RL).

7Davidmanheim1mo

The recent improvements have been exploit chaining and other things that are largely out of sample. If those are downstream of general coding ability, it seems much more like impressive generalization of types that would allow other advances than it does support of Steven's theory. (Which, to be clear, seems reasonably compelling to me, though not anywhere near fully convincing.)

[-]Steven Byrnes1mo*110

“Out of sample” is hard to define. There’s a trivial sense in which almost everything is “out of sample”, because any given specific prompt has probably not appeared on the internet. But I think there are better ways to define “out of sample”, even if it’s hard to make rigorous. Like if a kid has drawn pink trees and purple trees but has never drawn a gray tree (but they have drawn gray cars and gray houses), then the kid can almost definitely draw a gray tree, without “learning something new”. I propose that we should define the term “out of sample” such that this kind of thing doesn’t count. I don’t think that LLMs have come up with a way to exploit a security flaw that isn’t very similar to ways that humans in the pretraining data have previously exploited security flaws. So I don’t think LLM exploits are “out of sample” in that stronger sense; rather, the LLMs are applying existing human ideas in a new context, with impressive nuance and competence. It’s a kind of style transfer, in a very abstract sense, as Jeremy Howard argued. That doesn’t detract from how impressive it is, nor from its real-world implications, but I think it’s relevant for our understanding.

As for exploit c... (read more)

2Davidmanheim1mo

I think that there are very, very few examples of exploit chaining compared the the number needed to learn a skill in pretraining, given sample inefficiency. So I'm claiming that the way it's accomplishing that is either via generalized skills based on too few data points to things very far outside what it's seen, which implies a level of capability we don't see in similar domains, or via RL.

2Young Jae Koh1mo

Cybersecurity has plenty of pretraining data that human actions can sequence. Capture the flag is a very much LeetCode equivalent as a standard. I have only novice white-hat experience and it is mostly derived from similar ones. Even more relevant is that these CTF benchmarks are well suited for RLVR, and so would LeetCode be for RLVR. Cannot say for certain without diving into architecture and pretraining data, but I suspect that is why Gemini was eval aware when offensive cyber tasks were requested, and called it "CTF" as well. Perhaps it may be better to say that "Programming and cybersecurity tasks are well suited for models because of pre-training data and possibility to do post-training gyms".

2daijin1mo

Sure, there's probably no 'a successful cyberoffence that actually worked', but probably a lot of 'what a successful cyberoffence would probably look like', including writeups of past cyberoffences, such that the present barrange of cyberoffences probably falls within the latent space between the writeups and what would actually work as a cyberoffence.

3Dylan Bowman1mo

But then this is not Steven’s argument. He is saying specifically that imitation learning is doing most of the heavy lifting for all LLM capabilities. You would not be able to get to the level of cyberoffense current LLMs display with just imitation learning and a tiny bit of RL to unhobble.

7Steven Byrnes1mo

The point of this post was to compare LLMs-doing-research-math to LLMs-writing-fiction or LLMs-doing-research-educational-psychology, not to compare LLM base models to post-trained LLMs. Obviously post-trained LLMs can do things that non-post-trained LLMs can't; it would be completely absurd to suggest otherwise. I'm unhappy that you got the wrong impression and reworded the first sentence to make that clearer. :)

[-]beren1mo7836

In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards). (The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. It’s possible that companies are doing that today, I dunno, but I think LLMs were already clearly especially good at proof-based math (compared to other things) before autoformalization would have been feasible.)

Strong disagree here. Definitely companies are doing RL with Lean and other autoformalization tools in the loop. This is a core component of any serious math RL pipeline these days. Secondly, even if not full Lean you can go very far with computer algebra systems to validate expressions such as Sympy if comparing to a known algebraic answer. Also you can go pretty far just by bootstrapping from a base of math questions that you already know the answer to and just doing standard RLVR and this transfers quite well to proof-based arguments also. LLM as judge is used but often not as primary verifier but instead for things ... (read more)

[-]Steven Byrnes1mo172

I appreciate the info on LLM training and will edit that footnote.

My claim is that, if we want to understand how LLMs can construct correct natural-language proofs that involve abstruse mathematical objects like the de Rham cohomology group, a big part of the answer is that that approximately every time anyone has ever written down anything about the de Rham cohomology group, that thing they wrote has been true. So the LLM emerges from pretraining with an tendency to say true things, and only true things, about the de Rham cohomology group, and the same applies to everything else in advanced math. This is very very different from most other topics, like anxiety or middle school classroom discipline. Of course, a tendency to say true things and only true things about the de Rham cohomology group isn’t enough, and I’m certainly not questioning the importance of post-training for surfacing that tendency at the appropriate times, and honing it, etc.

Basically, I think you’re being insufficiently curious about the part where you said “this transfers quite well to proof-based arguments also”. It seems to me that AIME problems and computer algebra etc. are really quite different from prope... (read more)

[-]beren1mo230

This is an interesting point! I am legitimately unsure here. I think this could have some effect in that there is less 'noise' in the training data (probably midtraining) math data than otherwise (although certainly people write incorrect or confusing math on the internet!). I think this is a broader point in that most information the models read on the internet is likely correct in that it is basic and undebated. I.e. very few people are writing that the capital of France is Berlin, that water is made of argon and copper, and that the battle of Hastings was fought in 1754 etc. LLMs are also very resilient to 'noise' in that because they see just so many variants of the same information that the occasional crazy statement in pretraining just doesn't perturb their world model much. Definitely though in pretraining you would obviously expect areas where human discussion is just a mountain of slop to produce slop in return (there is also an interesting experiment lurking here where we train LLMs on increasing proportions of synthetic 'false math' data and see to what extent they can still generalise real math correctly).

I do think you underestimate the extent to which people have ex... (read more)

4shawnghu1mo

Just piggybacking on Beren's comment; I agree that you severely underestimated the amount of effort that goes into explicit optimization on math capabilities. In particular, I would actually put odds at like 85% that LLM training today involves several RLVR tasks that require proper use of the de Rham cohomology group. However, your broader point totally stands; there's no way to do stuff with RLVR unless pass rate is nonzero, so the initial performance on such problems has to come from somewhere nontrivial. Most effort towards producing RLVR "environments" is only a small transformation away from producing imitative learning data, though, so the distinction is somewhat moot (and as granted elsewhere in this comment section, both of these are largely possible because math is unusually verifiable).

[-]Elias Schmied1mo229

So why did they get so much better at math in the last 2 years? Are you saying the pretraining data improved that much in that time, enough to get them to now solve Navier-Stokes? Or are you saying more general metacognition / persistence / intelligence improved and that's what caused it?

[-]Steven Byrnes1mo166

The models have been getting better at imitative learning (model size, architecture, learning algorithm details), and the data they’re imitating has also been getting better (filtering, data augmentation, etc.), and then on top of that, the mid-training SFT tokens and RLAIF are clearly huge parts of the story. (See The nature of LLM algorithmic progress.)

My claim is, like, if the companies had put just as much effort (data curation, RLAIF, etc.) into making their LLMs good at educational psychology, as they did into making their LLMs good at math, they would not have remotely as much to show for that effort. The RLAIF would be garbage-in-garbage-out because the LLM judge would be incompetent; and the data curation would require individually poring over every book and paper in the whole literature, not just “yes to arxiv, no to freshman term papers, etc.”, like you can do in math.

1bling1mo

yes, quality data is much more abundant in easy to verify domains for the reasons you have spelled out. this is solid evidence for verifiability not just being essential for rl, but for imitative learning as well. so your post title is misleading since your core claims are still about verifiability, just verifiability on the side of the humans that created and curated the pretraining data and not only on the side of the rlvr graders verifying correct answers.

9Steven Byrnes1mo

I did mention that in footnote 3. But I will defend my title. First, I’m not sure what you’re thinking about, but for me the context of a post like this is “we want to refine our nuts-and-bolts understanding of LLM capabilities”. I.e., LLMs can often do all these very impressive things; we can ask: how and why? This is an obvious and centrally important question. It helps us understand the world, and helps us predict future progress, etc. In that context, you want to know what goes into LLM capabilities in terms of their immediate inputs: what role is training data quality playing, what role is neural architecture playing, etc. etc. If training data quality is the critical factor for such-and-such aspect of LLMs, then we can separately chat about what goes into training data quality, but that discussion topic is now a step removed from understanding LLM capabilities, which again is the conversation that this post is participating in. Second, I think “verifiability” is more subtle than you’re suggesting. Imagine Lean or other formal verifiers didn’t exist today. Then human mathematicians would still be able to do math. After all, humans were doing math for thousands of years before the any formal verifier existed. In that hypothetical universe, what would it mean to say that natural-language proofs are “verifiable”? It really means that humans can reliably come to a consensus on proofs being right or wrong. But this is a joint property of a domain and a specific community of humans. I have on a few occasions argued for hours, days, even weeks, with morons / cranks about the validity of math proofs and derivations, and boy let me tell you, we did not reach any consensus. Mathematicians have developed social technologies to keep the cranks out, to double-check proofs, etc., and that’s an essential ingredient in math literature quality. So (still in this imaginary universe where nobody has made a formal verifier), the claim “math is verifiable” is pretty dubious, at

[-]Thane Ruthenis1mo184

the leading LLM companies have said that they spend very little effort on RL-for-math

Sources? All statements I've seen along these lines are of the form "this impressive benchmark-breaking model is not our specialized-for-math model", which I interpret as "we did not train it differentially harder on math than is our baseline", not as "we train very little on math as a baseline". (Same as e. g. Mythos not being explicitly trained to be unusually good at hacking, which obviously doesn't mean they didn't RLVR it on hacking tasks.)

See e.g. see a discussion by Google here, confirming that the whole training loop was in natural language, at least as of last year.

I don't think last-year methods are a good reference here. Last-year models weren't good at outputting Lean proofs, now everyone and their mother are successfully prompting them into that. I would guess the AGI megacorps do use proper RLVR now, not RLAIF. (Like, I don't think there are technical barriers anymore, so why wouldn't they?)

There is a core to your argument that I do agree with, though. Compared to other domains, the math corpus offers a dramatically greater... breadth, I guess, of legible and useful tools and building... (read more)

6Steven Byrnes1mo

I just deleted that part, thanks for the pushback. [...] I just edited to be clearer that (for all I know) companies might be doing RLVR with Lean autoformalization in the loop. (Along with another mistake.)

[-]StanislavKrym1mo123

I have a few issues with this.

How could one test this? By using a different similarly scaled pretrain (think of Talkie raised on pre-1930 data, but far bigger-scaled and focused on preventing contamination)?
The world saw Ryan Greenblatt elicit 50% performance on ARC-AGI-1 from GPT-4o. How does this method interact with your arguments?

6Hastings1mo

You could do a contaminated pretrain by making different fractions of the train set artificially bad math papers and see if it breaks math ability?

2Young Jae Koh1mo

I really love this counterfactual intervention!

5Steven Byrnes1mo

Certainly if you apply LLMs to a problem where they can do tons of trial-and-error in real time with unambiguous feedback (as in Ryan’s thing), then sure, to the extent that they’re relying on that feedback, the feedback was evidently important. I just don’t think that’s a good way to think about LLMs solving math problems today, where the “verification” (such as it is) involves an LLM’s ability to notice some subtle error in an abstruse natural-language mathematical deduction, and the important question is how LLMs can do that, and you can’t answer that question by saying “because math is verifiable”, since that would be circular.

3Zach Paris1mo

I think there are probably two things going on here. 1. LLM’s learn during pretraining sequences of thought patterns that are usedul for thinking about and verifying proofs. 2. A lot of the time the best way to solve a problem is just to try everything somewhat intelligently and see what sticks then prune after that. In academia the seeing what sticks is largely determined by peer review which obviously isn’t perfect but gives a noisy verification signal such that over decades you get a final literature that is more correct than it was at the start. The economy is roughly the same with businesses. When OpenAI sends thousands of models at a problem they’re essentially hoping for a similar roughly evolutionary process to play out. Why I think verifiability for LLM’s matter so much is that they operate at a wholly different rate than we do so there’s very few ways for LLM’s to get the kind of verifiable feedback which human society typically relies on. Not to say you can’t approximate these social structures with other models but for empirical sciences there is no clear path to immediately verify a theory and there’s not much evidence yet imo that models are much better at theory building than humans when verification is slow and you only get to try a few things.

1Young Jae Koh1mo

Much easier to do with open-weight models where we can get probes, but the eliciting 50% performance tells us couple things. Pre-training (mid-training too) gave the token generation capabilities, but the post-training for 4o did not have particularly focused evaluations for ARC-AGI. Post-training is best for steering token outputs. By generating 8k python transformation per problem and selecting the best, you're getting the best of what could have been an over-fitting for a general model versus capabilities for ARC-AGI-1. RLVR makes verifiable problems far easier for steering during training!

1GooseVowel1mo

I also have question2. IIUC, The method Ryan describes is "generate a bunch of programs" and then "filter those". He wrote: > The distribution of programs you are searching over [after generating a bunch of them] has to be pretty close to the right program for Best-of-6k to work at all: if you did best-of-6k for random python programs, this would not work! I agree that it's way better than random... but is it better than the human literature? E.g., If you asked me to filter 6,000 NLP papers, I'd expect a great many of them to be "basically true" and a great many more to be "a good effort but ultimately just very wrong". Is that the sense in which 6,000 programs he generated and filtered... match the distribution of human output? Like Steven says here, it "spits out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans." Am I thinking about this right?

[-]Archimedes1mo90

So that’s my theory: it’s about pretraining data quality, not verifiability.

These seem highly entangled to me rather than a dichotomy to choose from. Verifiability is what allows for high-quality training data in many situations, especially once you get beyond scraping the internet and into the realm of synthetic data. It also allows for clean attribution of the reward signal in an RL environment.

Math has some of the cleanest verifiability of any domain, especially with tools like Lean, which is far more robust than LLMs judging themselves. Not only do you... (read more)

3Steven Byrnes1mo

I’m talking about proof-based advanced math in natural language. This activity has never been “verifiable” until Lean-autoformalization became possible. And I claim that the models were already remarkably good at proof-based advanced math in natural language before Lean-autoformalization became possible. Indeed, it seems pretty implausible to me to say that the order of events during training / recent history is that first the models become capable of doing Lean-autoformalization of natural-language proofs, and only then the models become good at natural-language proofs, thanks to RLVR-via-autoformalization. Is that what you think? To me, that really seems like the wrong order. Lean-autoformalization of a natural-language proof is hard, and seems to require access to a model that is already very good at natural-language proofs.

3Archimedes1mo

Lean-autoformalization is just a particularly strong form of verifiability. I do not think it's a prerequisite for strong proof skills (and wasn't used in the papers I linked). I do think "verifiable" is highly relevant, even if it can be squishier than formally verified. Compare a question like "Is this natural language proof valid?" to "Is this story well written?" Which is easier to train on? If you can create a scoring function for stories that works as reliably as judging the validity of a natural language proof, then I'd predict you could train LLMs to write really good stories. The pretraining data is the baseline to bootstrap from. From there, it's a matter of providing training signals for the model to learn from, and verifiable domains provide clean signals. Even if the pretraining data is rather error-ridden, those errors get averaged away in next-token prediction unless they're systematic.

4Steven Byrnes1mo

Sorry if I'm misunderstanding, but aren't they always systematic in cases we care about? If you look at (say) people writing papers about the neuroscience of schizophrenia, there are a zillion mutually-incompatible ideas and schools of thought. It's not the case that everyone is saying "truth plus random noise", so all you need is enough data and you can average it out to get the true neuroscience of schizophrenia, right? [...] Yes this is exactly my point. It is much easier to create an LLM that says whether a natural-language proof is correct, than an LLM that says whether a fiction story is good. And I'm asking the question: Why the difference? You want to say "because math is verifiable", and I'm drilling into that and saying it doesn't make sense. What does "math is verifiable" mean to you, and what exactly is the path from "math is verifiable" to "the LLM can say whether a natural-language proof is correct"? The answer to that question can't involve RLAIF, because you can't even start RLAIF until you already have an LLM that says whether a natural-language proof is good. It's a chicken-and-egg thing. You CAN say "the LLM can say whether a natural-language proof is correct, including e.g. noticing subtle flaws in a proof's use of p-adic Hodge theory, because we did RLVR on its ability to do high school contest math problems with right and wrong answers, along with RLVR on algebraic manipulation tasks that we could check with SymPy, etc." At least, that hypothesis would not have a chicken-and-egg problem. But it strikes me as a very far-fetched answer to the question, that cannot possibly be hitting the nail on the head. Those two kinds of things are very very different! Why would there be such strong transfer from one to the other? I'm open to nonzero transfer, I just expect it to be very indirect and high-level. Whatever the LLM knows about p-adic Hodge theory that enable it to catch mistakes in proofs, it didn't get that from RLVR on math problems with ri

3Archimedes1mo

In unverifiable domains, yes. That's largely what I interpret "unverifiable" to mean. By "math is verifiable", I'm describing a property of the domain, not the LLM. A disputed result can, in principle, be decomposed into local steps small enough that any trained mathematician (or automated proof checker) will agree on each step. When mathematicians disagree, the disagreement is traced to a specific step that gets settled. In schizophrenia research, there are no similar resolvable cruxes to surface, so disagreeing camps continue publishing and systematic error builds up without a correcting mechanism. Verifiability of a domain tends to correlate with the quality of the training data, but it's not the same thing. Math lends itself to checks that are not LLMs, like answer keys, computer algebra, and Lean. RL against those checks grounds the model in something consistent and correct, and the training pressure also reaches the LLM's reasoning. Any single right answer can sit on an invalid step, but across broad varieties of problems, RL reinforces valid reasoning. Lean isn't required for this and likely came later, but it can grade the whole chain, which makes the grounding cleaner. This is why the transfer from right-answer RLVR to proof-checking that Beren describes doesn't feel that far-fetched to me, and also why I'd predict math reasoning to be repairable even from a contaminated corpus (as in Hastings' proposed experiment), even one with coherently wrong results. A wrong pocket can survive in math if it's isolated enough (Mochizuki's disputed abc proof is perhaps the human version), but the odds of correction scale with its overlap with the rest of mathematics, and whatever matters broadly overlaps a lot. A contaminated psych corpus has very little to repair against. Would you predict differently? Knowledge of p-adic Hodge theory specifically will mostly come from pretraining. A subtle flaw in a p-adic Hodge theory proof boils down to some simpler step failing, l

[-]Pretentious Penguin1mo80

I’m very surprised by the claim that frontier labs aren’t including Lean verification in a RLVR loop for math. It seems intuitively like this would be very helpful. Is it possible that frontier labs started doing so in the last year? Or is there some reason why it’s not actually that helpful?

5Steven Byrnes1mo

I think it was already obvious that math was one of the areas that LLMs were best at (compared to other things like law, fiction, psychology, etc.) a year or two ago, which I think was well before autoformalization would have been realistic. So there has to be an explanation for “LLMs are unusually good at math (compared to how good they are at other things)” that doesn’t involve autoformalization. That said, I suppose it’s possible that people are doing that now, I wouldn’t know, and I hope I didn’t imply otherwise. UPDATE: I edited the footnote, hope it’s better now, thanks.

3shawnghu1mo

I think it's totally possible that progress up to two years ago, or even one year ago, can be mostly explained by imitative learning on the pre/mid-training data, but not progress over the last year. Models two years ago performed admirably but not perfectly on stuff like multivariable calculus problems, for which there undoubtedly was a massive corpus. Models one year ago still struggled with problems like AIME problems, and slightly more obscure things like questions in mathematical logic or more obscure corners of graduate level algebra, etc. I don't think there's any corpus in existence which gets you present capabilities with only a small amount of RLVR, though. Or maybe, as you've suggested in many other comments, the two factors are actually very entangled: maybe a lot of progress does come from imitative learning from really high quality data, BUT also the labs have been able to put forth a massive effort in producing new data to imitatively learn on, and this is because verification can be leveraged. To the extent that the new data is produced by LLMs themselves, this is a form of learning-from-self-with-verification, just on a farther-out outer loop than standard RLVR.

1Young Jae Koh1mo

Which would you weigh more heavily? That models included more math in pre-training or was there more post-training RLVR with math gyms with more synthetic math problem-sets?

2shawnghu1mo

It's hard to compare these. Perhaps a larger effect with respect to improvement in some metric was achieved by the former (if you also include filtering for higher-quality midtraining data), but of course the latter isn't actually possible/useful without the former, and the latter was absolutely necessary to bootstrap to solving Navier-Stokes, e.g. One way of factoring an answer to your question is that I think on the margin, improving the pretraining mix is probably more powerful per unit cost for contemporary models.

1Young Jae Koh1mo

Definitely! Yet I would say the Navier-Stokes solution is hard to narrow down its origins and margins even more. OpenAI said 88 hours, yet, did not state the token quantity, how it branched experiments, how often they had humans guiding taste and feedback. It is important to recall that there is both intentional and unintentional obfuscation of how they got to the solution. The margin comment is in very good taste, massive, incredibly expensive to run models may not mean as much if they can only be justified for a problem of that caliber.

[-]Clark Benham1mo60

I’ve been surprised to see LLMs make massive conceptual errors in doing business analysis, especially as Cowen links to them answering Econ PhD questions way better than I ever could. They clearly grasp all concepts but when it comes time to apply them they default to cliches. Could be a case of what situations/prompts trigger their full abilities and they’re “being lazy” here or it could be with a lot of bad data they think hard and decide their answers are optimal for the question even though numerically they’re totally off.

eg across ChatGPT and Claude o... (read more)

4Seth Herd1mo

When you say "ChatGPT and Claude" do you mean 5.6 and Opus 5 or Astra and Fable? I haven't used Astra much but Fable seems noticeably less likely to make such mistakes - possibly because it's a better general reasoner.

1Clark Benham1mo

5.5 and opus 5 I believe

[-]Rafael Harth1mo50

“easy to verify” only matters for the RL part of LLM training pipelines

I think easy to verify matters for execution. I don't see an argument in this post against this (which is fine if you just want to argue against some views).

LLMs are especially good at math because almost everything in the math literature is correct.

Okay but the reason math training data has low % of false statements is because math is easy to verify. So even if this were true, it would still be downstream of verifiability.

I suspect it's also not true though. If we got rid of the... (read more)

9Steven Byrnes1mo

I assume you mean “the LLM can notice that it’s doing something wrong in math and then backtrack”? Stuff like that? That’s fair, but it amounts to the same thing as the second bullet at the top (“So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.”) I reworded, thanks. [...] Yup, I had mentioned that in footnote 3. [...] No opinion about whether 95% is enough (and maybe you’d have to go through papers sentence-by-sentence), but my guess is that there’s some hypothetical philosophy pretraining corpus for which LLMs (post-RLAIF) could do philosophy at vaguely the same level that they’ve been doing math, which is “a very impressive level” in an everyday sense, and certainly a much higher level than how they do at philosophy today. As for “topic X”, no I don’t think LLMs can invent new concepts in the way that humans can, in math or philosophy, regardless of pretraining corpus, but that’s a different topic that I’m putting aside for this post, see footnote 1.

4Rafael Harth1mo

(My bad for not reading the footnotes!) [...] Right, you think that's a qualitatively different category. To me, coming up with a new concept/field is just the central example of progress-on-stuff-that-can't-be-easily-verified, and of a piece with the kind of thing you need to make progress on almost anything. So [doing good philosophy, but only in a known field] feels like an unnatural category (which isn't a critique, I'm just explaining why this view is odd to me). Like where is the line between doing good philosophy and just summarizing the now-very-high-quality-literature? So thinking about this, ummmmm so I think it would be able to answer all commonly asked questions correctly plus stuff that takes 1-2 reasoning steps... which probably does look like what you described, like it would look like pretty good philosophy to most people (I doubt many philosophy arguments need more than 1-2 steps), and certainly way way better than what they say now....... so it doesn't look like we disagree much at all there. ... but I still think LLM limitations are very much exactly about verifiability. I would just say the reason LLMs are better at judging math is that math is easier to verify in an objective sense (and figuring out what exactly that means and how humans verify non-math stuff would be an extremely important step toward building AGI, so umm hopefully no one does it anytime soon)

[-]Satron14d4-9

I think chess might be a counterexample to your thesis. If this benchmark is to be believed, Astra has a chess.com Elo of roughly 1600, which corresponds to a player with a good grasp of strategic principles who rarely blunders. For comparison, the average human chess.com Elo is roughly 600, which corresponds to a player with a very poor grasp of strategy who blunders constantly. So someone who thinks that pre-training data is what gives LLMs most of their capabilities in a given domain should expect LLMs to resemble the latter rather than the former.

2Steven Byrnes14d

When people talk about the gory details of autism, or of middle school classroom discipline, or of how to write a good work of fiction, it’s a big stew of truths and falsehoods and confusions that all superficially look the same. That italicized part is important. There’s bad math on the internet too, but the bad math and good math on the internet do not superficially look the same. The good pure math is in LaTeX on arxiv and in journals and books, whereas the bad pure math is in deranged blog posts that don’t use any fancy words, etc. In this respect, I think that chess is more like math, and less like autism or middle school classroom discipline etc. I suspect that there are obvious superficial markers of whether someone is playing chess well or not. For example, I suspect that many (most?) games on the internet are clearly labelled with the Elo of the two players. Also, you get to see who won! Separately, I’d guess that the average Elo of chess games in Astra’s training data is higher than the average Elo of a human on chess.com. I could be wrong about either of those, I don’t know where all the chess games are on the internet. But I did quickly ask an LLM where to find tons of chess games on the internet, and it suggested https://database.lichess.org/ which does in fact have Elo scores for black & white for each game as I surmised.

1arielb112d

Are there actually written truths in middle school classroom discipline, or to be less harsh on humanities people on software engineering design? There are lots of written pieces, which might be informative, but they are more like pieces of information than truths. I think that AI is basically superhuman at separating ordinary falsehoods from ordinary truths, but is not really capable of extracting a complex model from disparate pieces of information. At least in programming, which is the area I have the most experience on, AI seems to work in areas where you can get the correct answer by interpolating between existing code fragments that I am pretty sure are in the AI's training set, cases where a human could do surprisingly well much by "pure herd instinct" without deep understanding. I am not sure you can do middle school class discipline this way.

2Steven Byrnes11d

I think we’re generally in agreement. The internet has tons and tons of statements about middle-school classroom discipline that are specific enough to be either true or false. Of those statements, by definition, some of true and some are false. I don’t expect that LLMs would be very good at at discerning which is which, in general. They all superficially look the same, they’re all equally in the training data, there’s nothing in the LLM training process that would allow the LLM to make sense of this mess. Instead I’d expect LLMs to mostly go along with views about middle school classroom discipline that seem mainstream and prestigious, or in some cases to sycophantically endorse views that the user would probably like. (I didn’t check, and wouldn’t be able to anyway.)

1arielb111d

I don’t think you could get an AI that can do middle school classroom discipline as well as it does e.g. math by taking existing training sets using existing training techniques and doing something that can be described as “merely removing or labeling bad statements”. I think you need to either add more data, or to find some currently-unknown way of making the existing information more LLM-accessible that goes beyond “labeling bad statements”.

1_rpd12d

I think OpenAI's proof of quasi-Riemann is evidence against this belief ... > Among the 377 results that OpenAI announced on October 6, 2026, one stood out. It claimed to settle the “quasi-Riemann hypothesis” - a weaker version of the Riemann hypothesis, which is arguably the biggest open question in math. The new proof is, as Jared Duker Lichtman (opens a new tab) of Stanford University put it, “the most significant AI-for-math achievement so far, by quite a margin.” > ... > contrary to what mathematicians had long expected, the proof didn’t need to introduce any new concepts - it instead unearthed hidden connections between old ones. “The proof is a surprising synthesis of well-known ideas,” said Alexander Dunn (opens a new tab) of the Georgia Institute of Technology. https://www.quantamagazine.org/ai-proved-the-quasi-riemann-hypothesis-what-happens-now-20261019/

2arielb112d

Finding a connection between old ideas, especially by "deep-model-free" application of a technique from one area in another one, is exactly (!) the sort of "interpolation" that I just claimed LLMs are absolutely amazing at. (Again, not currently having an opinion whether "making an AI that takes over the world" is the sort of thing that can be done via interpolation).

[-]Leigh1mo42

Love this: "In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider."

Meanwhile, I am thinking to myself: "Hmmmmm. . . Steven is standing out from the pack again, with very different ideas, as usual, but it is Steven Byrnes, who seems to tend to be correct on most things within this domain, so perhaps he is right about this, as well."

I think what you are implying here is that the "radical new AI paradigms" ... (read more)

2Aprillion1mo

I imagine hitting a wall would require that LLMs would have literally zero capability at producing incremental insight needed for further ML improvement, since even if they would be terrible at it, you can always brute force search (a la self-play selection) for verifiable stuff if your budget is sufficient enough - so I would place my hope on making frontier-level spending illegal and not on hitting natural walls

1Asta7k1mo

Why isn't ML research more like coding, where things generally just work, even if the result is messy and riddled with small errors here and there? My impression is that in the pre-autoresearch era, the average quality of radically new AI paradigms was very high and they actually worked within their compute-constrained settings. And to borrow an argument Dwarkesh made recently: the fact that LLMs are bad at idea generation but good at hill-climbing well-defined problems (Navier–Stokes being one example) might actually be a blessing for automated capability and alignment research. These fields don't require a unifying theory of deep learning, they mostly require hill-climbing on some metric, be it perplexity, bits per token, or a benchmark score, and those are verifiable. Or you could hire a bunch of human annotators to judge ideas and hope that, after a while, RLAIF works well enough for the model to score outputs from other models, like how labs were hiring IMO gold medalists to build RL environments and grade LLM outputs.

[-]Shreyas Sharma1mo40

the leading LLM companies have said that they spend very little effort on RL-for-math

I'm curious as to the source for this. Even if true, it could be precisely because verifiability makes RL-for-math easy.

In the case of RL-for-math, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards)"

Your evidence for this is the discussion on Deepmind's IMO 2025 approach, however this is an inference-time scaffold and not RL-for-math.

That being said, the core idea—the correctness of imitation data is undervalued relative t... (read more)

3Steven Byrnes1mo

It’s not obvious how to build a synthetic data pipeline for natural-language proofs. I guess you’d have an LLM generate candidate proofs, and then judge them by Lean autoformalization, right? If so, that’s basically the same as RLVR-for-proofs-via-autoformalization as mentioned in the (recently updated) post. And my argument is that it was already clear that LLMs were unusually good at natural-language proofs before anyone could have built such a system.

3Steven Byrnes1mo

Yeah fair, I deleted that part a couple minutes ago, see changelog. [...] Whoops that’s embarrassing, I’ll fix it now.

[-]ankitmaloo1mo31

LLMs are good at math, coding, cybersec because of some advancements in post training, not pretraining data. pretraining data matters, ofcourse, but we are already in a post training regime.

i don't know how models do self verification. but a typical pipeline today is mopd driven. do sft -> setup RL infra -> generate a lot of trajectories -> rejection sampling (using a verifier) -> on policy distillation -> another round of RL. do it separately for different domains and merge it.

what changed with fable and sol (and then astra) was that they... (read more)

[-]ACCount1mo20

As a rule: imitation learning pre-training gets a model in the ballpark of "good performance", and post-training actually gets it there.

I don't see a single convincing reason why that wouldn't hold for math.

Going "text prediction -> autoregressive reasoning" is a sharp distribution mismatch. Some things, like knowledge and associations, transfer across the boundary well. They lay the foundation of performance - stronger pre-training makes for stronger models. But an error that was minor for text prediction can be crushing for autoregressive reasoning. ... (read more)

[-]snowballantrobus1mo21

Perhaps "easy to verify" also implies it is possible to generate massive quantities of high quality synthetic training data. This seems important given how inefficient current algorithms are at learning.

[-]Bryan Whys1mo10

That's also why they're great at being pos.

The dark web/hate web is in the pretraining data. It doesn't get filtered because it provides logic capabilities. Mandating that it not be used in frontier pretraining data might help temporarily, but it won't work over time due to scaling laws.

The only solution is to have an edited version of the dark web that isn't heinous, but is as competitive as the uncensored version. You have to simultaneously create organizations that analyze and filter the data for pretraining prior, and pump the internet with additional... (read more)

[-]Matthew_Opitz1mo10

One measuring stick I have for how close we are to "ASI" is the extent to which an AI can find a realm of knowledge where the mainstream of human learning has happened to stumble down a blind alley, a set of incorrect theories increasingly kept afloat with ad-hoc epicycles, where the AI confidently tries to offer alternative answers, and in explaining those answers, is able to build up an entire alternative field of literature on that topic from first principles. I know this sounds like a high bar, but ASI would be a big deal, so it has to be able to clea... (read more)

[-]another-anon-do-gooder1mo10

I'm surprised that no one (to my knowledge) has successfully trained an LLM just on high-quality prose (or jokes, etc.). I'm not sure if that's because no one has tried or because the outputs haven't been especially compelling. This paper was the most concerted effort on the topic that I could find to date, but it's results don't seem especially compelling https://arxiv.org/abs/2601.18353

[-]Ephraiem Sarabamoun1mo10

Thank you very much for sharing! In a sense, all capabilities really originate in pre-training. Mathematical ability, research ability, fiction writing ability, etc. all have to come from somewhere in the pre-training distribution. Clearly though, modern models excel in certain subjects such as mathematical ability more than others like fiction writing. I disagree with you in that I do think the fundamental reason for this is post-training. I suppose the key experiment that we can run to see which of us is right is by testing a base-model on mathematics vs... (read more)

[-]Josh Snider1mo10

This was a very thought provoking article even beyond the specific argument you were making. I'm tempted to make a chart with labels since it seems like there's a lot of equivocation between similar, but different things. It would be interesting to see the trade-off between adding more but low quality math and getting better verifiers.

[-]alexfeghhi1mo10

What's the evidence for RL not making LLMs much smarter? I can definitely see the argument where pretraining is where the LLM gets its understandings/world models and RL unlocks them towards goals, but there's also AIs like AlphaZero which are pure self-play. Surely with enough RL the LLM world models get better as well, right?

3npg1mo

AlphaZero gets a bunch of data on what works and what doesn't work from Monte Carlo Tree Search. It's running a bunch of experiments on chess or Go and slowly absorbing the results into the weights of the neural net. In the case of RL on informal tasks, there is no such external automated source of data. The model has to produce a bunch of attempts on any given task, and if none of those attempts succeed, it doesn't get any learning signal. The only "exploration" comes from tiny perturbations to the model weights, so if the model just doesn't have the ideative pathways to solve a particular problem - say, it needs to prove some theorem but it doesn't know about mathematical induction - no samples will contain a correct resolution to the problem and the model won't learn anything. The famous paper that studies this empirically is https://arxiv.org/abs/2504.13837, and Steven also has a nice blog post about this here.

1alexfeghhi1mo

I think what you're trying to say is that in order for RL to derive new concepts/strategies, the underlying architecture has to be "creative" enough to come up with those concepts in the first place and have them reinforced. This is a strong reason to be bearish on LLMs + RL to ASI. I'm curious about how strong this creativity limitation is. Sometimes I wonder about scenarios where some form of Continual Learning is discovered, and LLMs get constant RL data from each interaction with users. What would capabilities look like? I guess the crux is if the creativity limitation is a hard limit or just an extreme sample inefficiency.

2npg1mo

Sorry, didn't see this before. Yeah, pretty much. Although I actually believe that any frozen model architecture is limited by this kind of creativity wall. On the other hand, I suspect that a humanlike learning algorithm would allow for more genuine creative exploration even without external data inputs, because it would be able to interpret the weights and find interesting conceptual directions within them. I don't see any reason to think that such technology is on the horizon, but it is much much scarier (even in the absence of RSI in the classical sense).

[-]dan.parshall1mo11

This is encouraging, and I hope true!

[-]Sodium1mo02

This seems like a terrible take. Why didn't GPT-4.5 crush at math then, if it was such a big pretrain? Obviously you need RL to get good performance at solving math problems (and in order to effectively leverage inference time compute and work together for a long time in an agent swarm.)

Moderation Log