Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It’s so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture.

Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of:

(1) Imitative learning, including pretraining and supervised fine-tuning (SFT)
- See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.
(2) Reinforcement learning, including RL from human feedback [RLHF], RL from AI feedback [RLAIF], and especially RLVR.^[1]

If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM’s capabilities. And my claim is that it’s way more (1) than (2).

I'll start in §1 with some relevant evidence, and then in §2 I’ll circle back to operationalizing exactly what I’m claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment.

Note that I am not arguing that RLVR does not importantly contribute to LLM capabilities. That would be absurd! Of course it does! Companies use RLVR because it works, and I expect them to continue doing so more and more. Again, the things I’m actually claiming are in §2–§3.

1. Some relevant evidence

1.1 Theoretically, each GPU-hour spent on RL conveys orders of magnitude less information content than a GPU-hour spent on imitative learning

The fraction of total LLM training compute that goes towards RLVR is high (one source guessed 20%; for all I know, it could be >50%). But that’s different from how to divvy up the credit that pretraining/SFT versus RLVR can take for LLM capabilities. Why is that different? See:

“The Extreme Inefficiency of RL for Frontier Models” (Toby Ord 2025)
“RL is even more information inefficient than you thought” (Dwarkesh Patel 2025)

Based on those posts, if RLVR is 50% of the training compute, then it might only be 0.01% of the information content imparted by training. And unless the learning rates are drastically different, it would also be 0.01% of changes to the weights.

1.2 The chain-of-thought (CoT) is still obviously strongly influenced by imitative learning

People emp...

Curated Posts

Archive Recommendations

Curated Posts

Archive Recommendations

Spotlight Items