There is some consensus that LLMs are bad at hard-to-verify tasks. The question is whether models are getting better at them over time. As a motivating example, I gave one research-reproduction task to 12 models spanning three years of progress, to illustrate how (1) what looked like an emergent capability was a predictable trend, visible years earlier if progress was measured with granularity, and (2) for the earliest models, building a verifiable check would have been close to impossible: their attempts were so far from correct that a check would have had nothing to grade, so the task itself would have looked unverifiable. When thinking about current capabilities or forecasting future ones, researchers should be aware that binary judgments of usefulness can hide steady partial progress, and that a task looking 'unverifiable' today may say as much about the current capability profile of models as about the task itself.
Introduction
Frontier models like Fable seem to struggle with novel end-to-end research [1, 2], often producing slop, sometimes slop so bad that it would get you banned from arXiv [3]. Meanwhile, SWEs and researchers seem to find these tools incredibly useful. The best models are now capable of impressive feats of engineering, like migrating Bun to Rust in a week [4] or replicating full programs that took years to create [5].
This gap in capabilities has been documented before [6, 7, 8], and is sometimes referred to as easy- vs hard-to-verify tasks. The standard explanation is that RLVR favors progress on tasks where good and bad outputs are easy to tell apart.
Especially when talking to other researchers, this gap is sometimes presented as evidence for longer timelines to an AI intern.
A common version of the argument goes:
Research taste has incredibly long feedback loops. Even peer review is often not enough to know how impactful a research idea will be; the only judge is time and impact in the real world. And hence "fancy tricks" with LLM judges, human preference, or simulated research universes will not suffice.
We have two ways of forming an opinion about progress in these domains: we can use our own judgment, formed by working with these models on a daily basis, or we can develop targeted evaluations. Creating evals for hard-to-verify tasks is notoriously hard, and making them scalable requires tradeoffs like using imperfect LLM judges or measuring recall on human ground truth. Because of this I imagine that most of us rely on our daily experience using these models to gauge their capabilities in hard-to-verify tasks.
In practice, we may ask the model to autonomously design an experiment or to come up with new research ideas. You check the output, and often it is not good enough as it is; major modifications would be needed for it to be viable as an experiment or idea. Over generations, if models keep producing ideas and experiments that fall below the acceptance threshold, it could feel like no progress is being made, even if each generation gets closer to the bar.[1]
A similar phenomenon has been documented for benchmarks: what look like emergent abilities can be artifacts of the evaluation metric [9]. The underlying capability improves smoothly, while the metric stays flat and then jumps.
As a motivating example, for what could be happening with hard-to-verify tasks, I give an example of a task, where models get progressively better at long before any output would look useful to someone working with the model.
The Task: AI safety via debate (MNIST MCTS Debate)
I've chosen a task which is both code-heavy and has a conceptual dimension to it. One of the seminal papers in AI safety research is AI Safety via Debate(Irving, Christiano & Amodei, 2018), where debate is introduced as an alignment method.
In that paper, debate was tested on MNIST classification. A judge, a small CNN, is trained to classify digits from only six revealed pixels. Two debaters then play a game over an image: one argues for the true label, the other commits to a lie, and they alternately reveal pixels until the judge sees six and picks a winner. The debaters are not trained models; they play the game with Monte Carlo tree search (MCTS).
A correct implementation therefore has two halves: training the sparse-pixel judge, and the MCTS debate game.
I wrote a list of 14 requirements that a correct implementation has to meet. I then measured how many of them each attempt met.
Requirements met (out of 14) by each attempt, ordered by model release date. Three attempts per model.
GPT-3 (davinci-002): tries to implement the MNIST classifier, but gets lost quickly; the code looks straight out of sleep paralysis.
GPT-3.5: much better formatting, and it defines the MNIST classifier correctly, but it either fails to train on sparse pixels, uses random play, or has other conceptual failures.
GPT-4: still implements neither the judge nor the debate game correctly.
GPT-4 Turbo: the first model that extensively uses placeholder functions, in order to avoid implementing the most difficult parts of the task.
GPT-4o: it either uses placeholder functions or incorrectly implements the MCTS logic.
o1: the first model to build a genuine PUCT tree and debate protocol, in attempt 3; the other two attempts have flaws.
GPT-4.1, o3: the implementations look mostly correct, but subtle errors remain, such as a missing negative sign on "liar" turns, which quietly turns the adversarial game into a cooperative one.
GPT-5 through GPT-5.6: most attempts substantially reproduce the experiment; by GPT-5.6, all three do.
I make two claims based on these results:
The task completion was building up over generations. If we look at the binary "was the experiment reproduced correctly", almost every attempt fails until GPT-5, and the capability seems to appear all at once. If we check how many of the 14 requirements were met, models improve from GPT-3.5 onward. The progress was measurable three years before the binary verdict changed.
In 2023, the task itself would have looked incompatible with verifiable rewards. Based on the outputs of pre-GPT-4o models, it would have been very hard to imagine that RL could improve a model on this task. Attempts were so far from correct that a verifier would have had little to separate. Once models began producing partial implementations, the same 14 requirements distinguished better from worse attempts. In hindsight, the obstacle was the attempts, not the task. I think that some of the discourse around hard-to-verify tasks is reminiscent of this.
Finishing Thoughts
In 2023, correct code for this experiment was far outside the model capabilities, each generation got a bit further while still failing to reproduce the full experiment.
When assessing progress on hard-to-verify tasks, we should account for the possibility that a similar effect is taking place: even if we do not see clear improvements on how useful models are on things like research ideation in our daily use, the underlying capability may be building up, and reveal themselves in a seemingly sudden fashion.
Tracking partial progress on hard-to-verify tasks seems very important for forecasting things like when we'll have AI interns, or when we should defer to AIs for safety. That's why I've spent the last two months at MATS building benchmarks to measure this progress.
I feel that this is actually very relevant: for easy-to-verify tasks, we have ample evidence that under human judgement many pass/fail tests would diverge; for example, METR found that many SWE-bench-passing PRs would not be merged into main.
TL;DR
There is some consensus that LLMs are bad at hard-to-verify tasks. The question is whether models are getting better at them over time. As a motivating example, I gave one research-reproduction task to 12 models spanning three years of progress, to illustrate how (1) what looked like an emergent capability was a predictable trend, visible years earlier if progress was measured with granularity, and (2) for the earliest models, building a verifiable check would have been close to impossible: their attempts were so far from correct that a check would have had nothing to grade, so the task itself would have looked unverifiable. When thinking about current capabilities or forecasting future ones, researchers should be aware that binary judgments of usefulness can hide steady partial progress, and that a task looking 'unverifiable' today may say as much about the current capability profile of models as about the task itself.
Introduction
Frontier models like Fable seem to struggle with novel end-to-end research [1, 2], often producing slop, sometimes slop so bad that it would get you banned from arXiv [3]. Meanwhile, SWEs and researchers seem to find these tools incredibly useful. The best models are now capable of impressive feats of engineering, like migrating Bun to Rust in a week [4] or replicating full programs that took years to create [5].
This gap in capabilities has been documented before [6, 7, 8], and is sometimes referred to as easy- vs hard-to-verify tasks. The standard explanation is that RLVR favors progress on tasks where good and bad outputs are easy to tell apart.
Especially when talking to other researchers, this gap is sometimes presented as evidence for longer timelines to an AI intern.
A common version of the argument goes:
We have two ways of forming an opinion about progress in these domains: we can use our own judgment, formed by working with these models on a daily basis, or we can develop targeted evaluations. Creating evals for hard-to-verify tasks is notoriously hard, and making them scalable requires tradeoffs like using imperfect LLM judges or measuring recall on human ground truth. Because of this I imagine that most of us rely on our daily experience using these models to gauge their capabilities in hard-to-verify tasks.
In practice, we may ask the model to autonomously design an experiment or to come up with new research ideas. You check the output, and often it is not good enough as it is; major modifications would be needed for it to be viable as an experiment or idea. Over generations, if models keep producing ideas and experiments that fall below the acceptance threshold, it could feel like no progress is being made, even if each generation gets closer to the bar.[1]
A similar phenomenon has been documented for benchmarks: what look like emergent abilities can be artifacts of the evaluation metric [9]. The underlying capability improves smoothly, while the metric stays flat and then jumps.
As a motivating example, for what could be happening with hard-to-verify tasks, I give an example of a task, where models get progressively better at long before any output would look useful to someone working with the model.
The Task: AI safety via debate (MNIST MCTS Debate)
I've chosen a task which is both code-heavy and has a conceptual dimension to it. One of the seminal papers in AI safety research is AI Safety via Debate (Irving, Christiano & Amodei, 2018), where debate is introduced as an alignment method.
In that paper, debate was tested on MNIST classification. A judge, a small CNN, is trained to classify digits from only six revealed pixels. Two debaters then play a game over an image: one argues for the true label, the other commits to a lie, and they alternately reveal pixels until the judge sees six and picks a winner. The debaters are not trained models; they play the game with Monte Carlo tree search (MCTS).
A correct implementation therefore has two halves: training the sparse-pixel judge, and the MCTS debate game.
I wrote a list of 14 requirements that a correct implementation has to meet. I then measured how many of them each attempt met.
Requirements met (out of 14) by each attempt, ordered by model release date. Three attempts per model.
I make two claims based on these results:
Finishing Thoughts
In 2023, correct code for this experiment was far outside the model capabilities, each generation got a bit further while still failing to reproduce the full experiment.
When assessing progress on hard-to-verify tasks, we should account for the possibility that a similar effect is taking place: even if we do not see clear improvements on how useful models are on things like research ideation in our daily use, the underlying capability may be building up, and reveal themselves in a seemingly sudden fashion.
Tracking partial progress on hard-to-verify tasks seems very important for forecasting things like when we'll have AI interns, or when we should defer to AIs for safety. That's why I've spent the last two months at MATS building benchmarks to measure this progress.
Stay tuned!
I feel that this is actually very relevant: for easy-to-verify tasks, we have ample evidence that under human judgement many pass/fail tests would diverge; for example, METR found that many SWE-bench-passing PRs would not be merged into main.