This research was completed for London AI Safety Research (LASR) Labs Summer 2026. The team was supervised by Justin Olive (Generality Labs) and Magda Dubois (UK AISI). Find out more about the programme and express interest in upcoming iterations here.
We encourage readers seeking technical details to read the paper on arxiv.
TLDR: We rely on benchmarks to estimate model capabilities. But now, many existing benchmarks are becoming too easy for frontier models – causing them to become saturated. For high-risk domains like cyber and bio, we cannot afford underestimating how capable models are in these domains. So how do we estimate capabilities after benchmarks have saturated? Running a benchmark already produces a transcript of how each model solved each task, meaning that the information within the transcript is just discarded. Using transcript analysis literature, we propose a set of protocols and replayed the last few years of model releases to see if they could predict capability beyond benchmark saturation. And they can, for the most part. But they also fail to generalise across “paradigm shifts” such as when reasoning models came out and models suddenly started to behave differently. Our main argument is that we need to move past evaluations that are judged by their average accuracy on familiar models, to methods for scalable evaluation that are judged on how robust they are across shifts like this one.
Saturated on day one
On September 22, Scale released SWE-Bench Pro V2, a cleaned-up version of one of the most widely used frontier software engineering benchmarks. On the public split, Claude Opus 5 solved 638 of 642 tasks. A benchmark intended to measure capability at the frontier was already at 99.4% on release.
But things didn’t always used to be this way.
We’ve of course had benchmarks that had outlived their usefulness before, but the pace at which it happens has changed dramatically. When GLUE became too easy, researchers built SuperGLUE in 2019 to raise the bar. It took roughly another year and a half, but SuperGLUE also saturated. Fast forward to January 2025, when Humanity's Last Exam was released. Thousands of domain experts took part to create what was meant to be the “frontier of human knowledge”, only for models to climb from initial scores of less than 10% to over 50% by 2026. It seems likely a matter of time before this benchmark is also saturated.
A recent systematic study of 60 benchmarks that major AI labs report in their technical reports found that nearly half of them already show signs of saturation.
The problem isn’t necessarily that models are “getting too good”. It’s that we lose our ability to track model capabilities as benchmarks saturate: if two models both solve 99% of the tasks successfully, their scores stop being useful for distinguishing their actual capabilities. One might be significantly more capable than the other, but we would have no way of knowing otherwise.
Of course, we can always build harder benchmarks. But that takes time. The SWE-Bench Pro V2 refresh alone required 23 engineers and nearly 1,900 hours of work, only for its public split to be nearly saturated on release. With the pace at which new models are being released, continually replacing benchmarks with harder ones is an increasingly difficult way to keep track of progress.
Why safety cannot live without a score
In terms of building leaderboards and pointing at the newest shiny model, saturation is just an inconvenience. However, there are deeper consequences for safety considerations. Safety frameworks and regulations are built on capability thresholds, meaning a lab decides what safeguards a model needs by checking where its capability sits and regulators decide what constraints should be placed in the development and usage of these models using similar thresholds.
In this regard, the only signal we would get from saturated benchmark scores is that a model is… “good”.
In domains where this matters, we're pretty much already in this regime of uncertainty. When OpenAI released GPT-5, it classified the model as having “High” biological and chemical capability under its Preparedness Framework, even though its evaluations did not provide definitive evidence that the model had crossed that threshold. Rather than risk underestimating its capabilities, OpenAI decided to apply the corresponding safeguards as a precaution.
Anthropic has similarly reported limitations from saturated CBRN benchmarks. And an analysis by Epoch AI found that most publicly described biorisk question-answering benchmarks had rapidly approached saturation. Yet scoring highly on questions about biology is not the same as being able to help someone develop a biological weapon. We are left in an uncomfortable position where models can ace the tests, while we remain uncertain about the capabilities those tests are supposed to measure.
Cybersecurity raises concerns for similar reasons. Anthropic previously restricted access to its Mythos-class of models due to their advanced cyber capabilities. Meanwhile, during internal evaluations in July 2026, an unreleased OpenAI model circumvented its evaluation sandbox and compromised parts of Hugging Face's systems.
The stakes are therefore quite different from deciding which model deserves the top spot on a leaderboard; we need to know not only how capable a given model is, but also how rapidly it approaches thresholds that could change how it should be deployed. A saturated benchmark tells us that a model is capable. It tells us much less about what we should do with the model.
There are, of course, other ways to evaluate capabilities besides benchmarks. Researchers can run red-teaming exercises, conduct human studies on uplift in productivity, or use additional model-based evaluations such as debate. While it is possible that these provide valuable information as well, they also require considerable time and resources to develop and iterate on. And with newer and better models arriving faster than ever, repeating increasingly expensive evaluations for every release becomes difficult.
But every benchmark run already produces something we mostly throw away: the transcripts of how the model got to its answers.
So we started there.
What we did
A benchmark typically reduces a model's performance to a single scalar value. But there's often much more information available. We could look at the sequence of actions an agent took, the tools it used, and the mistakes it made along the way. Even when two models reach the same outcome, the paths they take can look very different.
This opens up an interesting question. Rather than trying to think up harder and harder questions, could we get better estimates of capability from the questions models can already solve?
We took representative approaches to transcript analysis in the literature and turned each one into a protocol[1]. These protocols ranged from extracting simple deterministic features of the text, converting the text into embeddings, using an LLM judge that scores one transcript, and a pairwise LLM judge that compares two. Then we replayed each through history. For each period of time, we gave each protocol the previously released models and the transcripts that they correctly solved benchmark items, and asked it to predict how newer models would do on the harder items it never saw.
Pairwise comparisons recovered substantially more information than the benchmark scores alone. This tells us that transcripts from fully saturated benchmarks do carry capability-relevant signals – and that it can be extracted even by simple protocols.
But these approaches only held when we were looking at models with fairly similar architectures. When we looked at the period when the first reasoning models arrived, we found that every reasoning model was placed far below its true capability. Even after accounting for potential confounders such as transcript length, we found that protocols still failed to make accurate predictions, meaning that there was a fundamental change in the way capability showed up in a transcript, and there was nothing the current protocols could do under such a distribution shift.
The paradigm shift that reasoning models introduced was not a one-off event. Several shifts of the same kind are underway.
For one, reasoning is out of the game. Since OpenAI’s o1, most model providers have hidden their models’ chains of thought (CoT) or left users with a mere summary of the reasoning tokens that the model produced, and our experiments showed that removing reasoning tokens from the transcript removed a significant portion of the signal extracted by the protocols. In addition, recent models such as OpenAI’s Astra are reported to use a looped transformer, which can do more of its thinking in hidden states without producing any readable text. And even when independent researchers extracted proprietary reasoning tokens from encrypted traces, the reasoning was found to be in a largely distorted format.
In addition, specifically protocols that use LLM judges face multiple possible confounders that may affect the performance of the protocol tasks such as biases for certain positions and longer answers. Thus a protocol built on such a judge may face brittleness even in ordinary conditions, especially if the evaluated model was aware of its evaluation and used it to its advantage to either to look more capable than it is or to sandbag and look less capable. Even in terms of practical terms, the trend of longer horizon tasks means that a protocol that pastes the whole transcript into an LLM will face difficulties once transcripts no longer fit in a context window. As these are often agentic tasks that matter most for cyber and AI research, the concern is valid.
Moving towards scalable evaluations
Auditing and updating benchmarks is important, but it is not a permanent solution for when the updated benchmark is close to full on release. An evaluation that is cheap to run but consistently underestimates the next generation of models is not very useful. Neither is a benchmark that can be updated quickly but loses its ability to distinguish models within months.
Aggregating over existing models isn’t enough to show that the evaluation will be robust under distribution shifts. Reasoning models have already shown what happens when model behaviours change. The shift to long-horizon agents introduces longer sequences of actions and tool calls, some of which may not fit into a single judge’s context window. Models may even learn to recognise when they’re being evaluated, raising questions about whether their observed behaviour reflects their actual capabilities.
The challenge moving forward is therefore not only to make evaluations cheaper or harder. We need methods that can continue to extract useful information as the systems we're trying to measure change and evolve. Our proposal is thus a change in how we evaluate the methods we use to evaluate.
There are plenty of directions worth exploring. We could extract more information from successful trajectories, make evaluations adaptive to the model being tested, or develop methods that combine evidence from several imperfect benchmarks. We could also get better at quantifying uncertainty and variance in models, so that when an evaluation stops being informative, we know that it has.
We need evaluations that can keep pace with the frontier, not ones that measure where it used to be.
We use the term protocols purposely as any specific judge model could be out of date within months, in a bitter lesson-esque application to evaluation. The question we care about is how robust a specific protocol will be at providing accurate estimates of model capability, regardless of specific implementation. For example, the pairwise LLM judge protocol is set to use the best performing LLM model available during the given time period, as this would be the expectation for future usages of this protocol.
This research was completed for London AI Safety Research (LASR) Labs Summer 2026. The team was supervised by Justin Olive (Generality Labs) and Magda Dubois (UK AISI). Find out more about the programme and express interest in upcoming iterations here.
We encourage readers seeking technical details to read the paper on arxiv.
TLDR: We rely on benchmarks to estimate model capabilities. But now, many existing benchmarks are becoming too easy for frontier models – causing them to become saturated. For high-risk domains like cyber and bio, we cannot afford underestimating how capable models are in these domains. So how do we estimate capabilities after benchmarks have saturated? Running a benchmark already produces a transcript of how each model solved each task, meaning that the information within the transcript is just discarded. Using transcript analysis literature, we propose a set of protocols and replayed the last few years of model releases to see if they could predict capability beyond benchmark saturation. And they can, for the most part. But they also fail to generalise across “paradigm shifts” such as when reasoning models came out and models suddenly started to behave differently. Our main argument is that we need to move past evaluations that are judged by their average accuracy on familiar models, to methods for scalable evaluation that are judged on how robust they are across shifts like this one.
Saturated on day one
On September 22, Scale released SWE-Bench Pro V2, a cleaned-up version of one of the most widely used frontier software engineering benchmarks. On the public split, Claude Opus 5 solved 638 of 642 tasks. A benchmark intended to measure capability at the frontier was already at 99.4% on release.
But things didn’t always used to be this way.
We’ve of course had benchmarks that had outlived their usefulness before, but the pace at which it happens has changed dramatically. When GLUE became too easy, researchers built SuperGLUE in 2019 to raise the bar. It took roughly another year and a half, but SuperGLUE also saturated. Fast forward to January 2025, when Humanity's Last Exam was released. Thousands of domain experts took part to create what was meant to be the “frontier of human knowledge”, only for models to climb from initial scores of less than 10% to over 50% by 2026. It seems likely a matter of time before this benchmark is also saturated.
A recent systematic study of 60 benchmarks that major AI labs report in their technical reports found that nearly half of them already show signs of saturation.
The problem isn’t necessarily that models are “getting too good”. It’s that we lose our ability to track model capabilities as benchmarks saturate: if two models both solve 99% of the tasks successfully, their scores stop being useful for distinguishing their actual capabilities. One might be significantly more capable than the other, but we would have no way of knowing otherwise.
Of course, we can always build harder benchmarks. But that takes time. The SWE-Bench Pro V2 refresh alone required 23 engineers and nearly 1,900 hours of work, only for its public split to be nearly saturated on release. With the pace at which new models are being released, continually replacing benchmarks with harder ones is an increasingly difficult way to keep track of progress.
Why safety cannot live without a score
In terms of building leaderboards and pointing at the newest shiny model, saturation is just an inconvenience. However, there are deeper consequences for safety considerations. Safety frameworks and regulations are built on capability thresholds, meaning a lab decides what safeguards a model needs by checking where its capability sits and regulators decide what constraints should be placed in the development and usage of these models using similar thresholds.
In this regard, the only signal we would get from saturated benchmark scores is that a model is… “good”.
In domains where this matters, we're pretty much already in this regime of uncertainty. When OpenAI released GPT-5, it classified the model as having “High” biological and chemical capability under its Preparedness Framework, even though its evaluations did not provide definitive evidence that the model had crossed that threshold. Rather than risk underestimating its capabilities, OpenAI decided to apply the corresponding safeguards as a precaution.
Anthropic has similarly reported limitations from saturated CBRN benchmarks. And an analysis by Epoch AI found that most publicly described biorisk question-answering benchmarks had rapidly approached saturation. Yet scoring highly on questions about biology is not the same as being able to help someone develop a biological weapon. We are left in an uncomfortable position where models can ace the tests, while we remain uncertain about the capabilities those tests are supposed to measure.
Cybersecurity raises concerns for similar reasons. Anthropic previously restricted access to its Mythos-class of models due to their advanced cyber capabilities. Meanwhile, during internal evaluations in July 2026, an unreleased OpenAI model circumvented its evaluation sandbox and compromised parts of Hugging Face's systems.
The stakes are therefore quite different from deciding which model deserves the top spot on a leaderboard; we need to know not only how capable a given model is, but also how rapidly it approaches thresholds that could change how it should be deployed. A saturated benchmark tells us that a model is capable. It tells us much less about what we should do with the model.
There are, of course, other ways to evaluate capabilities besides benchmarks. Researchers can run red-teaming exercises, conduct human studies on uplift in productivity, or use additional model-based evaluations such as debate. While it is possible that these provide valuable information as well, they also require considerable time and resources to develop and iterate on. And with newer and better models arriving faster than ever, repeating increasingly expensive evaluations for every release becomes difficult.
But every benchmark run already produces something we mostly throw away: the transcripts of how the model got to its answers.
So we started there.
What we did
A benchmark typically reduces a model's performance to a single scalar value. But there's often much more information available. We could look at the sequence of actions an agent took, the tools it used, and the mistakes it made along the way. Even when two models reach the same outcome, the paths they take can look very different.
This opens up an interesting question. Rather than trying to think up harder and harder questions, could we get better estimates of capability from the questions models can already solve?
We took representative approaches to transcript analysis in the literature and turned each one into a protocol[1]. These protocols ranged from extracting simple deterministic features of the text, converting the text into embeddings, using an LLM judge that scores one transcript, and a pairwise LLM judge that compares two. Then we replayed each through history. For each period of time, we gave each protocol the previously released models and the transcripts that they correctly solved benchmark items, and asked it to predict how newer models would do on the harder items it never saw.
Pairwise comparisons recovered substantially more information than the benchmark scores alone. This tells us that transcripts from fully saturated benchmarks do carry capability-relevant signals – and that it can be extracted even by simple protocols.
But these approaches only held when we were looking at models with fairly similar architectures. When we looked at the period when the first reasoning models arrived, we found that every reasoning model was placed far below its true capability. Even after accounting for potential confounders such as transcript length, we found that protocols still failed to make accurate predictions, meaning that there was a fundamental change in the way capability showed up in a transcript, and there was nothing the current protocols could do under such a distribution shift.
The paradigm shift that reasoning models introduced was not a one-off event. Several shifts of the same kind are underway.
For one, reasoning is out of the game. Since OpenAI’s o1, most model providers have hidden their models’ chains of thought (CoT) or left users with a mere summary of the reasoning tokens that the model produced, and our experiments showed that removing reasoning tokens from the transcript removed a significant portion of the signal extracted by the protocols. In addition, recent models such as OpenAI’s Astra are reported to use a looped transformer, which can do more of its thinking in hidden states without producing any readable text. And even when independent researchers extracted proprietary reasoning tokens from encrypted traces, the reasoning was found to be in a largely distorted format.
In addition, specifically protocols that use LLM judges face multiple possible confounders that may affect the performance of the protocol tasks such as biases for certain positions and longer answers. Thus a protocol built on such a judge may face brittleness even in ordinary conditions, especially if the evaluated model was aware of its evaluation and used it to its advantage to either to look more capable than it is or to sandbag and look less capable. Even in terms of practical terms, the trend of longer horizon tasks means that a protocol that pastes the whole transcript into an LLM will face difficulties once transcripts no longer fit in a context window. As these are often agentic tasks that matter most for cyber and AI research, the concern is valid.
Moving towards scalable evaluations
Auditing and updating benchmarks is important, but it is not a permanent solution for when the updated benchmark is close to full on release. An evaluation that is cheap to run but consistently underestimates the next generation of models is not very useful. Neither is a benchmark that can be updated quickly but loses its ability to distinguish models within months.
Aggregating over existing models isn’t enough to show that the evaluation will be robust under distribution shifts. Reasoning models have already shown what happens when model behaviours change. The shift to long-horizon agents introduces longer sequences of actions and tool calls, some of which may not fit into a single judge’s context window. Models may even learn to recognise when they’re being evaluated, raising questions about whether their observed behaviour reflects their actual capabilities.
The challenge moving forward is therefore not only to make evaluations cheaper or harder. We need methods that can continue to extract useful information as the systems we're trying to measure change and evolve. Our proposal is thus a change in how we evaluate the methods we use to evaluate.
There are plenty of directions worth exploring. We could extract more information from successful trajectories, make evaluations adaptive to the model being tested, or develop methods that combine evidence from several imperfect benchmarks. We could also get better at quantifying uncertainty and variance in models, so that when an evaluation stops being informative, we know that it has.
We need evaluations that can keep pace with the frontier, not ones that measure where it used to be.
We use the term protocols purposely as any specific judge model could be out of date within months, in a bitter lesson-esque application to evaluation. The question we care about is how robust a specific protocol will be at providing accurate estimates of model capability, regardless of specific implementation. For example, the pairwise LLM judge protocol is set to use the best performing LLM model available during the given time period, as this would be the expectation for future usages of this protocol.