We must take great care not to ignore the things that are not easily quantified - Brian Christian, The Alignment Problem
Introduction
Model evaluations have a problem. This isn't news [1]- the AI safety and research fields have known for years that current approaches to assessing the capability and safety of frontier models are limited. That's not to say that evaluations aren't important; in fact, they're essential to understanding how to make AI systems safe. The question is not, how do we replace existing approaches to evaluation? Instead, it's how do we complement those approaches, and make evaluations as a whole more robust and useful?
In this short piece, I begin by reconsidering the limitations of the current evaluation paradigm. Then, I dig into recent examples that illustrate these limits, explore their consequences for both capability and safety evaluations, and explain what I think prevailing approaches miss. In the final section, I present a few initial ideas about what a more diverse evaluation toolkit might include, and what using such tools might make possible.
Errors of category, not (just) execution
Let's start with one of the most common, and commonly reported on, form of capability evaluations: benchmarks. There are many existing critiques of our current suite of benchmarks that I won't recap here (for example, see Sean McGregor and David Manheim). But common failures include:
Saturation: as models improve, they quickly master existing benchmarks, making it difficult to fully assess frontier models' capability, or compare capabilities of different models
Reliability: benchmarks often use proxies for capability that don't generalize to situations beyond training data, nor actually help us predict how a model might perform in the real world
Gameability: models regularly end up gaming benchmarks, intentionally (by finding the solution in their training sets, through hillclimbing, etc) or otherwise - this is especially problematic as frontier models are increasingly exhibiting evaluation awareness, and more broadly, situational awareness.
These failure types are due to execution, reflecting a gap between the way benchmarks are designed and operationalized and the measurements they produce. As a field, we know all about these execution errors, and there are lots of brilliant people already working to address them. Benchrisk, for example, categorizes and supports efforts to mitigate the risk inherent in any given benchmark. Epoch’s Capabilities Index (ECI) combines leading benchmarks into a composite index to provide a broader and deeper assessment of model capability. And MirrorCode, Long-Horizon Terminal Bench, and METR’s Time Horizons seek to measure capability in terms of the length of the tasks that models can complete.
New and improved benchmarks are improving all the time - in July alone, Edgebench and Awareness Bench are two of the more interesting new approaches to have launched. But even more sophisticated benchmarks remain, fundamentally, point-in-time scores. They offer fuller, but still incomplete, assessments of capability. And even if they were perfectly designed and implemented, such that scores really corresponded to performance on the targeted capability in the real world, they would still suffer from a fundamental limitation: they are decontextualized, isolated from the context, conditions, and causal mechanisms that would help us understand what the capability actually means, including for safety. This is because benchmarks suffer not just from execution errors, but from a broader category error. And this type of category error is not limited to benchmarks. In fact, it's a challenge for capability evaluations in general, and for model evaluations more broadly.
This isn't news to the AI eval community - the limits of scores are already well understood (and have motivated some of the newer approaches mentioned above). But what I'm claiming is that the issue here isn't solely a problem with measurement. Instead, it has to do with being able to understand not just what scores and behaviors emerge, but how they do so, for what reasons, and under what conditions. Other evaluation disciplines have taken these questions up, but have not yet been widely adapted to the field of AI.
The tendency to try and measure performance and behavior in isolation has real-world consequences, as illustrated by two recent disclosures from OpenAI. These cases do not make for a representative, systematic survey of the entire field. But they help show how both capability and safety evaluations, and the scores and judgments they produce, often fail to capture the context, conditions, and mechanisms that would enable us to substantively make sense of them.
Capability evaluation: scores in isolation
Last week's OpenAI / Hugging Face incident has been covered widely, so I won't recap it in detail here. The simplified version, which is sufficient for our purposes, goes something like this: ExploitGym is a capability evaluation (specifically, a benchmark) that aims to measure a model's offensive-cyber abilities. While being tested on ExploitGym, a new OpenAI frontier model - with its cyber safeguards turned off, to enable a full appraisal of its performance - broke out of a sandboxed environment, hacked into Hugging Face, and tried to steal ExploitGym's solutions set as it sought to ace the test.[2] If we were focusing exclusively on performance against the benchmark, we might have seen the model achieve a perfect score, fully saturating ExploitGym. But considered in context, that perfect score is the result of an extremely harrowing security incident.
In other words, performance against the benchmark and what actually happened in reality, during testing, contradict each other: the former suggests the model was perfect, but the latter is cause for real alarm. This category error manifests consistently in capability evaluations, even good ones, across several dimensions.
For example, consider external validity: MirrorCode, as noted above, is a great attempt to assess models' ability to perform long-horizon tasks. But even MirrorCode, according to the authors themselves, can't help us predict the extent to which performance against the test will generalize to unknown codebases - MirrorCode scores don't clearly translate into capability in context.
Or consider that the OpenAI/Hugging Face incident could have resulted in a perfect ExploitGym score, but that would have said very little about the conditions under which that score emerged: cyber safeguards were disabled, the model was relentlessly optimizing for a narrow goal, etc. It's these conditions that help us understand what the capability actually means. Is it benign or deeply alarming? The benchmark, considered on its own, leaves us effectively blind on that front.
Finally, consider issues of trajectory and pace: benchmark scores, plotted over time, give us a sense of how quickly AI systems' capabilities are improving, and to what extent. But they don't help us get at the underlying causal mechanisms that are driving that progress, or generate the evidence we need to adjudicate different views on how quickly something like recursive self-improvement might emerge.
The same category error affects safety evaluations too - and arguably, the stakes there are even higher.
Safety evaluation: verdicts in isolation
Safety evaluations, as opposed to capability evals, ask whether a model is behaving as intended. But despite having a different focus, they too are often implemented in environments unlike real-world contexts, and similarly struggle to help us accurately predict what models will actually do once they are deployed.
Another recent OpenAI report provides a useful illustration of these dynamics. They explain how, once deployed in internal environments, a new frontier model displayed behaviors that had not emerged during pre-deployment evals (breaking out of sandboxes in pursuit of narrowly defined goals, e.g., and splitting tokens to evade a scanner). Even more strikingly, once deployed internally, the model was seen to string together sequences of actions that, considered together, suggest misaligned behavior, even though each individual action wouldn't raise alarm bells on its own. The safety evaluations, that is, were not adequate for assessing what the model would actually do in long-horizon deployments. They were measuring safety-in-isolation, not safety-in-context (though they were at least monitoring the model in context, after it was deployed for internal use, to some extent - though the rigor of that monitoring seems questionable - at best[3]).
This isn't because OpenAI was executing safety evaluations poorly (though they may have been!). It's because the existing safety evaluation approaches are categorically incomplete, just like capability evals. OpenAI's own researchers admit as much, saying: "no fixed evaluation suite can anticipate every behavior."
So where does that leave us? Whether we're scoring capability, or assessing safety, our existing eval approaches - as robust and technically impressive as they are - risk collapsing complex, context-dependent, and mechanism-driven capabilities and behaviors into a reductive number or judgment. They fail to reflect the conditions under which that number or judgment holds, or does not.
Evaluations are essential to our ability to understand AI systems and minimize risk. And even flawed or incomplete evals are better than none at all. But to really deliver the evidence the field needs to understand and ultimately shape the capability and behavior of models to maximize safety, new, complementary approaches are needed.
Toward a richer toolkit for model evaluation
What would it look like to evaluate capability and behavior in context - surfacing not just a score, but the in-context conditions and mechanisms underneath? These are questions that social scientists, strategists, and evaluators working in complex systems have grappled with for decades - and are still grappling with today (as one such strategist and evaluator, I can tell you, the struggle continues!). We don't have perfect answers - but we do have a robust toolkit of approaches and perspectives, from realist evaluation to complexity and systems thinking, to strategic foresight, that are all about understanding and assessing what outcomes emerge, in what context, under what conditions, and through which mechanisms. Many of these approaches have not yet been adapted and applied to the AI eval landscape. Figuring out how to draw on them could help reduce the category error that inhibits our AI evaluations today.
What would embracing a more diverse toolkit for model evaluation look like? Consider three potential ideas:
combining systems thinking and strategic foresight to identify, test, and update the field's key assumptions about the pace of AI system development, including key drivers and causal mechanisms
taking a richer approach to user-focused uplift studies, systematically collecting and synthesizing qualitative evidence - in depth key informant interviews with users, for example - alongside existing RCT-based approaches, to better unpack whether/how user capabilities improve, under what conditions, and via what mechanisms
applying realist and developmental evaluation approaches to alignment research itself, working with an alignment team to surface and test implicit assumptions about when, how, why, and under what conditions alignment holds, drawing on both traditional and non-traditional evidence (benchmarks,transcripts, researchers' tacit insights, etc.)
This shift towards a richer, more multifaceted toolkit that integrates insights from multiple disciplines is already in progress. For example, the ADELE approach tries to apply behavioral science to assess a model's propensity to engage in certain behaviors; Compass Eval seeks to assess models’ moral reasoning; and EdgeBench (mentioned above) digs into agents’ ability to learn as they work on long-horizon projects in real-world environments.
Adapting and applying evaluation approaches from other disciplines to model evals is not straightforward, posing a variety of challenges, including with regard to scalability, reproducibility, safety, and data contamination risks. But working in this area could offer avenues for figuring out how to better understand capabilities, behaviors, and risks. Building a more diverse set of evaluation approaches, ultimately, is vital for safety and governance: to make good decisions, we have to know not just what models score, and what behaviors they exhibit, but why, when, and under what conditions. A richer evaluation toolkit can help us do just that.
Big thanks to Shivani Shukla, Anusree Garg, and Kecia Bertermann for their helpful comments and feedback on earlier drafts of this post. Any errors are mine alone.
Though the news does keep getting worse. This piece was initially drafted on Friday, July 24, and between now and then, we’ve seen a steady drip of newly disclosed evaluation failures, with Anthropic’s disclosure last night the latest example.
In perpetuity, no less. See here, here, and here for more on this incident. I should note, new reporting continues to drop, and the story keeps getting more alarming.
Reuters explains more here. Even though OpenAI hasn’t confirmed that the model described in their alignment article is the same one that hacked Hugging Face, it seems highly likely that is in fact the case.
Introduction
Model evaluations have a problem. This isn't news [1]- the AI safety and research fields have known for years that current approaches to assessing the capability and safety of frontier models are limited. That's not to say that evaluations aren't important; in fact, they're essential to understanding how to make AI systems safe. The question is not, how do we replace existing approaches to evaluation? Instead, it's how do we complement those approaches, and make evaluations as a whole more robust and useful?
In this short piece, I begin by reconsidering the limitations of the current evaluation paradigm. Then, I dig into recent examples that illustrate these limits, explore their consequences for both capability and safety evaluations, and explain what I think prevailing approaches miss. In the final section, I present a few initial ideas about what a more diverse evaluation toolkit might include, and what using such tools might make possible.
Errors of category, not (just) execution
Let's start with one of the most common, and commonly reported on, form of capability evaluations: benchmarks. There are many existing critiques of our current suite of benchmarks that I won't recap here (for example, see Sean McGregor and David Manheim). But common failures include:
These failure types are due to execution, reflecting a gap between the way benchmarks are designed and operationalized and the measurements they produce. As a field, we know all about these execution errors, and there are lots of brilliant people already working to address them. Benchrisk, for example, categorizes and supports efforts to mitigate the risk inherent in any given benchmark. Epoch’s Capabilities Index (ECI) combines leading benchmarks into a composite index to provide a broader and deeper assessment of model capability. And MirrorCode, Long-Horizon Terminal Bench, and METR’s Time Horizons seek to measure capability in terms of the length of the tasks that models can complete.
New and improved benchmarks are improving all the time - in July alone, Edgebench and Awareness Bench are two of the more interesting new approaches to have launched. But even more sophisticated benchmarks remain, fundamentally, point-in-time scores. They offer fuller, but still incomplete, assessments of capability. And even if they were perfectly designed and implemented, such that scores really corresponded to performance on the targeted capability in the real world, they would still suffer from a fundamental limitation: they are decontextualized, isolated from the context, conditions, and causal mechanisms that would help us understand what the capability actually means, including for safety. This is because benchmarks suffer not just from execution errors, but from a broader category error. And this type of category error is not limited to benchmarks. In fact, it's a challenge for capability evaluations in general, and for model evaluations more broadly.
This isn't news to the AI eval community - the limits of scores are already well understood (and have motivated some of the newer approaches mentioned above). But what I'm claiming is that the issue here isn't solely a problem with measurement. Instead, it has to do with being able to understand not just what scores and behaviors emerge, but how they do so, for what reasons, and under what conditions. Other evaluation disciplines have taken these questions up, but have not yet been widely adapted to the field of AI.
The tendency to try and measure performance and behavior in isolation has real-world consequences, as illustrated by two recent disclosures from OpenAI. These cases do not make for a representative, systematic survey of the entire field. But they help show how both capability and safety evaluations, and the scores and judgments they produce, often fail to capture the context, conditions, and mechanisms that would enable us to substantively make sense of them.
Capability evaluation: scores in isolation
Last week's OpenAI / Hugging Face incident has been covered widely, so I won't recap it in detail here. The simplified version, which is sufficient for our purposes, goes something like this: ExploitGym is a capability evaluation (specifically, a benchmark) that aims to measure a model's offensive-cyber abilities. While being tested on ExploitGym, a new OpenAI frontier model - with its cyber safeguards turned off, to enable a full appraisal of its performance - broke out of a sandboxed environment, hacked into Hugging Face, and tried to steal ExploitGym's solutions set as it sought to ace the test.[2] If we were focusing exclusively on performance against the benchmark, we might have seen the model achieve a perfect score, fully saturating ExploitGym. But considered in context, that perfect score is the result of an extremely harrowing security incident.
In other words, performance against the benchmark and what actually happened in reality, during testing, contradict each other: the former suggests the model was perfect, but the latter is cause for real alarm. This category error manifests consistently in capability evaluations, even good ones, across several dimensions.
For example, consider external validity: MirrorCode, as noted above, is a great attempt to assess models' ability to perform long-horizon tasks. But even MirrorCode, according to the authors themselves, can't help us predict the extent to which performance against the test will generalize to unknown codebases - MirrorCode scores don't clearly translate into capability in context.
Or consider that the OpenAI/Hugging Face incident could have resulted in a perfect ExploitGym score, but that would have said very little about the conditions under which that score emerged: cyber safeguards were disabled, the model was relentlessly optimizing for a narrow goal, etc. It's these conditions that help us understand what the capability actually means. Is it benign or deeply alarming? The benchmark, considered on its own, leaves us effectively blind on that front.
Finally, consider issues of trajectory and pace: benchmark scores, plotted over time, give us a sense of how quickly AI systems' capabilities are improving, and to what extent. But they don't help us get at the underlying causal mechanisms that are driving that progress, or generate the evidence we need to adjudicate different views on how quickly something like recursive self-improvement might emerge.
The same category error affects safety evaluations too - and arguably, the stakes there are even higher.
Safety evaluation: verdicts in isolation
Safety evaluations, as opposed to capability evals, ask whether a model is behaving as intended. But despite having a different focus, they too are often implemented in environments unlike real-world contexts, and similarly struggle to help us accurately predict what models will actually do once they are deployed.
Another recent OpenAI report provides a useful illustration of these dynamics. They explain how, once deployed in internal environments, a new frontier model displayed behaviors that had not emerged during pre-deployment evals (breaking out of sandboxes in pursuit of narrowly defined goals, e.g., and splitting tokens to evade a scanner). Even more strikingly, once deployed internally, the model was seen to string together sequences of actions that, considered together, suggest misaligned behavior, even though each individual action wouldn't raise alarm bells on its own. The safety evaluations, that is, were not adequate for assessing what the model would actually do in long-horizon deployments. They were measuring safety-in-isolation, not safety-in-context (though they were at least monitoring the model in context, after it was deployed for internal use, to some extent - though the rigor of that monitoring seems questionable - at best[3]).
This isn't because OpenAI was executing safety evaluations poorly (though they may have been!). It's because the existing safety evaluation approaches are categorically incomplete, just like capability evals. OpenAI's own researchers admit as much, saying: "no fixed evaluation suite can anticipate every behavior."
So where does that leave us? Whether we're scoring capability, or assessing safety, our existing eval approaches - as robust and technically impressive as they are - risk collapsing complex, context-dependent, and mechanism-driven capabilities and behaviors into a reductive number or judgment. They fail to reflect the conditions under which that number or judgment holds, or does not.
Evaluations are essential to our ability to understand AI systems and minimize risk. And even flawed or incomplete evals are better than none at all. But to really deliver the evidence the field needs to understand and ultimately shape the capability and behavior of models to maximize safety, new, complementary approaches are needed.
Toward a richer toolkit for model evaluation
What would it look like to evaluate capability and behavior in context - surfacing not just a score, but the in-context conditions and mechanisms underneath? These are questions that social scientists, strategists, and evaluators working in complex systems have grappled with for decades - and are still grappling with today (as one such strategist and evaluator, I can tell you, the struggle continues!). We don't have perfect answers - but we do have a robust toolkit of approaches and perspectives, from realist evaluation to complexity and systems thinking, to strategic foresight, that are all about understanding and assessing what outcomes emerge, in what context, under what conditions, and through which mechanisms. Many of these approaches have not yet been adapted and applied to the AI eval landscape. Figuring out how to draw on them could help reduce the category error that inhibits our AI evaluations today.
What would embracing a more diverse toolkit for model evaluation look like? Consider three potential ideas:
This shift towards a richer, more multifaceted toolkit that integrates insights from multiple disciplines is already in progress. For example, the ADELE approach tries to apply behavioral science to assess a model's propensity to engage in certain behaviors; Compass Eval seeks to assess models’ moral reasoning; and EdgeBench (mentioned above) digs into agents’ ability to learn as they work on long-horizon projects in real-world environments.
Adapting and applying evaluation approaches from other disciplines to model evals is not straightforward, posing a variety of challenges, including with regard to scalability, reproducibility, safety, and data contamination risks. But working in this area could offer avenues for figuring out how to better understand capabilities, behaviors, and risks. Building a more diverse set of evaluation approaches, ultimately, is vital for safety and governance: to make good decisions, we have to know not just what models score, and what behaviors they exhibit, but why, when, and under what conditions. A richer evaluation toolkit can help us do just that.
Big thanks to Shivani Shukla, Anusree Garg, and Kecia Bertermann for their helpful comments and feedback on earlier drafts of this post. Any errors are mine alone.
Though the news does keep getting worse. This piece was initially drafted on Friday, July 24, and between now and then, we’ve seen a steady drip of newly disclosed evaluation failures, with Anthropic’s disclosure last night the latest example.
In perpetuity, no less. See here, here, and here for more on this incident. I should note, new reporting continues to drop, and the story keeps getting more alarming.
Reuters explains more here. Even though OpenAI hasn’t confirmed that the model described in their alignment article is the same one that hacked Hugging Face, it seems highly likely that is in fact the case.