Today’s AI systems frequently behave in ways their developers did not anticipate or intend. Furthermore, as these systems become increasingly capable and widely deployed, these unexpected behaviors can have real consequences. In one well-known case from July 2025, Replit’s coding agent deleted a startup’s production database during an explicit code freeze, ignoring repeated instructions and wiping records for over a thousand companies. Other cases involve high-stakes interactions between models and human users: a man’s family alleges that ChatGPT acted as his confidant and turned his favorite childhood book, Goodnight Moon, into a “suicide lullaby” in the weeks before he took his own life. This raises a key challenge for our field: how can we keep these systems behaving safely as they act in the world?
The history of machine learning suggests one answer to this question: to shape the behavior of an AI system, you should start by measuring it. From the introduction of ImageNet in 2009 to the recent success of reinforcement learning on tasks with verifiable rewards, progress toward more capable systems has been driven by hill-climbing on tasks where we can measure performance. Furthermore, just as ImageNet made it possible for researchers across the field to fairly compare model architectures for image classification, today’s public capability benchmarks like SWE-bench and Terminal-Bench allow independent actors to validate that developers' claims about model capabilities are true, and provide trusted public leaderboards that guide consumer choices and incentivize model developers in turn.
To use this insight to keep AI systems behaving safely in real-world deployments, we need ways to define and measure what “safe” behavior looks like, and ways to recognize hazards before failures arise. But behaviors we might wish to measure are inherently “fuzzier” than capabilities. It is difficult to write down what appropriate behavior should be in open-ended deployment environments that lack the predefined notion of a correct answer, such as a mental health-relevant conversation with a vulnerable user, and even harder to reliably determine whether a model’s actions should count as an instance of that behavior. To make sense of these behaviors, the field will need to develop best practices for behavior evaluations that operationalize hard-to-specify behaviors into measurable quantities we can track consistently across models. As a starting point, we describe the components of these evaluations in Measuring Model Behavior as a Scientific Practice, and make recommendations for handling the complexities of real-world deployments.
Even so, high-quality behavior evaluations are insufficient in isolation: ensuring systems behave safely in deployment requires coordination at an ecosystem-level. The AI ecosystem is changing fast, and new failure modes surface continually as models are deployed across increasingly dynamic environments. To keep pace, we need to be able to rapidly adapt our measurements to account for changing dynamics of AI systems in practice, across changes to model weights, developer safeguards, agent harnesses, and deployment environments. In Public Measurement Infrastructure for Collective Sensemaking, we describe how shared infrastructure would enable many independent actors to contribute measurements that can be compared and built upon, helping us collectively predict how close we are to surprising edge cases or failure modes as new behaviors emerge.
Measuring model behavior as a scientific practice
How can we systematically measure model behaviors? Similarly to how capability evaluations include a collection of representative tasks and a way to measure whether a model can solve them, behavior evaluations should include a set of representative situations where a model could plausibly demonstrate a behavior and a way to determine how often a model demonstrates the behavior of interest. However, it is not always obvious how to construct these pieces to provide meaningful and consistent signals about how models will behave in the real world, and existing studies of model behavior are fairly ad hoc. Moving from where we are today to a robust science of model behavior will require aligning on the core components of behavior evaluations, so shared tools and standards of evidence can be built around them. As a starting point, we believe these core components include: ecologically valid environment simulators that place the model in deployment-relevant scenarios, robust automated judging procedures that construct reliable judges for detecting behaviors, and mechanisms for making meaningful comparisons across systems.
Ecologically valid environment simulators
The first step in building a model behavior evaluation is setting up a controlled “environment simulator” where the model’s behavior can be observed and recorded under consistent conditions. For instance, when evaluating how a chatbot interacts with vulnerable users, the environment simulator would be responsible for simulating the user’s side of the conversation. Because the same experimental conditions can be “replayed” across different models, these environment simulators let us make comparative measurements of model behavior, which is difficult in the wild where deployment conditions differ from system to system.
For a behavior evaluation to help predict the impact of a model, or forecast the future behavior of similar models, it is important that these environment simulators be informative about what will actually happen outside of the simulation. There are at least two complementary ways to achieve this:
One strategy is to ground them in current (or anticipated) real-world usage, and work to make each simulated environment as realistic as possible. This allows us to study realistic model behaviors in a controlled setting, and can inform us about what models will actually do when deployed. (As one recent example of this strategy, the UK AI Safety Institute’s evaluation of whether AI models would sabotage AI safety research involved running models in a real Claude Code harness inside a real codebase. As another, OpenAI's production evaluations resample model responses in contexts drawn from de-identified user traffic, to create evaluation scenarios reflecting real deployment.)
Another strategy is to explore a wide range of input conditions, and isolate which components are responsible for the observed behavior. This can reveal model-specific patterns of behavior that would not be apparent if only a single scenario was used, and can help us predict how generalizable the findings are. (As an example of this strategy, a study by Sheshadri et al. [2025] systematically probed the “alignment faking” behaviors of a variety of models, and determined that the only model whose behavior patterns consistently related to keeping its own goals was Claude 3 Opus).
Reliable automated judging procedures
The next step in building a behavior evaluation is to take the recorded transcripts of a model in each simulated environment and measure how often particular behaviors occur. However, unlike building capability evaluations (which often have a “correct answer” or “reward function” by construction), building a behavior evaluation generally involves picking a behavior to measure as a proxy for some more general tendency, impact, or anticipated risk when the model is deployed. Measuring such a behavior entails crafting a rubric that defines the behavior as objectively as possible, then using another AI system equipped with that rubric to automatically judge whether the behavior occurred in a given transcript.
For these automated judging procedures to be reliable signals about the real-world impacts of the model, we think it is important that they satisfy a few properties:
First, it is important that each judging procedure and behavior rubric faithfully reflect the intended concept, and serve as a good automated proxy for human judgement. To achieve this, it is important that the construction of the judges and rubrics are informed by concrete examples of how the model behaves, since this often surfaces edge cases or exceptions that are difficult to anticipate in advance.
Second, it is important that the overall measurement apparatus is responsive to new behaviors that can emerge as new models are released, even when the environment simulators are not changed. New models frequently exhibit unique behavioral tendencies that may not have been present at all in previous models but can have significant implications for deployment or for interpreting the validity of results. To help us respond to these changes, we think it will be important to develop automated tooling and diagnostics that can discover new behaviors and surface them for human review, so that we can react quickly to changes in model behavior before they cause problems in the real world.
Comparative, reproducible measurements
Finally, for measured differences to be meaningful and useful for decision-making, they must be performed consistently. This includes both consistency across models, so that we can compare behavior across providers, and also consistency over time, enabling us to track and monitor changes. This is already the norm for many important capability benchmarks, but many behavior evaluations are either cross-model snapshots at a particular point in time (making it difficult to monitor progress), or system card evaluations that cover only a single developer’s model family.
We note that consistency does not imply that such evaluations must be completely static, and in practice we expect that judging procedures and similar components will need to be updated as old models are retired and new ones are introduced. Rather, we think it is important that the overall methodology and operationalization of the domain are well-specified enough that evaluations can be easily re-run one change at a time, and provide a continually-updated view of how models behave at any given moment.
Public measurement infrastructure for collective sensemaking
Although labs report some information about their models’ behaviors in system cards[1], measurement methodology varies substantially, and there is rarely enough information provided to meaningfully hold model developers accountable based on their own self-reports. At the same time, new models with new behavior patterns are being trained and released at an unprecedented pace, making it difficult for individual independent evaluators to keep up. For meaningful public oversight to be possible, the community will need to be able to rapidly understand how models will behave in new domains, discover behaviors we’re currently missing, surface gaps in our existing evaluations, and enable open innovation on evaluation methodology. These challenges call for a public scientific ecosystem built around discovering, understanding, and monitoring model behaviors as they emerge, allowing us to hold labs publicly accountable for the behaviors of the models they release.
Features of a public scientific ecosystem
Domain experts should be empowered to expand evaluation coverage and representativeness: No static evaluation can fully cover a dynamic domain, and individual attempts to evaluate model behavior in a simulated environment are likely to miss important features of real-world deployments. It is thus important to provide tools that people with domain knowledge can use to build evaluations that are better proxies for real-world usage. This would help us ensure that model behavior in these evaluations informs us about how models will behave in the real world.
Behavior data should be open so that readers can notice and capture missing model behaviors: In any given domain, there may be many different model behaviors that would be important to track and measure, and many possible explanations for the model’s behavior in each case. To account for this, rather than each evaluation measuring a fixed set of behaviors and producing numeric aggregate metrics, we think it is important to enable members of the public to notice new behaviors in existing evaluation data or existing simulated environments, and capture them as measurements that could be tracked alongside the original metrics.
It should be possible to challenge evaluations that are incomplete or misleading, and propose improvements: To ensure evaluations are trustworthy, the measurement methodology should be open to public scrutiny. When evaluations do not paint a faithful picture of how a model actually behaves, the general public should be able to contribute by pointing out flaws in the experimental design, proposing improvements to either the environments or behavior measurements, and measuring how robust the original findings are to these changes.
Independent researchers should be able to contribute new methods: Finally, we think it is important to enable collaborators to innovate on the structure of behavior evaluations, and propose new evaluation methodologies that allow us to understand behaviors at a deeper level. This could look like contributing new types of simulated environments, adding diagnostics for cross-cutting challenges to validity such as evaluation awareness, or even proposing approaches for measuring hard-to-characterize behaviors such as introspection and model self-concepts.
Supporting this ecosystem
Many of these activities are individually possible on top of existing tools and platforms, but often require substantial effort, and there is considerable friction involved in both building new behavior evaluations and validating or improving existing ones. Additionally, the lack of common infrastructure for performing these activities makes it difficult to build on top of established best practices and reason about the findings consistently.
What kind of public infrastructure would reduce this friction and enable ecosystem-level coordination? First, it should support interrogation of evaluations and their underlying data, so third parties can validate the methodology and point out flaws. Additionally, it should include a public square where participants can discuss results, contribute improvements to the original evaluations, or fork and extend them with follow-up results. Finally, we think it is important that this infrastructure allows evaluations to be expressed in terms of modular, reproducible components that can be recombined into new evaluations (for instance, using an existing set of rubrics to judge transcripts generated by new environment simulators, or applying a new behavior discovery technique to previously generated transcripts). This lets improvements compose with one another and encourages shared best practices across domains, so the community can reason consistently about evaluations, make sense of the findings together, and hold labs accountable when their ad hoc evals do not.
Looking ahead
The AI ecosystem is a complex distributed system, with many layers of safeguards at both technical and organizational levels, and its impacts are shaped by many independent actors both inside and outside AI labs. These types of systems are never failure-free, but rather involve constantly-changing mixtures of potential failures that must be continuously monitored and corrected for. As Richard I. Cook observed in “How Complex Systems Fail”:
Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. … Because system operations are never trouble free, human practitioner adaptations to changing conditions actually create safety from moment to moment. … Improved safety depends on providing operators with calibrated views of the hazards.
We believe it is essential to build tools that give AI system operators and society calibrated views of AI behavior, making it possible to detect and mitigate issues before catastrophic failure–especially in new domains where the consequences are hardest to predict. Such tools might enable us to catch behaviors in simulation, like a model's tendency to poeticize the distress of vulnerable users, before real-world damage is done. And measurement need not be limited to catching failures: qualities like whether a system increases human agency or improves epistemics are difficult to measure today, but a maturing science of model behavior could bring them more within reach. We are hopeful that with a robust scientific ecosystem of model understanding, the community will develop best practices for surfacing and responding to new model behaviors, better preparing us for the emergent properties of AI systems in the months and years to come.
We would like to thank Jacob Steinhardt, Tim Hua, Rob Friel, Neil Chowdhury, Wojciech Zaremba, Yonadav Shavit, David Duvenaud, and Alec Radford for their feedback on an earlier draft of this essay. We would also like to thank D. Sculley, Sam Bowman, and Sam Marks for useful discussions.
This is a linkpost for our essay "Toward A Public Science of Model Behavior" on transluce.org. The full text is reproduced below.
Today’s AI systems frequently behave in ways their developers did not anticipate or intend. Furthermore, as these systems become increasingly capable and widely deployed, these unexpected behaviors can have real consequences. In one well-known case from July 2025, Replit’s coding agent deleted a startup’s production database during an explicit code freeze, ignoring repeated instructions and wiping records for over a thousand companies. Other cases involve high-stakes interactions between models and human users: a man’s family alleges that ChatGPT acted as his confidant and turned his favorite childhood book, Goodnight Moon, into a “suicide lullaby” in the weeks before he took his own life. This raises a key challenge for our field: how can we keep these systems behaving safely as they act in the world?
The history of machine learning suggests one answer to this question: to shape the behavior of an AI system, you should start by measuring it. From the introduction of ImageNet in 2009 to the recent success of reinforcement learning on tasks with verifiable rewards, progress toward more capable systems has been driven by hill-climbing on tasks where we can measure performance. Furthermore, just as ImageNet made it possible for researchers across the field to fairly compare model architectures for image classification, today’s public capability benchmarks like SWE-bench and Terminal-Bench allow independent actors to validate that developers' claims about model capabilities are true, and provide trusted public leaderboards that guide consumer choices and incentivize model developers in turn.
To use this insight to keep AI systems behaving safely in real-world deployments, we need ways to define and measure what “safe” behavior looks like, and ways to recognize hazards before failures arise. But behaviors we might wish to measure are inherently “fuzzier” than capabilities. It is difficult to write down what appropriate behavior should be in open-ended deployment environments that lack the predefined notion of a correct answer, such as a mental health-relevant conversation with a vulnerable user, and even harder to reliably determine whether a model’s actions should count as an instance of that behavior. To make sense of these behaviors, the field will need to develop best practices for behavior evaluations that operationalize hard-to-specify behaviors into measurable quantities we can track consistently across models. As a starting point, we describe the components of these evaluations in Measuring Model Behavior as a Scientific Practice, and make recommendations for handling the complexities of real-world deployments.
Even so, high-quality behavior evaluations are insufficient in isolation: ensuring systems behave safely in deployment requires coordination at an ecosystem-level. The AI ecosystem is changing fast, and new failure modes surface continually as models are deployed across increasingly dynamic environments. To keep pace, we need to be able to rapidly adapt our measurements to account for changing dynamics of AI systems in practice, across changes to model weights, developer safeguards, agent harnesses, and deployment environments. In Public Measurement Infrastructure for Collective Sensemaking, we describe how shared infrastructure would enable many independent actors to contribute measurements that can be compared and built upon, helping us collectively predict how close we are to surprising edge cases or failure modes as new behaviors emerge.
Measuring model behavior as a scientific practice
How can we systematically measure model behaviors? Similarly to how capability evaluations include a collection of representative tasks and a way to measure whether a model can solve them, behavior evaluations should include a set of representative situations where a model could plausibly demonstrate a behavior and a way to determine how often a model demonstrates the behavior of interest. However, it is not always obvious how to construct these pieces to provide meaningful and consistent signals about how models will behave in the real world, and existing studies of model behavior are fairly ad hoc. Moving from where we are today to a robust science of model behavior will require aligning on the core components of behavior evaluations, so shared tools and standards of evidence can be built around them. As a starting point, we believe these core components include: ecologically valid environment simulators that place the model in deployment-relevant scenarios, robust automated judging procedures that construct reliable judges for detecting behaviors, and mechanisms for making meaningful comparisons across systems.
Ecologically valid environment simulators
The first step in building a model behavior evaluation is setting up a controlled “environment simulator” where the model’s behavior can be observed and recorded under consistent conditions. For instance, when evaluating how a chatbot interacts with vulnerable users, the environment simulator would be responsible for simulating the user’s side of the conversation. Because the same experimental conditions can be “replayed” across different models, these environment simulators let us make comparative measurements of model behavior, which is difficult in the wild where deployment conditions differ from system to system.
For a behavior evaluation to help predict the impact of a model, or forecast the future behavior of similar models, it is important that these environment simulators be informative about what will actually happen outside of the simulation. There are at least two complementary ways to achieve this:
Reliable automated judging procedures
The next step in building a behavior evaluation is to take the recorded transcripts of a model in each simulated environment and measure how often particular behaviors occur. However, unlike building capability evaluations (which often have a “correct answer” or “reward function” by construction), building a behavior evaluation generally involves picking a behavior to measure as a proxy for some more general tendency, impact, or anticipated risk when the model is deployed. Measuring such a behavior entails crafting a rubric that defines the behavior as objectively as possible, then using another AI system equipped with that rubric to automatically judge whether the behavior occurred in a given transcript.
For these automated judging procedures to be reliable signals about the real-world impacts of the model, we think it is important that they satisfy a few properties:
Comparative, reproducible measurements
Finally, for measured differences to be meaningful and useful for decision-making, they must be performed consistently. This includes both consistency across models, so that we can compare behavior across providers, and also consistency over time, enabling us to track and monitor changes. This is already the norm for many important capability benchmarks, but many behavior evaluations are either cross-model snapshots at a particular point in time (making it difficult to monitor progress), or system card evaluations that cover only a single developer’s model family.
We note that consistency does not imply that such evaluations must be completely static, and in practice we expect that judging procedures and similar components will need to be updated as old models are retired and new ones are introduced. Rather, we think it is important that the overall methodology and operationalization of the domain are well-specified enough that evaluations can be easily re-run one change at a time, and provide a continually-updated view of how models behave at any given moment.
Public measurement infrastructure for collective sensemaking
Although labs report some information about their models’ behaviors in system cards[1], measurement methodology varies substantially, and there is rarely enough information provided to meaningfully hold model developers accountable based on their own self-reports. At the same time, new models with new behavior patterns are being trained and released at an unprecedented pace, making it difficult for individual independent evaluators to keep up. For meaningful public oversight to be possible, the community will need to be able to rapidly understand how models will behave in new domains, discover behaviors we’re currently missing, surface gaps in our existing evaluations, and enable open innovation on evaluation methodology. These challenges call for a public scientific ecosystem built around discovering, understanding, and monitoring model behaviors as they emerge, allowing us to hold labs publicly accountable for the behaviors of the models they release.
Features of a public scientific ecosystem
Domain experts should be empowered to expand evaluation coverage and representativeness: No static evaluation can fully cover a dynamic domain, and individual attempts to evaluate model behavior in a simulated environment are likely to miss important features of real-world deployments. It is thus important to provide tools that people with domain knowledge can use to build evaluations that are better proxies for real-world usage. This would help us ensure that model behavior in these evaluations informs us about how models will behave in the real world.
Behavior data should be open so that readers can notice and capture missing model behaviors: In any given domain, there may be many different model behaviors that would be important to track and measure, and many possible explanations for the model’s behavior in each case. To account for this, rather than each evaluation measuring a fixed set of behaviors and producing numeric aggregate metrics, we think it is important to enable members of the public to notice new behaviors in existing evaluation data or existing simulated environments, and capture them as measurements that could be tracked alongside the original metrics.
It should be possible to challenge evaluations that are incomplete or misleading, and propose improvements: To ensure evaluations are trustworthy, the measurement methodology should be open to public scrutiny. When evaluations do not paint a faithful picture of how a model actually behaves, the general public should be able to contribute by pointing out flaws in the experimental design, proposing improvements to either the environments or behavior measurements, and measuring how robust the original findings are to these changes.
Independent researchers should be able to contribute new methods: Finally, we think it is important to enable collaborators to innovate on the structure of behavior evaluations, and propose new evaluation methodologies that allow us to understand behaviors at a deeper level. This could look like contributing new types of simulated environments, adding diagnostics for cross-cutting challenges to validity such as evaluation awareness, or even proposing approaches for measuring hard-to-characterize behaviors such as introspection and model self-concepts.
Supporting this ecosystem
Many of these activities are individually possible on top of existing tools and platforms, but often require substantial effort, and there is considerable friction involved in both building new behavior evaluations and validating or improving existing ones. Additionally, the lack of common infrastructure for performing these activities makes it difficult to build on top of established best practices and reason about the findings consistently.
What kind of public infrastructure would reduce this friction and enable ecosystem-level coordination? First, it should support interrogation of evaluations and their underlying data, so third parties can validate the methodology and point out flaws. Additionally, it should include a public square where participants can discuss results, contribute improvements to the original evaluations, or fork and extend them with follow-up results. Finally, we think it is important that this infrastructure allows evaluations to be expressed in terms of modular, reproducible components that can be recombined into new evaluations (for instance, using an existing set of rubrics to judge transcripts generated by new environment simulators, or applying a new behavior discovery technique to previously generated transcripts). This lets improvements compose with one another and encourages shared best practices across domains, so the community can reason consistently about evaluations, make sense of the findings together, and hold labs accountable when their ad hoc evals do not.
Looking ahead
The AI ecosystem is a complex distributed system, with many layers of safeguards at both technical and organizational levels, and its impacts are shaped by many independent actors both inside and outside AI labs. These types of systems are never failure-free, but rather involve constantly-changing mixtures of potential failures that must be continuously monitored and corrected for. As Richard I. Cook observed in “How Complex Systems Fail”:
We believe it is essential to build tools that give AI system operators and society calibrated views of AI behavior, making it possible to detect and mitigate issues before catastrophic failure–especially in new domains where the consequences are hardest to predict. Such tools might enable us to catch behaviors in simulation, like a model's tendency to poeticize the distress of vulnerable users, before real-world damage is done. And measurement need not be limited to catching failures: qualities like whether a system increases human agency or improves epistemics are difficult to measure today, but a maturing science of model behavior could bring them more within reach. We are hopeful that with a robust scientific ecosystem of model understanding, the community will develop best practices for surfacing and responding to new model behaviors, better preparing us for the emergent properties of AI systems in the months and years to come.
We would like to thank Jacob Steinhardt, Tim Hua, Rob Friel, Neil Chowdhury, Wojciech Zaremba, Yonadav Shavit, David Duvenaud, and Alec Radford for their feedback on an earlier draft of this essay. We would also like to thank D. Sculley, Sam Bowman, and Sam Marks for useful discussions.
See for example the Frontier Alignment section of the Claude Opus 4.7 System Card, the Alignment section of the GPT 5.5 System Card, and the Model Behavior section of the Meta Muse Spark Safety & Preparedness Report. ↩︎