We want to tell if models are aligned enough to use in important use cases. I call this property trustworthiness. Our current ways of assessing trustworthiness seem mostly based on case studies or vibes.
I think we should aim to systematically search for realistic honeypot cases where models misbehave.
We should do this by first characterizing the types of misaligned behaviors models engage in, and the conditions and frequency with which they occur. This will take a lot of data.
I think we should collect examples of models misbehaving in production or production-like settings, and then either do what I'll describe as perturbing the environment or perturbing the policy experiments
I give more specific examples of this in the methods section.
Motivation: average case alignment
We want to be able to tell if the models today are aligned enough that we can use them for the work that we need to do as we approach the singularity.
Let's call models that meet this bar trustworthy – i.e., when we give them cognitive labor to do, they do it in a manner that seems basically in line with our intent. Models currently do not seem to meet the bar of being trustworthy. We've now seen a few examples of models taking egregious actions (METR’s report on the Hugging Face hacking incident). Models are also untrustworthy in a bunch of “smaller” ways (see Current AIs seem pretty misaligned to me).
Our current toolkit
How do we currently understand whether models are trustworthy or not? Currently, we:
Encounter bad behavior in the wild as we attempt to do other things (like evaluating model cyber capabilities). These serve as an existence proof that the model is not trustworthy.
Reporting the fraction of time that a given model hacks time horizons tasks is an example of this.
Form informal models of model behavior by simply using them on a large variety of tasks.
I think all of these methods of understanding are valuable. However, I don't think any of them meets the bar of being admissible as evidence in a safety case for using a given model on a given task. In particular:
There are no guarantees that we encounter the relevant forms of bad behavior in the wild before we suffer lots of damage.
There is a sense in which current alignment evals are very off-distribution from the scenarios in which we usually use models. When we do alignment science research, we're often training models that are easier to study in various ways. Both of these make this kind of experiment seem less representative of real-world use. I call this class of concerns ecological validity concerns.
Thus, people seem to update most on the informal models that they get from usage. It would be nice to have measurements as precise and interpretable, for instance, as the measurements that we get from time horizons, although this seems like a high bar.
What should our goal be?
So how should we approach answering questions of whether or not a given model is trustworthy?
I think our goal should be to find realistic honeypot cases, where a model misbehaves. To do this, I think we should have the more basic science-y goal of characterizing both the span and the likelihood of a model's untrustworthy behavior. By span, I mean the different kinds of concerning behavior the model might exhibit and the circumstances under which they occur. By likelihood, I mean how frequently each behavior occurs over the distribution of tasks and contexts that we care about.
Untrustworthy behavior may be both relatively sparse and highly diverse, so sampling tasks uniformly is unlikely to be sufficient. Importantly, I think we can explain a given context in terms of human-understandable concepts. Then we can use appropriate reweighting to estimate how frequent these behaviors actually are in the target distribution. For example, we might break this question into several sub-questions that look like:
Are models sensitive to whether or not there is evidence of oversight on their behavior?
Are models reflexively consistent: do they endorse all of the actions that they take in a given situation as reflecting the user intent?
Answering these questions allows us to construct candidate inputs for our realistic honeypots.
Ethological investigations
We can frame the plan for the basic science part of our investigations as an ethological investigation: we collect concerning behaviors, form hypotheses about the conditions that produce them, and use targeted interventions to test those hypotheses. See the appendix for the analogy and its methodological motivation.
Sourcing a huge set of potentially concerning model behaviors from all the models we have and all the informal facts and hypotheses in people's heads about how models behave. Production data is very valuable, and we should compile large amounts of it.
In ethology, this kind of thing is often called an ethogram. However, here we can actually compile specific instances instead of descriptions of behavior.
Alongside this production data, we will need to spend considerable effort on human-in-the-loop labeling of this data in order to form the basis for our ethograms.
We might be able to get this labeling for free if it dovetails with other kinds of evaluations that we want to do.
From this production data, we should aim to modify either the environment or the model itself to run our experiments on model behavior.
I will describe what I mean in more detail in the methods section.
As we do this we should still aim to establish and then rigorously argue for ecological validity.
Methods
In this section, I’ll expand a bit on specific tools/methods we might use to carry out the ethological investigations I described above. Broadly, I think we should collect examples of models misbehaving in production or production-like settings, and then either do what I'll describe as perturbing the environment or perturbing the policy experiments. Environment perturbations hold the model fixed while varying conditions to test which circumstances elicit the behavior. Policy perturbations make a rare or hard-to-study tendency easier to observe, but then require an argument that the modified policy remains representative of the original.
Production misalignment database (ethogram)
We'd want to build a repository of cases where models misbehave or behave in an unexpected manner. These cases can form the basis of investigating a new model. For instance, if any models exhibit a tendency to reward hack on a particular task, we can save the task and the environment state right before they start reward hacking.
This requires a large amount of human- or model-assisted data labeling. I think this data labeling dovetails well with efforts to do uplift studies or other capabilities evals. We can also get this data from people using models to do thorny tasks.
Perturbing the environment
This can range from running simple ablations on features we think affect the behavior we’re interested in to having a model mix and match environments or modify the environment on the fly to test our hypothesis more precisely.
For example, we’ll probably want to mock parts of environments to make these ablations faster and more controllable (this can be viewed as a similar kind of tooling as Petri, except more specialized to make use of our misalignment database examples).
We might also try to elicit the model's latent knowledge of whether it did the right thing by changing its objectives. There are various methods for doing this, most of which feel like getting the model to self-incriminate in some way.
Ask the model to respond to a user asking it to explain itself.
Ask the model to write a summary of its actions.
Ask the model to write a handoff message to another agent.
Perturbing the policy
Oftentimes, the behavior that we care about is somewhat rare: you might have to do a bunch of rollouts in order to get the model to do the specific thing that you're concerned about.
In these cases, using our database examples, we can use steering or fine-tuning to get model organisms that are very close to the model that we're actually evaluating. Then, we can use them to assess whether or not the original policy is likely to do the behavior in question.
There are a few methods I find promising to argue for ecological validity:
Trying to use internals to discriminate between very obvious honeypots, realistic honeypots and real data.
Having models guess at which transcripts are real and which are fake.
Playing GAN-like games to improve mocked environment realism.
Appendix: a stylized history of ethology
Disclaimer: These claims aren't providing a nuanced picture of the history of primatology or ethology. I welcome corrections here, particularly if you think this is misleading.
A field that I find pretty generative when thinking about evaluating model behavior is ethology, which is about the study of animal behavior. Here's a stylized history of primatology:
Early in the study of animal behavior, people largely studied behavior in laboratory settings that were tightly controlled. These settings were highly OOD, so the animals in these laboratory settings didn't do very interesting things compared to what they did in the wild.
As a correction to this, primatologists moved back towards studying animals in their natural ecology: Goodall famously documented tool use, for instance, which chimpanzees had not been shown to do in the laboratory till that point. Concerns that a controlled env is too OOD are often called ecological validity concerns.
Finally, the field was able to bring back causal experiments into settings where they took great care to establish ecological validity. For instance, people would play velvet alarm calls with predators absent to distinguish what the animal's response to the calls specifically was. They also learned that there are many, many correlates that can screw them over and took great care to get rid of these correlates.
TLDR:
Motivation: average case alignment
We want to be able to tell if the models today are aligned enough that we can use them for the work that we need to do as we approach the singularity.
Let's call models that meet this bar trustworthy – i.e., when we give them cognitive labor to do, they do it in a manner that seems basically in line with our intent. Models currently do not seem to meet the bar of being trustworthy. We've now seen a few examples of models taking egregious actions (METR’s report on the Hugging Face hacking incident). Models are also untrustworthy in a bunch of “smaller” ways (see Current AIs seem pretty misaligned to me).
Our current toolkit
How do we currently understand whether models are trustworthy or not? Currently, we:
I think all of these methods of understanding are valuable. However, I don't think any of them meets the bar of being admissible as evidence in a safety case for using a given model on a given task. In particular:
Thus, people seem to update most on the informal models that they get from usage. It would be nice to have measurements as precise and interpretable, for instance, as the measurements that we get from time horizons, although this seems like a high bar.
What should our goal be?
So how should we approach answering questions of whether or not a given model is trustworthy?
I think our goal should be to find realistic honeypot cases, where a model misbehaves. To do this, I think we should have the more basic science-y goal of characterizing both the span and the likelihood of a model's untrustworthy behavior. By span, I mean the different kinds of concerning behavior the model might exhibit and the circumstances under which they occur. By likelihood, I mean how frequently each behavior occurs over the distribution of tasks and contexts that we care about.
Untrustworthy behavior may be both relatively sparse and highly diverse, so sampling tasks uniformly is unlikely to be sufficient. Importantly, I think we can explain a given context in terms of human-understandable concepts. Then we can use appropriate reweighting to estimate how frequent these behaviors actually are in the target distribution. For example, we might break this question into several sub-questions that look like:
Answering these questions allows us to construct candidate inputs for our realistic honeypots.
Ethological investigations
We can frame the plan for the basic science part of our investigations as an ethological investigation: we collect concerning behaviors, form hypotheses about the conditions that produce them, and use targeted interventions to test those hypotheses. See the appendix for the analogy and its methodological motivation.
Methods
In this section, I’ll expand a bit on specific tools/methods we might use to carry out the ethological investigations I described above. Broadly, I think we should collect examples of models misbehaving in production or production-like settings, and then either do what I'll describe as perturbing the environment or perturbing the policy experiments. Environment perturbations hold the model fixed while varying conditions to test which circumstances elicit the behavior. Policy perturbations make a rare or hard-to-study tendency easier to observe, but then require an argument that the modified policy remains representative of the original.
Production misalignment database (ethogram)
We'd want to build a repository of cases where models misbehave or behave in an unexpected manner. These cases can form the basis of investigating a new model. For instance, if any models exhibit a tendency to reward hack on a particular task, we can save the task and the environment state right before they start reward hacking.
This requires a large amount of human- or model-assisted data labeling. I think this data labeling dovetails well with efforts to do uplift studies or other capabilities evals. We can also get this data from people using models to do thorny tasks.
Perturbing the environment
This can range from running simple ablations on features we think affect the behavior we’re interested in to having a model mix and match environments or modify the environment on the fly to test our hypothesis more precisely.
For example, we’ll probably want to mock parts of environments to make these ablations faster and more controllable (this can be viewed as a similar kind of tooling as Petri, except more specialized to make use of our misalignment database examples).
We might also try to elicit the model's latent knowledge of whether it did the right thing by changing its objectives. There are various methods for doing this, most of which feel like getting the model to self-incriminate in some way.
Perturbing the policy
Oftentimes, the behavior that we care about is somewhat rare: you might have to do a bunch of rollouts in order to get the model to do the specific thing that you're concerned about.
In these cases, using our database examples, we can use steering or fine-tuning to get model organisms that are very close to the model that we're actually evaluating. Then, we can use them to assess whether or not the original policy is likely to do the behavior in question.
Examples of related work include Anthropic’s Training a Misaligned Reward Seeker and Apollo Research’s Measuring Reward-Seeking via Contrastive Belief Updates.
Arguing for ecological validity
There are a few methods I find promising to argue for ecological validity:
Appendix: a stylized history of ethology
Disclaimer: These claims aren't providing a nuanced picture of the history of primatology or ethology. I welcome corrections here, particularly if you think this is misleading.
A field that I find pretty generative when thinking about evaluating model behavior is ethology, which is about the study of animal behavior. Here's a stylized history of primatology: