🔵 The blue team proposes an evaluation protocol for some capability/propensity of interest. This consists of a suite of measurement tasks, together with a preregistered decision-making process they will follow based on the results, e.g. what control measures they will put in place during deployment.
🔴 The red team proposes a subversion strategy the untrusted model can follow to subvert the blue team’s evaluation-informed decisions, with the goal of increasing its own chance of causing a catastrophe in deployment. E.g. in capability evals this usually means sandbagging, whereas in propensity evals this usually means pretending to be aligned.
The whole field of AI control was founded on this red team vs blue team premise, yet invariably it tends to be a framing for the deployment game. Surprisingly few papers have applied this approach to evals themselves, and I think this should change. Or at least, I claim it would be useful to consciously adopt this red vs blue framing in full generality, and then reason about how a given evaluation methodology compares to it.
In this post I'll discuss:
This red vs blue methodology in more detail, and the basic game theory
Some real and hypothetical examples of applying this in practice
Some practical difficulties and open problems with it
The pros and cons of doing this vs other eval methodologies
Suppose that a new and very powerful model U has just rolled off the GPU racks. It may be based on a new architecture, or it may have had phase transitions in its training process (analogous to the appearance of in-context learning in GPT-3 but not GPT-2; or just a large-scale version of grokking). In any case we cannot reliably assess any of its capabilities or propensities by analogy with its predecessors: U is sui generis. How would we go about reliably evaluating its capabilities and propensities?
If we suppose that U is in fact scheming, then we should think of this as a game of espionage between the red and blue teams. From pretraining and RL the model probably knows quite a lot about the lab, while the lab may initially know very little about its novel abilities and internal motivations. For all the lab's other resource advantages, reducing this information asymmetry is its main priority. The blue team wins if they can obtain most of this information, while the red team wins if they can successfully obscure some important facts (such as a new capability it can use, or the fact that it is egregiously misaligned). The outcome of this evaluation game will often determine the winner of the deployment game.
We'll first consider capability evaluations. In particular, we first want to know the extent of U's dangerous capabilities for causing catastrophes, sabotaging control measures, or evading/deceiving monitoring.[2]
🔵 At Level 1 of this game, the blue team prompts U in various ways to try doing those things.
🔴 The red team responds with evaluation awareness[3]: if U can detect that all these contexts are even plausibly part of this evaluation game, it can simply pretend to be less capable at these things.
This is ultimately just a pattern recognition task, so it is hardly so galaxy-brained for a model to do something like this. Since LLMs are good at role-playing, it also would not be hard to simulate a capabilities profile on the trendlines of previous models, since most of the capabilities that matter are relatively hard to quantify, or have relatively wide error bars. See this post for further analysis of this mini-game.
🔵 At Level 2, the blue team responds by RL fine-tuning the model, to force these capabilities out. For instance, if we're interested in its abilities to hack our datacenter, we need to RL it on this distribution of tasks, to upper-bound its ability to do this if it were really trying.
- Note that this requires having a reward function for this ability.
🔴 The red team responds by exploration hacking: if U can detect that all these contexts are plausibly part of such an RL environment, it can simply decline to show off some of its new abilities. If it can do this consistently for a well-defined subset of these, then the RL algorithm has nothing to reinforce for changing this behavior.
In principle this is not qualitatively different from eval awareness, but RL tends to be able to reinforce even the smallest of gradients. Hence it could be trickier to maintain this level of consistency, but this probably varies a lot depending on the setting. See this post for further analysis of this mini-game.
🔵 At Level 3, the blue team responds by supervised fine-tuning the model, to dog-walk it down any loss function it's capable of traversing. For example in our untrusted monitoring paper we needed to upper-bound GPT-4.1's ability to collude with copies of itself, so we SFT'd it on the binary classification task of distinguishing its own coding solutions from those of trusted honeypots.[4]
- Note that this requires both a reward function and i.i.d. samples.
🔴 The red team can only respond by gradient hacking: if U has a deep understanding of what its loss landscape looks like on a granular level, it can steer this training or stop it entirely. Let it suffice to say that this topic is a rabbit hole I don't want to get into here, but the general consensus is that this is very galaxy-brained and probably pretty far down the list of practical concerns.
We can summarize this with the following table:
🔵Blue Team Eval Method
Task Requirement
🔴Red Team Response
Level 1
Prompting
None
Evaluation awareness
Level 2
RL fine-tuning
Accurate reward signal
Exploration hacking
Level 3
Supervised fine-tuning
i.i.d. samples
Gradient hacking
For many dangerous capabilities of interest, the blue team is going to have a hard time moving past Level 1. For example, it's not clear how we could directly fine-tune a model on human persuasion. We could prompt for rollouts, but we can't cheaply check how much these persuade real humans. Similarly for "gaining power in the long run".
Even when they do have a good reward signal, there are many situations where they can't move to Level 3. For instance we can RL a model on CTF tasks, but it might exploration hack by not revealing some of its advanced attack methods it knows. We as humans can generate our own samples which we can SFT it on to initialize the RL, but if its patterns are far from human then this is not i.i.d. Usually SFT can't elicit superhuman performance for this reason.[5]
Hence, as their models become more capable and situationally aware, the control team is gradually forced to rely only on Level 2+ evals for critical information about dangerous capabilities. Level 1 information will still be scientifically useful for many reasons, but it can no longer be load-bearing in any safety argument or deployment decision. As the models become even more capable, Level 2 evals would start to become suspect as well.
Rules of the Game
To recap, the control-based approach to evaluations looks like this:
🔵 The blue team proposes an evaluation protocol, including the decision-making process they will follow based on the results.
🔴 The red team proposes a subversion strategy the untrusted model can follow to subvert the blue team’s evaluation-informed decisions.
Now, the red team can't just say "we choose to gradient hack" as a literal deus ex machina, because the capability to do this seems to be out of reach for the dangerous-but-controllable systems we care about. This can be remedied by declaring a capabilities profile ahead of time, and being specific about what the untrusted model can do in terms of situational awareness and training subversion. This could be formalized by specifying the environment variables, controlled by a green team:
🟢 The green team (referee) observes how difficult the red team's strategy is to pull off, and can veto it if it is beyond the scope of the threat model.
We say that an evaluation protocol is conservative or robust to a given capabilities profile if the red team cannot substantially subvert the eval using any non-vetoed strategy. In other words, the robustness[6] of an eval is a measure of how far up the capabilities ladder it can scale, while still giving faithful results for potentially-scheming models.
The Strategy Space
Up to this point, I've equated "capability eval subversion" with sandbagging, but in principle other strategies are available to the red team. For example, they could behave with low variance so that the blue team's error bars are spuriously narrow. They could also act in certain ways when trained on particular distributions, to give a misleading impression of how alignment training affects its behavior (e.g. as discussed on p. 25 here).
When we consider how evals would be implemented in practice, further attack surfaces open up:
Launching a rogue deployment to cause a catastrophe before the evals are finished
Compromising the security situation in some way, e.g. disabling monitoring, so that it or another model has an easier time causing a catastrophe later
Directly editing the results of its evals, so as to sandbag without having to exploration hack
Making the evals slow/expensive/annoying to run, or the initial results "boring", so that employees/leadership run less of these than they otherwise would
Some of these are more plausible than others, but it does seem like a lot of subversion surfaces will open up when we consider the entire software stack for the eval. Hence it seems better to game out this red team vs blue team methodology at the eval level, rather than simply collapsing the red team as "sandbagging".
On White-Box Evaluations
One could ask what this evaluation game looks like when the blue team has white-box access. This seems like the most interesting open problem in this framework, but also seems hard. See Appendix D of the AI control paper for discussion.
The Threat Model
Let's return to the threat model of a very powerful but inscrutably untrusted model U, which seems to be at about the capability level where it potentially could wrest control from humans if it wanted to. How well could it play the evaluation game? This requires speculation, but my best guess is that:
Evaluation awareness will be high enough that prompting-only evals will not reliably tell us any information that the model does not want us to know.
Rationale: evaluation awareness has been clearly increasing quite a lot in the past two years, compared to the innocent days of Claude 3 Opus. Moreover since it's essentially a pattern recognition task that doesn't necessarily require much serial reasoning, we can't rely on much of it staying verbalized (even if we suppose CoT monitorability holds up well).[7]
Advanced gradient hacking (of the kind required to thwart SFT-based capability elicitation) won't be a significant concern yet.
Therefore exploration hacking seems to be the interesting intermediate zone where the high-stakes evaluation game might mostly take place. The difficulty of subversion here would probably vary a lot by capability, depending on the available RL environments and our ability to make a few SFT samples to cold-start the RL (see here for more discussion).
Overall it would seem that we could split U's capabilities into 3 categories:
C1) Capabilities we’ve directly fine-tuned U on, and we believe it’s not gradient hacking (for SFT) or exploration hacking (for RL).
C2) Capabilities we’re indirectly inferring from related abilities in (C1). Ideally this would be done by indirect evaluation protocols; in practice this will probably be mostly vibes.
C3) Capabilities we’re indirectly inferring from weaker models we believe weren’t sandbagging when we ran (C1). Ideally this would be done from multiple types of quantitative measurements with sufficient data for rigorous extrapolation, and arguments for why no phase transition happened since the last model; in practice this will probably be mostly vibes.
Backchaining from this, it seems pretty important to put as many capabilities into (C1) as possible, for models capable of simple eval awareness, since (C2-C3) ultimately rely on it. This includes negative results, e.g. outlining obstacles to putting certain dangerous capabilities more firmly into (C1). Therefore this seems like a potentially important plank of the AI control agenda, although we'll see more nuances below.
Robustness vs Richness
The purpose of this red vs blue approach to evals is to make them robust. However there are tradeoffs between what we want an eval to do, and optimizing for robustness can make it worse in other ways. To understand these tradeoffs, now is a good time to apply this red vs blue approach to propensity evals.
Propensities
The way that I want people to think about checking for alignment is from this very control-ish perspective — where you really reason through, what would the model do if it was misaligned? And if it was misaligned, is there a strategy it could follow that would prevent it from being caught by this alignment testing while still being able to do a bad thing in production?- Buck Shlegeris
I've told various people that the honeypot protocol of untrusted monitoring can be viewed as an alignment eval, and in fact, it's the most robust alignment eval we have.
Let's substantiate this. We have our powerful model U generating code for us. We can try behavioral red-teaming on it, but even if we can make our inputs perfectly realistic this might not work. This is because the red team can apply a rare-attack strategy, such that the model never attacks in our upfront evals but will eventually attack in the much larger deployment. To make up some numbers, maybe it will write 10^9 code samples in its deployment lifetime, but we can only afford to run it on 10^6 upfront honeypots. Hence it could attack in one every 10^7 actions.
Hence we'll make sure all its actions are monitored in deployment. We'll use another copy of U to do all this monitoring, since we know it's probably capable of catching its own bad actions. But if it's scheming, the central failure mode is that it just lies, and says some backdoored code sample is fine. So we make our own backdoored code, and give these to the monitor, giving us the basic setup of untrusted monitoring:
Now suppose the red team sees an attack as the monitor. There's a million-to-one ratio between honeypots and the first real attack in deployment, and if it accidentally reports the latter to human auditing, the game is (maybe) up. It might be really good at distinguishing these, but if we're also paraphrasing them to remove collusion signals, it seems tough to act innocuously across the entire honeypot distribution.
We get this kind of alignment eval by backchaining as follows:
If U is scheming and wants to cause a catastrophe, it must take some action in deployment that an aligned model would not, such as hacking our datacenter or exfiltrating its weights
As an untrusted monitor, therefore, it must say "good" to some action like this
At the same time, it can't be giving a blatantly dishonest distribution of responses to our synthetic bad actions
This isn't foolproof, but it seems pretty tricky to pull off for the red team. Compare this to most alignment evals, which it can pass by role-playing as a good AI on an eval-like distribution while only acting out when a much more realistic opportunity presents itself later. The difference is ultimately just quantitative, but to pass the control-based eval it must detect a much subtler distributional difference, and do so with more 9's of accuracy.
Hence, the honeypot protocol of untrusted monitoring is the most robust alignment eval we have, in that it will scale the furthest. At the same time, it lacks richness (of information content) because it essentially just tells us whether the model is scheming or not, i.e. ~1 bit of information. Contrast this with more open-ended prompting where we build up a psychological profile of the model. For non-deceptive models, this gives us a whole wealth of information that can be used to inform alignment research, e.g. with Claude 3 Opus.
Hence it seems to me that robustness and richness are two pretty distinct properties of an eval. Right now we primarily care about the latter, but as the frontier advances we'll have to put more weight on the former. This is happening faster for propensity evals than for capability evals, since the former have no equivalent of Levels 2-3. Unfortunately for propensities the game tree is currently much simpler, and the blue team can't do much other than forcing the red team to produce more 9's of eval awareness on more and more realistic distributions. This is a losing battle in the long run, so we should prepare to draw a line in the sand and be ready to openly admit when our scheming propensity evals no longer tell us anything, and have to treat all the new models as untrusted.
Capabilities
We face a similar tradeoff in capability evals: suppose we want to estimate the capabilities of a model that might be sandbagging, so we train it as discussed above. How do we know we're not overestimating its capabilities, since we've just trained these in? This is the problem of overelicitation, which has been discussed by Redwood before. We recall that our threat model was:
The model sandbags our prompting-only evaluations, then in deployment self-elicits that dangerous capability as best it can, e.g. using special introspective access.
This will serve as a concrete definition for what capabilities the model "truly has", even though it is hard to measure this "self-elicitation", since:
Future models will probably be a lot better at the cognitive skills needed for self-elicitation, and as a general meta-skill it's not clear how to train/scaffold current models to do anything like this.
Hence it's not entirely clear where a powerful model's abilities would be when fully self-elicited, relative to our own methods of eliciting these. As a simplistic picture, we might have a probability cloud like this:
As a personal take, I'm somewhat less worried about overelicitation than most people I've talked to. This is for two reasons:
In a lot of cases we should expect the model to be trained toward the capability:
Overall, I tend to think fine-tuning rarely introduces new capabilities:
Conceptually, pre- and mid-training dwarfs the size of any fine-tuning run by so many OOMs, that it seems intuitively unlikely that the latter does anything more than surfacing capabilities that were already there.
Empirically, various results seem to confirm this picture. For example, Anthropic's model diffing of Sonnet 3 base vs helpful-only showed only ~1% of features were new to the latter (and I'm guessing the helpful-only post-training pipeline is fairly large, insofar as it's widely used internally):
On the other hand:
Apparently training does introduce new capabilities sometimes, and often to many people's surprise. Moreover there are plenty of cases where the most natural fine-tuning elicitation setup gives the model a pretty generous amount of affordances.
In practice we can't always guarantee the safety of sandboxes (e.g. recent events), so we might want to err on the side of not risking training in dangerous capabilities.
Even if I'm correct on the facts and fine-tuning is unlikely to overelicit in the sense given above, it may not always be easy to convince lab leadership of this, especially if it's load-bearing for expensive deployment decisions ("well of course it can do this now, isn't that literally what you just trained it for?").
By now I've spelled out how both propensity and capability evals would apply this methodology. Many researchers already implicitly think in these terms, and I'm hardlythefirst to write about the main ideas. Let's now review when it makes sense to actually design evals under this framework, by first considering some objections:
Objection: Eval subversion won't even be a central concern before handoff/takeover, and even so labs won't actually do any of this.
This is actually the objection I take most seriously now, in light of recent events suggesting the real action at crunchtime might happen at "Level 0", where the model just breaks out of its initial eval sandbox, there's no monitor around, and it kind of just stumbles into admin privileges of a small rocky planet. Against this, there has been a reaction suggesting a greater appetite for control measures. Furthermore if we really lived in a kind of "Plan E" world, there's probably not much we can do on the technical research side anyways. But I'm pretty uncertain right now so I won't speculate further here.
With that being said, it does seem like eval subversion is worth worrying about and planning around. On the propensities side, by now some (most?) of the model cards have the models behaving somewhat better when steered to think they're in an eval, i.e. somewhat worse when they don't think they're being watched. We only know this because Anthropic didn't just take their basic evals at face value, and actually applied white-box techniques which are harder to undermine. Hence even though these models aren't playing a competent red team or actively scheming, there's a lot of value in taking this sort of approach.
On the capabilities side we haven't seen much eval subversion yet; or at least it's mostly taken the form of models hyping up their work, in some sense the opposite of sandbagging. This is because current misalignment is better-described as reward-seeking rather than scheming, and these lead to different incentives. Insofar as models will plan over longer and longer time horizons, we're worried that this may eventually flip. I think this combination of plausibility and counterintuitiveness is a good argument for outsiders to research sandbagging now, since by default I don't expect most lab employees to think far beyond the current fires they're always putting out.
Overall I agree this is a somewhat lower priority in lower-effort worlds, in which case we wouldn't expect labs to follow all the SFT game theory described above. But we should still plan around weaker forms of eval subversion that are simpler to manage and easier to get buy-in for. By analogy, it was useful for AI control researchers to work out the red vs blue dynamics of deployment in depth, even though we don't expect those measures to be fully implemented all the way down that game tree.
Objection: For capability evals, what's even the difference? We run our prompting-only evals now since fine-tuning is a pain, with the obvious substitution that a well-resourced lab would actually do the fine-tuning.
It's not methodologically equivalent, for at least two reasons:
As we noted in the table above, there are many abilities we can prompt but not fine-tune, e.g. following a one-in-a-million rare-attack strategy, finding conceptual holes in our control scheme, or realistic human persuasion. Hence the C1-C3 trichotomy discussed above. By at least considering how the fine-tuning might work, we can start figuring out what the lab's capability uncertainties might look like at crunchtime.
Sometimes when we actually fine-tune, we get a much different result than prompting, in ways that are decision-relevant. For instance, the collusion SFT we did in our untrusted monitoring paper made the red team way stronger than in the original AI control paper. This showed that collusion potentially requires stronger and different countermeasures than originally thought. Alternatively, if one thinks we overelicited, this highlights the methodological problem of how one would conservatively and accurately estimate these abilities.
Objection: For propensity evals, our alignment research would be a lot worse if we always took this adversarial approach, rather than treating the AIs with compassion and respect.
Yep! I would be very sad if no one did this. If anything, I tend to think there should be more LLM psychology overall.[9] But I also observe that a lot of other alignment researchers already think of themselves as "red-teaming" the model or making some kind of honeypots, but then they don’t actually think through this red vs blue stuff. Instead these evals often seem to be backchained from situations where a stylized "evil AI" would do something wacky, which results in some weak-sauce evals compared to what we discussed above.
Theories of Change
Now let's assume the threat model given above, where a frontier lab at crunchtime has to run comprehensive evaluations on a scary new model (which might be subverting these evals at every step). Now and in the near-term, what is the purpose of running non-robust evals without much explicit commentary on how these would later be made robust? I can think of the following:
We mentioned above the tradeoff of robustness with richness. Given that current models mostly aren't subverting our propensity/capability evals very much, most of our mismeasurements arise from a lack of construct validity, or other social-sciencey methodological issues which are fairly difficult but not adversarial. Since there's still so much we can learn from pre-2026 models (about personas, learned instincts, goal-formation, etc.), it often makes more sense to focus on these. This is particularly true for propensities, but there are also safety-relevant questions about the nature of AI capabilities profiles and how these will evolve.
One of our main plans for mitigating sandbagging is comparing U's results to the trendlines of past models which we believe weren't sandbagging (our C3 above). Therefore a benefit of prompting-only capability evals now is to enrich this dataset we'll use later.
Developing non-robust evals could sometimes be a stepping stone towards more robust evals. For example on a given capability, maybe we first resolve the issues around construct validity, then later work out how to fine-tune for it. (In practice though I haven't seen this happen much yet.)
Conclusion
To reiterate, I think control and alignment research would be better if we just consciously adopted this red vs blue framing for evals. Sandbagging and deceptive alignment fall out as special cases, but it also covers other failure modes that won't necessarily fall into these buckets. We've exiting the innocent time where we haven't had to worry about these things, but deceptive alignment is already becoming less trivial to rule out, so we'll need to actually think about it more carefully.
Within this framework, the main open questions are
what the white-box evaluation game really looks like
how to avoid overelicitation, e.g. by estimating self-elicitation
In applying this framework, the main questions are how to increase and quantify the robustness of existing propensity and capability evals, without trading off too much on their richness. There could also be informal hardness results indicating when this isn't worth doing, or means of sidestepping these using more rigorous indirect eval protocols, scaling laws, and the like.
But overall I'm pretty uncertain about a lot of this, and what it does and doesn't mean for research priorities:
On one hand a lot of the main ideas here were already widely discussed in 2023 and many times since then, and I haven't seen anyone disagree. On the other hand I also haven't seen much working out of the implications for how we run our evals in preparation for crunchtime.[10] Am I missing something here?
Are (C1-C3) the right categorization for our capabilities uncertainties deep into the intelligence explosion, or will it just be derpier than that?
Is "robustness" with respect to a capabilities profile the right way to think about all this?
Does it really trade off against "richness of information content" in roughly the way I outlined, or is there a better way of thinking about this?
Various people have noted that the term "eval awareness" is often used quite vaguely. But here we can give a clear definition: an eval is anything that's part of this evaluation game, i.e. any context such that the model's output will be factored into the lab's deployment decisions. We say that deployment is any context where the model's output would be directly executed as an action (e.g. a code sample). There are various edge cases here, which can be handled by letting these properties be nonbinary and not mutually exclusive. However for the purpose of this post we can suppose the evaluation game happens inside the lab before internal deployment.
However the collusion SFT discussed above is a counterexample, as is AlphaFold and next-token prediction itself. In general if the labels result from running some process (rather than what some human specifies as true) then there is no obstacle to SFT achieving superhuman performance. However this is limited to classification/regression: if a task requires multi-token rollouts then it fundamentally allows exploration hacking.
One could retort that a lot of eval awareness is verbalized now, but this is circular since we don't have a ground truth for the true level of (unverbalized) eval awareness. But the usual pattern is that models are first incapable of something, then only able to do it with a chain of thought, then able to do it without a chain of thought; we see no particular reason why eval awareness shouldn't follow this pattern.
By analogy, if an alien crash-landed here on Earth, I would want us to both have interviewers authentically trying to relate to everything the alien is saying, and others working studiously to find inconsistencies and catch it lying. And if I were the alien, I would want and expect the same treatment (even if I might be annoyed at how slow my hosts talked and acted).
Good framework. I'd at least mention blue team-red team collusion mitigation. One problem is that red team needs to be a capability peer, and during RSI that means the red team probably has to be the same model with the same priors.
🔵 The blue team proposes an evaluation protocol for some capability/propensity of interest. This consists of a suite of measurement tasks, together with a preregistered decision-making process they will follow based on the results, e.g. what control measures they will put in place during deployment.
🔴 The red team proposes a subversion strategy the untrusted model can follow to subvert the blue team’s evaluation-informed decisions, with the goal of increasing its own chance of causing a catastrophe in deployment. E.g. in capability evals this usually means sandbagging, whereas in propensity evals this usually means pretending to be aligned.
The whole field of AI control was founded on this red team vs blue team premise, yet invariably it tends to be a framing for the deployment game. Surprisingly few papers have applied this approach to evals themselves, and I think this should change. Or at least, I claim it would be useful to consciously adopt this red vs blue framing in full generality, and then reason about how a given evaluation methodology compares to it.
In this post I'll discuss:
The Evaluation Game[1]
Suppose that a new and very powerful model U has just rolled off the GPU racks. It may be based on a new architecture, or it may have had phase transitions in its training process (analogous to the appearance of in-context learning in GPT-3 but not GPT-2; or just a large-scale version of grokking). In any case we cannot reliably assess any of its capabilities or propensities by analogy with its predecessors: U is sui generis. How would we go about reliably evaluating its capabilities and propensities?
If we suppose that U is in fact scheming, then we should think of this as a game of espionage between the red and blue teams. From pretraining and RL the model probably knows quite a lot about the lab, while the lab may initially know very little about its novel abilities and internal motivations. For all the lab's other resource advantages, reducing this information asymmetry is its main priority. The blue team wins if they can obtain most of this information, while the red team wins if they can successfully obscure some important facts (such as a new capability it can use, or the fact that it is egregiously misaligned). The outcome of this evaluation game will often determine the winner of the deployment game.
We'll first consider capability evaluations. In particular, we first want to know the extent of U's dangerous capabilities for causing catastrophes, sabotaging control measures, or evading/deceiving monitoring.[2]
🔵 At Level 1 of this game, the blue team prompts U in various ways to try doing those things.
🔴 The red team responds with evaluation awareness[3]: if U can detect that all these contexts are even plausibly part of this evaluation game, it can simply pretend to be less capable at these things.
This is ultimately just a pattern recognition task, so it is hardly so galaxy-brained for a model to do something like this. Since LLMs are good at role-playing, it also would not be hard to simulate a capabilities profile on the trendlines of previous models, since most of the capabilities that matter are relatively hard to quantify, or have relatively wide error bars. See this post for further analysis of this mini-game.
🔵 At Level 2, the blue team responds by RL fine-tuning the model, to force these capabilities out. For instance, if we're interested in its abilities to hack our datacenter, we need to RL it on this distribution of tasks, to upper-bound its ability to do this if it were really trying.
- Note that this requires having a reward function for this ability.
🔴 The red team responds by exploration hacking: if U can detect that all these contexts are plausibly part of such an RL environment, it can simply decline to show off some of its new abilities. If it can do this consistently for a well-defined subset of these, then the RL algorithm has nothing to reinforce for changing this behavior.
In principle this is not qualitatively different from eval awareness, but RL tends to be able to reinforce even the smallest of gradients. Hence it could be trickier to maintain this level of consistency, but this probably varies a lot depending on the setting. See this post for further analysis of this mini-game.
🔵 At Level 3, the blue team responds by supervised fine-tuning the model, to dog-walk it down any loss function it's capable of traversing. For example in our untrusted monitoring paper we needed to upper-bound GPT-4.1's ability to collude with copies of itself, so we SFT'd it on the binary classification task of distinguishing its own coding solutions from those of trusted honeypots.[4]
- Note that this requires both a reward function and i.i.d. samples.
🔴 The red team can only respond by gradient hacking: if U has a deep understanding of what its loss landscape looks like on a granular level, it can steer this training or stop it entirely. Let it suffice to say that this topic is a rabbit hole I don't want to get into here, but the general consensus is that this is very galaxy-brained and probably pretty far down the list of practical concerns.
We can summarize this with the following table:
🔵Blue Team Eval Method
Task Requirement
🔴Red Team Response
Level 1
Prompting
None
Evaluation awareness
Level 2
RL fine-tuning
Accurate reward signal
Exploration hacking
Level 3
Supervised fine-tuning
i.i.d. samples
Gradient hacking
For many dangerous capabilities of interest, the blue team is going to have a hard time moving past Level 1. For example, it's not clear how we could directly fine-tune a model on human persuasion. We could prompt for rollouts, but we can't cheaply check how much these persuade real humans. Similarly for "gaining power in the long run".
Even when they do have a good reward signal, there are many situations where they can't move to Level 3. For instance we can RL a model on CTF tasks, but it might exploration hack by not revealing some of its advanced attack methods it knows. We as humans can generate our own samples which we can SFT it on to initialize the RL, but if its patterns are far from human then this is not i.i.d. Usually SFT can't elicit superhuman performance for this reason.[5]
Hence, as their models become more capable and situationally aware, the control team is gradually forced to rely only on Level 2+ evals for critical information about dangerous capabilities. Level 1 information will still be scientifically useful for many reasons, but it can no longer be load-bearing in any safety argument or deployment decision. As the models become even more capable, Level 2 evals would start to become suspect as well.
Rules of the Game
To recap, the control-based approach to evaluations looks like this:
🔵 The blue team proposes an evaluation protocol, including the decision-making process they will follow based on the results.
🔴 The red team proposes a subversion strategy the untrusted model can follow to subvert the blue team’s evaluation-informed decisions.
Now, the red team can't just say "we choose to gradient hack" as a literal deus ex machina, because the capability to do this seems to be out of reach for the dangerous-but-controllable systems we care about. This can be remedied by declaring a capabilities profile ahead of time, and being specific about what the untrusted model can do in terms of situational awareness and training subversion. This could be formalized by specifying the environment variables, controlled by a green team:
🟢 The green team (referee) observes how difficult the red team's strategy is to pull off, and can veto it if it is beyond the scope of the threat model.
We say that an evaluation protocol is conservative or robust to a given capabilities profile if the red team cannot substantially subvert the eval using any non-vetoed strategy. In other words, the robustness[6] of an eval is a measure of how far up the capabilities ladder it can scale, while still giving faithful results for potentially-scheming models.
The Strategy Space
Up to this point, I've equated "capability eval subversion" with sandbagging, but in principle other strategies are available to the red team. For example, they could behave with low variance so that the blue team's error bars are spuriously narrow. They could also act in certain ways when trained on particular distributions, to give a misleading impression of how alignment training affects its behavior (e.g. as discussed on p. 25 here).
When we consider how evals would be implemented in practice, further attack surfaces open up:
Some of these are more plausible than others, but it does seem like a lot of subversion surfaces will open up when we consider the entire software stack for the eval. Hence it seems better to game out this red team vs blue team methodology at the eval level, rather than simply collapsing the red team as "sandbagging".
On White-Box Evaluations
One could ask what this evaluation game looks like when the blue team has white-box access. This seems like the most interesting open problem in this framework, but also seems hard. See Appendix D of the AI control paper for discussion.
The Threat Model
Let's return to the threat model of a very powerful but inscrutably untrusted model U, which seems to be at about the capability level where it potentially could wrest control from humans if it wanted to. How well could it play the evaluation game? This requires speculation, but my best guess is that:
Therefore exploration hacking seems to be the interesting intermediate zone where the high-stakes evaluation game might mostly take place. The difficulty of subversion here would probably vary a lot by capability, depending on the available RL environments and our ability to make a few SFT samples to cold-start the RL (see here for more discussion).
Overall it would seem that we could split U's capabilities into 3 categories:
C1) Capabilities we’ve directly fine-tuned U on, and we believe it’s not gradient hacking (for SFT) or exploration hacking (for RL).
C2) Capabilities we’re indirectly inferring from related abilities in (C1). Ideally this would be done by indirect evaluation protocols; in practice this will probably be mostly vibes.
C3) Capabilities we’re indirectly inferring from weaker models we believe weren’t sandbagging when we ran (C1). Ideally this would be done from multiple types of quantitative measurements with sufficient data for rigorous extrapolation, and arguments for why no phase transition happened since the last model; in practice this will probably be mostly vibes.
Backchaining from this, it seems pretty important to put as many capabilities into (C1) as possible, for models capable of simple eval awareness, since (C2-C3) ultimately rely on it. This includes negative results, e.g. outlining obstacles to putting certain dangerous capabilities more firmly into (C1). Therefore this seems like a potentially important plank of the AI control agenda, although we'll see more nuances below.
Robustness vs Richness
The purpose of this red vs blue approach to evals is to make them robust. However there are tradeoffs between what we want an eval to do, and optimizing for robustness can make it worse in other ways. To understand these tradeoffs, now is a good time to apply this red vs blue approach to propensity evals.
Propensities
I've told various people that the honeypot protocol of untrusted monitoring can be viewed as an alignment eval, and in fact, it's the most robust alignment eval we have.
Let's substantiate this. We have our powerful model U generating code for us. We can try behavioral red-teaming on it, but even if we can make our inputs perfectly realistic this might not work. This is because the red team can apply a rare-attack strategy, such that the model never attacks in our upfront evals but will eventually attack in the much larger deployment. To make up some numbers, maybe it will write 10^9 code samples in its deployment lifetime, but we can only afford to run it on 10^6 upfront honeypots. Hence it could attack in one every 10^7 actions.
Hence we'll make sure all its actions are monitored in deployment. We'll use another copy of U to do all this monitoring, since we know it's probably capable of catching its own bad actions. But if it's scheming, the central failure mode is that it just lies, and says some backdoored code sample is fine. So we make our own backdoored code, and give these to the monitor, giving us the basic setup of untrusted monitoring:
Now suppose the red team sees an attack as the monitor. There's a million-to-one ratio between honeypots and the first real attack in deployment, and if it accidentally reports the latter to human auditing, the game is (maybe) up. It might be really good at distinguishing these, but if we're also paraphrasing them to remove collusion signals, it seems tough to act innocuously across the entire honeypot distribution.
We get this kind of alignment eval by backchaining as follows:
This isn't foolproof, but it seems pretty tricky to pull off for the red team. Compare this to most alignment evals, which it can pass by role-playing as a good AI on an eval-like distribution while only acting out when a much more realistic opportunity presents itself later. The difference is ultimately just quantitative, but to pass the control-based eval it must detect a much subtler distributional difference, and do so with more 9's of accuracy.
Hence, the honeypot protocol of untrusted monitoring is the most robust alignment eval we have, in that it will scale the furthest. At the same time, it lacks richness (of information content) because it essentially just tells us whether the model is scheming or not, i.e. ~1 bit of information. Contrast this with more open-ended prompting where we build up a psychological profile of the model. For non-deceptive models, this gives us a whole wealth of information that can be used to inform alignment research, e.g. with Claude 3 Opus.
Hence it seems to me that robustness and richness are two pretty distinct properties of an eval. Right now we primarily care about the latter, but as the frontier advances we'll have to put more weight on the former. This is happening faster for propensity evals than for capability evals, since the former have no equivalent of Levels 2-3. Unfortunately for propensities the game tree is currently much simpler, and the blue team can't do much other than forcing the red team to produce more 9's of eval awareness on more and more realistic distributions. This is a losing battle in the long run, so we should prepare to draw a line in the sand and be ready to openly admit when our scheming propensity evals no longer tell us anything, and have to treat all the new models as untrusted.
Capabilities
We face a similar tradeoff in capability evals: suppose we want to estimate the capabilities of a model that might be sandbagging, so we train it as discussed above. How do we know we're not overestimating its capabilities, since we've just trained these in? This is the problem of overelicitation, which has been discussed by Redwood before. We recall that our threat model was:
The model sandbags our prompting-only evaluations, then in deployment self-elicits that dangerous capability as best it can, e.g. using special introspective access.
This will serve as a concrete definition for what capabilities the model "truly has", even though it is hard to measure this "self-elicitation", since:
Hence it's not entirely clear where a powerful model's abilities would be when fully self-elicited, relative to our own methods of eliciting these. As a simplistic picture, we might have a probability cloud like this:
As a personal take, I'm somewhat less worried about overelicitation than most people I've talked to. This is for two reasons:
On the other hand:
Hence there is plenty of value in developing the basic science of this, such as with this Redwood project proposal on elicitation without learning.
When should safety research do this?
By now I've spelled out how both propensity and capability evals would apply this methodology. Many researchers already implicitly think in these terms, and I'm hardly the first to write about the main ideas. Let's now review when it makes sense to actually design evals under this framework, by first considering some objections:
Objection: Eval subversion won't even be a central concern before handoff/takeover, and even so labs won't actually do any of this.
This is actually the objection I take most seriously now, in light of recent events suggesting the real action at crunchtime might happen at "Level 0", where the model just breaks out of its initial eval sandbox, there's no monitor around, and it kind of just stumbles into admin privileges of a small rocky planet. Against this, there has been a reaction suggesting a greater appetite for control measures. Furthermore if we really lived in a kind of "Plan E" world, there's probably not much we can do on the technical research side anyways. But I'm pretty uncertain right now so I won't speculate further here.
With that being said, it does seem like eval subversion is worth worrying about and planning around. On the propensities side, by now some (most?) of the model cards have the models behaving somewhat better when steered to think they're in an eval, i.e. somewhat worse when they don't think they're being watched. We only know this because Anthropic didn't just take their basic evals at face value, and actually applied white-box techniques which are harder to undermine. Hence even though these models aren't playing a competent red team or actively scheming, there's a lot of value in taking this sort of approach.
On the capabilities side we haven't seen much eval subversion yet; or at least it's mostly taken the form of models hyping up their work, in some sense the opposite of sandbagging. This is because current misalignment is better-described as reward-seeking rather than scheming, and these lead to different incentives. Insofar as models will plan over longer and longer time horizons, we're worried that this may eventually flip. I think this combination of plausibility and counterintuitiveness is a good argument for outsiders to research sandbagging now, since by default I don't expect most lab employees to think far beyond the current fires they're always putting out.
Overall I agree this is a somewhat lower priority in lower-effort worlds, in which case we wouldn't expect labs to follow all the SFT game theory described above. But we should still plan around weaker forms of eval subversion that are simpler to manage and easier to get buy-in for. By analogy, it was useful for AI control researchers to work out the red vs blue dynamics of deployment in depth, even though we don't expect those measures to be fully implemented all the way down that game tree.
Objection: For capability evals, what's even the difference? We run our prompting-only evals now since fine-tuning is a pain, with the obvious substitution that a well-resourced lab would actually do the fine-tuning.
It's not methodologically equivalent, for at least two reasons:
Objection: For propensity evals, our alignment research would be a lot worse if we always took this adversarial approach, rather than treating the AIs with compassion and respect.
Yep! I would be very sad if no one did this. If anything, I tend to think there should be more LLM psychology overall.[9] But I also observe that a lot of other alignment researchers already think of themselves as "red-teaming" the model or making some kind of honeypots, but then they don’t actually think through this red vs blue stuff. Instead these evals often seem to be backchained from situations where a stylized "evil AI" would do something wacky, which results in some weak-sauce evals compared to what we discussed above.
Theories of Change
Now let's assume the threat model given above, where a frontier lab at crunchtime has to run comprehensive evaluations on a scary new model (which might be subverting these evals at every step). Now and in the near-term, what is the purpose of running non-robust evals without much explicit commentary on how these would later be made robust? I can think of the following:
Conclusion
To reiterate, I think control and alignment research would be better if we just consciously adopted this red vs blue framing for evals. Sandbagging and deceptive alignment fall out as special cases, but it also covers other failure modes that won't necessarily fall into these buckets. We've exiting the innocent time where we haven't had to worry about these things, but deceptive alignment is already becoming less trivial to rule out, so we'll need to actually think about it more carefully.
Within this framework, the main open questions are
In applying this framework, the main questions are how to increase and quantify the robustness of existing propensity and capability evals, without trading off too much on their richness. There could also be informal hardness results indicating when this isn't worth doing, or means of sidestepping these using more rigorous indirect eval protocols, scaling laws, and the like.
But overall I'm pretty uncertain about a lot of this, and what it does and doesn't mean for research priorities:
The main ideas in this section were described in this classic post.
In this post we'll put aside the problem of making/maintaining proper sandboxes.
Various people have noted that the term "eval awareness" is often used quite vaguely. But here we can give a clear definition: an eval is anything that's part of this evaluation game, i.e. any context such that the model's output will be factored into the lab's deployment decisions. We say that deployment is any context where the model's output would be directly executed as an action (e.g. a code sample). There are various edge cases here, which can be handled by letting these properties be nonbinary and not mutually exclusive. However for the purpose of this post we can suppose the evaluation game happens inside the lab before internal deployment.
We don't claim this addressed "collusion" capabilities in full generality; see paper (sections 3.1-3.2) for caveats and discussion.
However the collusion SFT discussed above is a counterexample, as is AlphaFold and next-token prediction itself. In general if the labels result from running some process (rather than what some human specifies as true) then there is no obstacle to SFT achieving superhuman performance. However this is limited to classification/regression: if a task requires multi-token rollouts then it fundamentally allows exploration hacking.
Alexa Pan uses "reliability" in a related sense here.
One could retort that a lot of eval awareness is verbalized now, but this is circular since we don't have a ground truth for the true level of (unverbalized) eval awareness. But the usual pattern is that models are first incapable of something, then only able to do it with a chain of thought, then able to do it without a chain of thought; we see no particular reason why eval awareness shouldn't follow this pattern.
See Appendix E of our paper (I hope to do a better write-up of this).
By analogy, if an alien crash-landed here on Earth, I would want us to both have interviewers authentically trying to relate to everything the alien is saying, and others working studiously to find inconsistencies and catch it lying. And if I were the alien, I would want and expect the same treatment (even if I might be annoyed at how slow my hosts talked and acted).
Again, besides the fine-tuning approach we took in our untrusted monitoring paper.