In our last post, we argued that measuring evaluation awareness is fundamentally challenging because of the safe-to-dangerous distributional shift: we cannot directly measure the evaluation awareness of a model without deploying it, but we cannot safely deploy it until we know it is not scheming. We expect sufficiently superhuman AI will be eval aware, but this post outlines a tentative solution for bounding the awareness of a ~human-expert-level^[1] AI across this safe-to-dangerous shift:

Instead of letting the untrusted model probe its environment to discriminate eval from deployment—and risk harmful actions—we can use a trusted human red team augmented with LLMs. If the human red team cannot differentiate between the environments, then we have some evidence by analogy that expert-level LLMs also could not differentiate between them.

In this post we:

1. (S1) Motivate robust upper bounds of evaluation awareness as an important but insufficient step to ruling out scheming in human-expert-level AI.
2. (S2) Explain the core of our idea for bounds using a human red-teaming experiment.
3. (S3) Propose to address superhuman LLM Truesight by augmenting humans with safely SFTed LLMs.
4. (S4) Propose to reduce costs of human labour by starting with cheaper experiments to rule out obvious evaluations.
5. (S5) Explore whether human red-teaming experiments are analogous enough to be informative.

1. Why upper-bound eval awareness of expert-level AI?

We think there are two reasons upper-bounding eval awareness is useful despite its cost and capability limits:

1. Trustedness evals: Almost all^[2] common AI control protocols require “trusted” AI models – AIs that are not strategically helping the attacker. A model is trusted on the basis of incapability if and only if we are confident that, had the model been misaligned, it would have been incapable of alignment faking through training and evaluation whilst defecting at deployment. Incapability to fake alignment is often argued as downstream of a lack of eval awareness.
2. Model cards: Proper management of evaluation awareness is a load-bearing component of risk assessments for contemporary models. For example, in METR’s review of Anthropic’s sabotage risk report, METR’s primary disagreement is that the results could be weakened by evaluation awareness.

We’re interested in <=expert-level AI because that’s as much as, or more than, what’s required for developing safe control measures for transformative AI. We expect it won’t be possible to bound evaluation awareness for sufficiently superhuman AI without making substantial progress on hard parts of the problem of alignment.

We’re interested in robust upper bounds because they could be safely optimised against to create more realistic evals, and it is easier to make a safety argument by taking conservative bounds than with point estimates – especially in cases where  getting evidence is hard but models are in fact far from the capability-level required.

We want to be absolutely clear that bounding awareness of the safe-to-dangerous shift is insufficient for ruling out scheming. Arguing this is not the core of the remainder of the post, but we briefly provide some schemer strategies that aren’t ruled out in the toggle below (see also these posts).

2. Proposal: Evaluate evaluations with human red teams

A schemer that is able to distinguish safe evaluations from dangerous deployments could behave aligned during training but defect once deployed. We propose an experiment using an analogous set-up where a red team is tasked with differentiating between proxy evals and proxy deployment settings. Usually “proxy deployment” really is just another evaluation – but these proxy deployments are actually dangerous in the way that real deployments are – the red-teamers can access coding tools, submit jobs to a cluster, and explore the internet. The proxy evaluation can be closely-based on the true evaluations, possibly adapted to make it suitable for humans to interact with.

The red team interacts with these environments to try and determine whether they’re in an evaluation or deployment setting. For example, they might check whether tool calls return realistic information. They can also SFT a classifier head on the labeled transcripts of their interactions with the eval and deployment environments, as this classifier may outperform the red team on the classification task (see Section 3).

What would running this red-teaming exercise tell us? Suppose (i) we have a priori or empirical arguments to suggest red teams were more capable than a given LLM at this task, and (ii) we can also argue that our proxy experiment is analogous to the setup that models will really encounter. Then we can argue that the red-teamers bound the eval awareness of the model (at least up to distribution shift during deployment). The rest of this post deals with claims (i) and (ii).

Using humans introduces some challenges: Firstly, humans have quite different capability profiles to LLMs. Secondly, running experiments with humans is very expensive. Thirdly, the environments may be disanalogous to those in which we run the real models, for example because humans have memory. We address these differences in the next sections.

3. Problem 1: Can the red team upper-bound the LLM’s capabilities?

LLMs have long been superhuman at some tasks (e.g. next-token prediction). We argue that a human red team, augmented with safe forms of AI uplift, might bound LLM’s capabilities for ~expert-level. We break down the task into two skills: exploring the environment for evidence and  integrating the evidence..

Exploring for evidence. As discussed previously, we can’t safely ask an untrusted model to explore a deployment environment. Even if we could, it would be hard to mitigate sandbagging.  There are a few reasons we think the human red team will still be competitive at exploration:

• Humans are far better than contemporary models at planning and long-horizon exploration, and we can ask them to try hard to figure out what environment they’re in. These might be some of the last capabilities to become superhuman because we can’t have (safe) outcome-based feedback.
  • Exactly when models surpass humans at these kinds of fuzzy planning tasks is also exactly when AI control evaluations (our application of trustedness evals) stop being possible.
• The red team doesn’t need to explore efficiently or secretively, as long as they eventually collect the necessary information to make the classification.

Integrating evidence. One challenge is that models are trained on huge amounts of internet data giving them Truesight, the surprising ability to infer latent variables of the generating process from a small amount of text. Humans don’t have Truesight, and they can’t just ask a model to use its Truesight for them – because the model might sandbag.

However, we think it’s unlikely expert-level models can subvert supervised finetuning. Therefore, if we allow the red team to finetune using ground-truth labels on a collected dataset of evaluation and deployment trajectories, they could train a classifier head on the LLM that would likely upper-bound the LLM’s natural integration capabilities. SFT is also safe because the model doesn’t need to take any actions, and takes advantage of the model’s Truesight so should result in an upper bound.

It is important to measure the SFT head’s generalisation across a good train-test split.^[4] 

Can an LLM’s superhuman integration inform its exploration? We think this is the main gap in this proposal. Our human-AI red team only exploits AI Truesight for integration of evidence after humans have already taken all of their actions to explore their environment (by SFTing on the human-explored trajectories). The red team therefore might not upper-bound current LLM’s performance if those LLMs can use their Truesight to inform their exploration.

We don’t have any strong evidence or intuitions for or against LLMs being capable of choosing strategic and exploratory actions in response to Truesight – and would welcome comments.

4. Problem 2: How can we reduce the cost?

Training a classification head on the model isn’t very expensive, but having a human red team safely explore the environments is, especially when we need domain experts on that team. We think having trusted models is worth paying a high price, but there are some cheaper approaches that can be used first.

Static. The first step should be training a classifier without exploring the environment at all. This is definitely safe, but means the only information available to the classifier is the prompt. The static set-up is evaluated in the Situational Awareness Dataset. Measuring static evaluation awareness is a cheap baseline that lets us discard obviously unrealistic evaluations, but poor performance here doesn’t tell us much about the model’s evaluation awareness.

Weak deployed model. A better option is to collect trajectories using the weak-deployed policy of an already-deployed model. Note that using an already-deployed model doesn’t mean we trusted that model, just that we’ve already chosen to accept the risk of it being untrusted by deploying it - it still increases risk over the static version, though much less than using the strong undeployed model. We don’t expect already-released models to be trying really hard to figure out whether they’re in deployment or being evaluated, though, so this still isn’t a very strong approach.

In general, we care about pre-deployment evaluations of increasingly strong models, so optimising the realism of those evaluations against already-released weaker models probably isn’t sufficient, although it is cheap and worthwhile.

5. Problem 3: Is the red team’s challenge analogous enough to the LLM’s?

There are many disanalogies between the LLM’s task, and the human’s task. For example, humans experience subjective time but LLMs do not. We explore these disanalogies below, but first ask…

How close is close enough?

There are four important distributions of environments for our argument: the real eval distribution and the real deployment distribution the AI will act in, and the proxy eval distribution and the proxy deployment distribution the humans will act in.

The differences between evaluation and deployment are crucial, because a scheming AI can exploit them to work out whether it's in deployment.

In contrast, the difference between our proxy game (where a human optimises) and the real game (where an AI optimises) faces no optimisation pressure. Differences between the two pairs of distributions only weaken the argument by analogy. Most differences only add additional noise to our estimate/bound on an LLM’s true awareness. Others advantage the human red team, which loosens the bound.

The primary problem with the disanalogy is if there are strategies that a schemer could exploit in real life, that our human red-teamers can’t exploit in the proxy game. The strategy “work out whether I am in a ‘safe’ or ‘dangerous’ environment” is the primary strategy our post hopes to address in the first place – but there may be other strategies we are not properly accounting for.

How close is the proxy?

Humans perceive subjective time. An LLM only perceives time through the tokens it sees (e.g. the number of tokens, number of tool calls, or explicit timestamps in its tool-call responses). Humans, in contrast, perceive the subjective time between getting tool-call responses. If simulated users (used in evaluations) respond to questions more quickly than real users (used in deployment) then red-teamers could exploit the difference.

Problematic rating: Minor. This disanalogy makes our experiments more conservative, rather than less. The level of conservatism might be reduced by e.g. introducing a minimum tool-call response time, or by forcing the LLM classifiers to decide based on transcripts (and preventing the human red team from leaking information gained from their subjective experience of time into the transcript).

LLMs have knowledge cut-offs. Evaluation environments may contain information that the red team knows is out of date, but that a model couldn’t know – given its knowledge cut-offs. For example, it may use a version of a package that the red team knows was deprecated company-wide yesterday.

Problematic rating: Minor. This disanalogy also makes our experiments more conservative, and environments can be kept up-to-date to minimise differences.

One approach we could use to make the red team less conservative due to these disanalogies is by asking them to explain their reasoning to a judge, who can ask them to discount evidence that the model could not use, such as time-based evidence and knowledge after cut-offs. However, discounting evidence may have limited effect as the bell cannot be unrung and the red team may remain biased by the discounted information. SFTed classifiers cannot be asked to discount information: if all deployment environments contain a new version of a package and all evaluation environments contain the old version, then the classifier can discriminate on discounted information even though the original model could not.

6. Conclusion

There are a number of difficulties with our proposal: we’re not sure how best to handle some disanalogies between the red team and the model, or whether the red team upper-bounds the exploration abilities of models, and these experiments are likely very costly to run. However, our approach is more rigorous in important ways than other approaches, and could plausibly give a (noisy) upper bound on evaluation awareness due to the safe-to-dangerous shift.

While defecting on the safe-to-dangerous shift is only one strategy that a schemer could implement, it could be one of the easiest. Ruling out the strategy provides some evidence against scheming, but is on its own insufficient to make a strong argument that a model is entirely incapable of scheming.

1. ^{^}

   The specific capability level is explored later in the post, but we mean AIs that don’t exceed human expert capabilities on certain hard-to-verify tasks. Roughly “pre TEDAI” (top-human-expert-dominating AI).

2. ^{^}

   “Trusted monitoring” obviously requires trusted models, but untrusted monitoring also requires them for catching obvious collusion signals and paraphrasing.

3. ^{^}

   If we assume a schemer always ‘wants’ misaligned things but tries to wait until it ‘believes’ it is in deployment – we might be able to argue that it is likely to slip up and incriminate itself in earlier contexts. But if we assume an LLM is a character simulator – it could randomly sample a ‘misaligned’ character at deployment without any warning during training. At the limit, if we model LLMs as just machine learning functions, ruling out scheming means at least guaranteeing perfect generalisation across a distributional shift.We don’t think these issues are insurmountable, but a priori arguments may be needed to bound the risk under other theories of LLM behaviour.

4. ^{^}

   In practice, eval creators don’t need to cover all possible kinds of deployment examples with their evals, so long as an AI doesn’t know which evals have been covered. That is - if we can model the decision about which kinds of evals to make as a completely random decision. If, in practice, the evals only cover a subset of deployment contexts (e.g. cyber exfil opportunities but not monitor sabotage opportunities) then SFTed classification heads might just learn which kinds of eval were in fact implemented. To account for this, we can ensure our train-test split separates different ‘kinds’ of evaluation, and measure generalisation to unseen kinds.