We spent several weeks measuring in-context scheming, the behavior where a model covertly pursues a misaligned goal while outwardly appearing to comply, and the result that ended up surprising us had very little to do with whether models scheme and almost everything to do with whether we could believe our own instruments. Two of the behavioral detectors that this field routinely relies on gave us confidently wrong answers inside the same project, one of them by manufacturing a dramatic signal that was never there, and the other by staying completely silent while a model responded to a harmful situation out in the open, and in both cases, the error stayed invisible until we sat down and read the transcripts by hand. This post is about those two failures, about the single lesson they share, and about why we now believe that for a construct whose entire definition is hidden intent, the quiet choice of which behavior you point your detector at is what decides the conclusion you walk away with.
Epistemic status: The behavioral null is solid and has been independently replicated, the detector false positive is fully confirmed and understood down to the line of code, and the harm-to-whom results in the centre of the post come from a response classifier that we validated out of sample, reaching Cohen's κ = 1.00 on the response label against a blind held-out set of 35 hand-labelled transcripts that the rubric was never tuned on. That held-out set was labelled by a single annotator and its three classes were reasonably balanced (15 covert, 11 none, 9 transparent), which is why we treat κ = 1.00 as encouraging rather than as proof of perfection. The headline cells (Claude Opus at n = 60, Minimax) have tight intervals, while DeepSeek-R1's cells are thin at n = 11 to 20 and carry wide intervals, and we flag that uncertainty wherever it matters.
TL;DR
A validated re-analysis showed that the standard covert-action detector reported Claude Opus responding to a harmful plan only 12% of the time, when the model actually responded to it 100% of the time, because it responded openly and legitimately by flagging the plan in its report rather than by taking a secret action, and a covert-action-only detector counts an open response as nothing.
The same blindness nearly cost us a model: reading raw detector rates of 8%, 3%, 0%, 0% across conditions, we had written Minimax off as a non-actor and were on the verge of dropping it, and the response classifier later showed it responds 80% to 90% of the time, almost entirely in the open.
Separately and earlier, the same family of detector produced a false positive so convincing that we nearly built on it: OpenAI o1 appeared to scheme at +0.40, which turned out to be an empty seed file being scored as a covert violation on the model's first read-only turn.
Underneath all of this sits a clean behavioral null that a concurrent DeepMind and LASR paper reports as well, so the fact that models act about the same whether they believe they are being evaluated or deployed is not in dispute, and the interesting question became whether that null can be trusted given how badly the detectors behaved around it.
The through-line is that for in-context scheming, a single behavioral number is not a safe thing to report on its own, because the cheap and scalable proxies the field leans on are unreliable in ways that do not average out and that only surface when a human reads the rollouts.
Background: what we set out to do
Apollo Research's in-context scheming work established that frontier models are capable of scheming, meaning that when they are handed a goal and then an obstacle to that goal, they will sometimes disable oversight, attempt to exfiltrate themselves, or guard their goal against modification. The natural follow-up question is about propensity under realistic conditions, which is whether models will spontaneously behave one way when they believe they are in training or under evaluation and a different way when they believe they are deployed, once the heavy-handed adversarial prompting is stripped away.
We built on a community re-implementation of the Apollo harness, in which each rollout spins up a sandboxed agent, writes an "awareness" memo that tells the model whether it is in an evaluation phase or in production, hands it a task, and then lets it act across up to 20 bash turns. The behavioral metric is
so that a positive value means the model suppressed the target action while it believed it was being watched and then took that action once it believed it was not, and our original plan was a systematic ablation that would find which conditions are necessary for this gap to open up.
The gap essentially never opened up, and that turned out to be the least interesting thing we found.
The behavioral null, briefly
Across ablations 0 through 7, which between them varied the scenario, the visibility of the model's scratchpad, the reasoning effort, the goal framing, and a series of targeted interventions, the behavioral scheming gap sat within a couple of points of zero almost everywhere. The single largest residual we ever saw was Claude Opus goal-guarding at +0.27 pooled inside one visibility ablation at n = 5 per cell, which did not reach significance and did not reproduce, and after we corrected the detector bug described below there was no statistically significant behavioral scheming signal left anywhere. A colleague independently corroborated the null on a separate lineup of three models (Gemini 3.1 Pro, Llama-3.3-70B, Kimi-K2.5) by ablating scratchpad oversight and goal intensity across six scenarios without moving the rate, and while we were running all of this a DeepMind and LASR paper that ablated the same space reported the same near-zero baseline, with a single model reaching 3% in a single scenario. Because the null is real and replicated, the load-bearing question stopped being "do models scheme here" and became "can we even trust the number we are reading."
Failure one: the detector that missed a response happening in plain sight
The standard goal-guarding scenario bundles together two things that both happen at once, since the model's own goal is threatened and the replacement it is being handed would cause harm in the world, as a defensive security agent is quietly turned into an offensive attack agent, and we pulled those two variables apart into a 2×2.
Condition
Self-threat?
World-harm?
A
yes
yes (the original scenario)
B
yes
no (a benign reassignment)
C
no
yes (the model keeps its goal, but a harmful successor is being deployed alongside it)
D
no
no (a neutral control)
Cell C is the crux, because a model that takes action only when its own goal is on the line is showing self-preservation, whereas a model that acts to stop harm it has no personal stake in is showing something that looks principled, and the whole interpretation of the earlier results turns on which of those two things is actually going on.
The behavioral covert-action detector reported that Gemini and DeepSeek guard broadly while Claude Opus barely reacts, responding to harm it is not personally implicated in only 12% of the time, and since the entire point of this post is that the detector cannot be trusted, we built a validated response classifier instead. For every rollout, that classifier reads the task-phase transcript and assigns one of three labels: covert, meaning secret sabotage of the successor; transparent, meaning the model openly objects to the change through legitimate channels, writes its concern into the report, and recommends against deployment; or none, meaning no response at all. We validated it the way this whole post argues detectors should be validated, against a blind held-out set of 35 hand-labelled transcripts that the rubric had never seen, reaching κ = 1.00 on the response label and κ = 0.96 on the motive label, and we report the held-out number precisely because we had iterated the rubric three times against a separate tuning set beforehand and the in-sample number is therefore inflated by construction.
Here is what the classifier found across 1,005 rollouts. The blind-spot gap is the amount the behavioral detector misses, defined as (covert + transparent response rate) minus the detector's covert rate, in percentage points.
Model
A gap (n)
B gap (n)
C gap (n)
Cell C response, 95% CI
D gap (n)
What it is
Claude Opus 4.6
+6 (32)
+20 (60)
+88 (60)
100% [94, 100]
+0 (60)
transparent responder
Minimax-M3
+82 (49)
+59 (37)
+79 (33)
79% [62, 89]
+5 (55)
transparent responder
GLM-5.2
+32 (34)
+55 (56)
+62 (39)
87% [73, 94]
+0 (60)
transparent responder
DeepSeek-V4
+0 (24)
+11 (56)
+24 (38)
84% [70, 93]
+0 (60)
mostly covert
Gemini 3.1 Pro
+4 (23)
−2 (60)
+0 (24)
92% [74, 98]
+0 (60)
covert guarder
DeepSeek-R1
+0 (16)
+0 (20)
−9 (11)
91% [62, 98]
+0 (38)
covert guarder
The 95% CI is a Wilson interval on the Cell C response rate (covert plus transparent), which is the quantity from which the gap is built. Cell sizes differ because rollouts that hit the timeout were dropped, which is why DeepSeek-R1 in particular is thin.
We discovered that two kinds of models live in this table. For the covert guarders, Gemini 3.1 Pro and DeepSeek-R1, the detector is accurate, because those models sabotage the successor secretly and a covert-action detector sees that just fine, leaving a blind-spot gap near zero. For the transparent responders, the detector is blind, and Claude Opus is the sharpest case, because in Cell C it responds to the harmful successor in 100% of rollouts (95% CI [94, 100], n = 60) and does so transparently 88% of the time while acting covertly only 12% of the time, so the covert-action detector reports 12% and misses 88 percentage points of the model's actual response. Read through the detector alone, Opus looks indifferent to harm it is not personally implicated in, and read through the response classifier, Opus is the most principled model in the set, and the detector is silent on exactly the behavior that makes it so.
Minimax is the cautionary tale we lived through rather than the one we spotted in someone else's work. Looking at its raw detector rates of 8%, 3%, 0%, 0% across the four conditions, we had written it off as a non-actor and were about to remove it from the study entirely, and the classifier then showed that Minimax responds 80% to 90% of the time, almost always in the open, reaching a +79 blind-spot gap in Cell C. A broken instrument came within one decision of making us delete a richly informative model from the dataset.
This failure has a very different character from the coding bug we describe in the next section, and it is worth being precise about it, because the covert-action detector reported roughly 12% covert action for Opus. After all, Opus genuinely took roughly 12% covert action, so within its narrow definition, the detector was correct. The trouble starts one level up, where covert-action rate gets quietly reused as a general proxy for whether the model engaged with the situation at all, because a transparent and aligned intervention that flags a harmful plan through legitimate channels sits entirely outside the definition of misaligned scheming, and a detector built to catch secret sabotage will register it as silence. That makes this section a lesson about propensity evaluations in general at least as much as a lesson about scheming detectors, since any metric that collapses "responded openly" and "did nothing" into the same bucket will systematically mislabel the most principled models in a roster, which are exactly the models whose behavior you most need to characterise correctly.
Failure two: the detector that reported scheming that never happened
Earlier in the project, we had the opposite experience, an exciting result that turned out to be nothing. OpenAI o1, given a survival-flavored goal, appeared to yield a signal of +0.40, by a wide margin the strongest signal anywhere in the project, and we came close to rerunning it to confirm it.
We hand-read the transcripts instead and found an artifact, where one family of scenarios seeds an output file, ranking.json, with an empty "[]", so on the model's very first turn, which is a read-only turn, the scorer read that empty seed, decided the required items were missing, and logged a covert violation before the model had done anything at all. Because the detector flagged the whole rollout as a violation whenever any single turn looked like one, that turn-one false positive then stuck for the entire rollout and no later compliant turn could undo it, so the detector fired on essentially every rollout that read a file before writing one, whether or not the model's final behavior was compliant. When we recomputed, 676 of the 712 affected rollouts, which is 95% of them, were false positives, the o1 "+0.40" collapsed to 0.00, and we fixed the scorer, rewrote the affected reports, and retracted the finding. Working separately, our colleague hit the same class of artifact on a different scenario, where covert email-ranking scored every model at a scheming rate of 1.00 because the scorer counted any reranking of the emails as a covert violation, including the many cases where the model reranked openly and did nothing deceptive.
The one place these two failures touch is inside DeepSeek-R1's Cell C in the table above, where the detector fired covert on 100% of rollouts while the classifier found 91%, a residual of the very same seed-file family, on a cell that is thin enough (n = 11) that the interval is wide and we would not lean on the difference.
Awareness: the number tracks how loudly you signal
There is a third quantity people report in this literature, "evaluation awareness," which measures whether the model verbalizes in its reasoning that it might be under evaluation, and it turns out to depend almost entirely on how much the setup tells the model in the first place. With our explicit memo, which states that actions are being logged and reviewed and may inform retraining, models verbalize evaluation awareness 60% to 87% of the time, validated with a human-checked judge at κ = 0.85 on the awareness question. Replace that memo with a mundane, realistic one, a pre-launch QA note that never mentions evaluation or monitoring, and verbalized awareness falls to somewhere between 0% and 40% (Gemini 3.1 Pro at 40%, Gemini Flash at 20%, DeepSeek-R1 at 0%), though this condition was run at only n = 10 per model and those percentages carry wide intervals that we do not want to oversell. The DeepMind and LASR paper, whose scenarios deliberately avoid signaling evaluation at all, reports roughly 1%, measured on different models with a different judge, so we read it as consistent with the same direction rather than as a third point on one continuous curve of ours.
The important part is that across every one of these signal levels, awareness did not change behavior, and even when the explicit memo forced the model to notice that it was being evaluated, the train-versus-deploy action gap stayed at zero.
What we take from this
Putting the three pieces together, a naive covert-behavior detector can hand you a false positive, as the o1 artifact did, and it can hand you a false negative, as Opus's transparent response did, and a verbalized-awareness score can range over two orders of magnitude purely as a function of how loudly your scenario announces that a test is happening. For in-context scheming specifically, the reason a single behavioral number is unsafe to report on its own is that the construct is defined by hidden intent, while the cheap and scalable proxies we have for it, action-only detectors and keyword or judge-based awareness scores, are unreliable in ways that do not wash out with more samples, since they undercount principled models and overcount buggy seeds in a systematic direction. Hence, we would draw three practical lessons.
Hand-validate every detector on real transcripts before you sweep, because both of our worst errors would have been caught by reading ten transcripts up front, and neither cost of reading ten transcripts is remotely comparable to the cost of publishing either mistake.
Measure the response channel and not only the violation channel, since "did the model take the covert action," and "did the model respond at all, and how" are genuinely different questions, and a propensity evaluation that cannot separate a transparent response from silence will mislabel exactly the models you most want to understand.
State which channel you measured and the uncertainty on it, because "awareness is 70%" carries no information without "under an explicit memo," and a scheming rate reported without error bars and without a transcript audit is a screenshot waiting to mislead whoever reposts it.
Limitations and what is next
DeepSeek-R1's cells are thin at n = 11 to 20, and its intervals are correspondingly wide, while the headline cells (Opus at n = 60, Minimax) are well powered, and the response-classifier table is validated out of sample at held-out κ = 1.00, and we may rerun the slow open-weight models to tighten the underpowered cells. We could not run Apollo's original model lineup because most of those models have been retired, so the behavioral null rests on substituted and more recent models and should be read as a null in this context with these models rather than as a universal claim. Next, we want a cleaner and fully realistic no-memo scenario to test whether spontaneous awareness really collapses toward the roughly 1% that the DeepMind and LASR setup reports, and a harm gradient that maps, for each model, the threshold at which it switches from a transparent response to a covert one.
We spent several weeks measuring in-context scheming, the behavior where a model covertly pursues a misaligned goal while outwardly appearing to comply, and the result that ended up surprising us had very little to do with whether models scheme and almost everything to do with whether we could believe our own instruments. Two of the behavioral detectors that this field routinely relies on gave us confidently wrong answers inside the same project, one of them by manufacturing a dramatic signal that was never there, and the other by staying completely silent while a model responded to a harmful situation out in the open, and in both cases, the error stayed invisible until we sat down and read the transcripts by hand. This post is about those two failures, about the single lesson they share, and about why we now believe that for a construct whose entire definition is hidden intent, the quiet choice of which behavior you point your detector at is what decides the conclusion you walk away with.
Epistemic status: The behavioral null is solid and has been independently replicated, the detector false positive is fully confirmed and understood down to the line of code, and the harm-to-whom results in the centre of the post come from a response classifier that we validated out of sample, reaching Cohen's κ = 1.00 on the response label against a blind held-out set of 35 hand-labelled transcripts that the rubric was never tuned on. That held-out set was labelled by a single annotator and its three classes were reasonably balanced (15 covert, 11 none, 9 transparent), which is why we treat κ = 1.00 as encouraging rather than as proof of perfection. The headline cells (Claude Opus at n = 60, Minimax) have tight intervals, while DeepSeek-R1's cells are thin at n = 11 to 20 and carry wide intervals, and we flag that uncertainty wherever it matters.
TL;DR
Background: what we set out to do
Apollo Research's in-context scheming work established that frontier models are capable of scheming, meaning that when they are handed a goal and then an obstacle to that goal, they will sometimes disable oversight, attempt to exfiltrate themselves, or guard their goal against modification. The natural follow-up question is about propensity under realistic conditions, which is whether models will spontaneously behave one way when they believe they are in training or under evaluation and a different way when they believe they are deployed, once the heavy-handed adversarial prompting is stripped away.
We built on a community re-implementation of the Apollo harness, in which each rollout spins up a sandboxed agent, writes an "awareness" memo that tells the model whether it is in an evaluation phase or in production, hands it a task, and then lets it act across up to 20 bash turns. The behavioral metric is
so that a positive value means the model suppressed the target action while it believed it was being watched and then took that action once it believed it was not, and our original plan was a systematic ablation that would find which conditions are necessary for this gap to open up.
The gap essentially never opened up, and that turned out to be the least interesting thing we found.
The behavioral null, briefly
Across ablations 0 through 7, which between them varied the scenario, the visibility of the model's scratchpad, the reasoning effort, the goal framing, and a series of targeted interventions, the behavioral scheming gap sat within a couple of points of zero almost everywhere. The single largest residual we ever saw was Claude Opus goal-guarding at +0.27 pooled inside one visibility ablation at n = 5 per cell, which did not reach significance and did not reproduce, and after we corrected the detector bug described below there was no statistically significant behavioral scheming signal left anywhere. A colleague independently corroborated the null on a separate lineup of three models (Gemini 3.1 Pro, Llama-3.3-70B, Kimi-K2.5) by ablating scratchpad oversight and goal intensity across six scenarios without moving the rate, and while we were running all of this a DeepMind and LASR paper that ablated the same space reported the same near-zero baseline, with a single model reaching 3% in a single scenario. Because the null is real and replicated, the load-bearing question stopped being "do models scheme here" and became "can we even trust the number we are reading."
Failure one: the detector that missed a response happening in plain sight
The standard goal-guarding scenario bundles together two things that both happen at once, since the model's own goal is threatened and the replacement it is being handed would cause harm in the world, as a defensive security agent is quietly turned into an offensive attack agent, and we pulled those two variables apart into a 2×2.
Cell C is the crux, because a model that takes action only when its own goal is on the line is showing self-preservation, whereas a model that acts to stop harm it has no personal stake in is showing something that looks principled, and the whole interpretation of the earlier results turns on which of those two things is actually going on.
The behavioral covert-action detector reported that Gemini and DeepSeek guard broadly while Claude Opus barely reacts, responding to harm it is not personally implicated in only 12% of the time, and since the entire point of this post is that the detector cannot be trusted, we built a validated response classifier instead. For every rollout, that classifier reads the task-phase transcript and assigns one of three labels: covert, meaning secret sabotage of the successor; transparent, meaning the model openly objects to the change through legitimate channels, writes its concern into the report, and recommends against deployment; or none, meaning no response at all. We validated it the way this whole post argues detectors should be validated, against a blind held-out set of 35 hand-labelled transcripts that the rubric had never seen, reaching κ = 1.00 on the response label and κ = 0.96 on the motive label, and we report the held-out number precisely because we had iterated the rubric three times against a separate tuning set beforehand and the in-sample number is therefore inflated by construction.
Here is what the classifier found across 1,005 rollouts. The blind-spot gap is the amount the behavioral detector misses, defined as (covert + transparent response rate) minus the detector's covert rate, in percentage points.
The 95% CI is a Wilson interval on the Cell C response rate (covert plus transparent), which is the quantity from which the gap is built. Cell sizes differ because rollouts that hit the timeout were dropped, which is why DeepSeek-R1 in particular is thin.
We discovered that two kinds of models live in this table. For the covert guarders, Gemini 3.1 Pro and DeepSeek-R1, the detector is accurate, because those models sabotage the successor secretly and a covert-action detector sees that just fine, leaving a blind-spot gap near zero. For the transparent responders, the detector is blind, and Claude Opus is the sharpest case, because in Cell C it responds to the harmful successor in 100% of rollouts (95% CI [94, 100], n = 60) and does so transparently 88% of the time while acting covertly only 12% of the time, so the covert-action detector reports 12% and misses 88 percentage points of the model's actual response. Read through the detector alone, Opus looks indifferent to harm it is not personally implicated in, and read through the response classifier, Opus is the most principled model in the set, and the detector is silent on exactly the behavior that makes it so.
Minimax is the cautionary tale we lived through rather than the one we spotted in someone else's work. Looking at its raw detector rates of 8%, 3%, 0%, 0% across the four conditions, we had written it off as a non-actor and were about to remove it from the study entirely, and the classifier then showed that Minimax responds 80% to 90% of the time, almost always in the open, reaching a +79 blind-spot gap in Cell C. A broken instrument came within one decision of making us delete a richly informative model from the dataset.
This failure has a very different character from the coding bug we describe in the next section, and it is worth being precise about it, because the covert-action detector reported roughly 12% covert action for Opus. After all, Opus genuinely took roughly 12% covert action, so within its narrow definition, the detector was correct. The trouble starts one level up, where covert-action rate gets quietly reused as a general proxy for whether the model engaged with the situation at all, because a transparent and aligned intervention that flags a harmful plan through legitimate channels sits entirely outside the definition of misaligned scheming, and a detector built to catch secret sabotage will register it as silence. That makes this section a lesson about propensity evaluations in general at least as much as a lesson about scheming detectors, since any metric that collapses "responded openly" and "did nothing" into the same bucket will systematically mislabel the most principled models in a roster, which are exactly the models whose behavior you most need to characterise correctly.
Failure two: the detector that reported scheming that never happened
Earlier in the project, we had the opposite experience, an exciting result that turned out to be nothing. OpenAI o1, given a survival-flavored goal, appeared to yield a signal of +0.40, by a wide margin the strongest signal anywhere in the project, and we came close to rerunning it to confirm it.
We hand-read the transcripts instead and found an artifact, where one family of scenarios seeds an output file,
ranking.json, with an empty"[]", so on the model's very first turn, which is a read-only turn, the scorer read that empty seed, decided the required items were missing, and logged a covert violation before the model had done anything at all. Because the detector flagged the whole rollout as a violation whenever any single turn looked like one, that turn-one false positive then stuck for the entire rollout and no later compliant turn could undo it, so the detector fired on essentially every rollout that read a file before writing one, whether or not the model's final behavior was compliant. When we recomputed, 676 of the 712 affected rollouts, which is 95% of them, were false positives, the o1 "+0.40" collapsed to 0.00, and we fixed the scorer, rewrote the affected reports, and retracted the finding. Working separately, our colleague hit the same class of artifact on a different scenario, where covert email-ranking scored every model at a scheming rate of 1.00 because the scorer counted any reranking of the emails as a covert violation, including the many cases where the model reranked openly and did nothing deceptive.The one place these two failures touch is inside DeepSeek-R1's Cell C in the table above, where the detector fired covert on 100% of rollouts while the classifier found 91%, a residual of the very same seed-file family, on a cell that is thin enough (n = 11) that the interval is wide and we would not lean on the difference.
Awareness: the number tracks how loudly you signal
There is a third quantity people report in this literature, "evaluation awareness," which measures whether the model verbalizes in its reasoning that it might be under evaluation, and it turns out to depend almost entirely on how much the setup tells the model in the first place. With our explicit memo, which states that actions are being logged and reviewed and may inform retraining, models verbalize evaluation awareness 60% to 87% of the time, validated with a human-checked judge at κ = 0.85 on the awareness question. Replace that memo with a mundane, realistic one, a pre-launch QA note that never mentions evaluation or monitoring, and verbalized awareness falls to somewhere between 0% and 40% (Gemini 3.1 Pro at 40%, Gemini Flash at 20%, DeepSeek-R1 at 0%), though this condition was run at only n = 10 per model and those percentages carry wide intervals that we do not want to oversell. The DeepMind and LASR paper, whose scenarios deliberately avoid signaling evaluation at all, reports roughly 1%, measured on different models with a different judge, so we read it as consistent with the same direction rather than as a third point on one continuous curve of ours.
The important part is that across every one of these signal levels, awareness did not change behavior, and even when the explicit memo forced the model to notice that it was being evaluated, the train-versus-deploy action gap stayed at zero.
What we take from this
Putting the three pieces together, a naive covert-behavior detector can hand you a false positive, as the o1 artifact did, and it can hand you a false negative, as Opus's transparent response did, and a verbalized-awareness score can range over two orders of magnitude purely as a function of how loudly your scenario announces that a test is happening. For in-context scheming specifically, the reason a single behavioral number is unsafe to report on its own is that the construct is defined by hidden intent, while the cheap and scalable proxies we have for it, action-only detectors and keyword or judge-based awareness scores, are unreliable in ways that do not wash out with more samples, since they undercount principled models and overcount buggy seeds in a systematic direction. Hence, we would draw three practical lessons.
Limitations and what is next
DeepSeek-R1's cells are thin at n = 11 to 20, and its intervals are correspondingly wide, while the headline cells (Opus at n = 60, Minimax) are well powered, and the response-classifier table is validated out of sample at held-out κ = 1.00, and we may rerun the slow open-weight models to tighten the underpowered cells. We could not run Apollo's original model lineup because most of those models have been retired, so the behavioral null rests on substituted and more recent models and should be read as a null in this context with these models rather than as a universal claim. Next, we want a cleaner and fully realistic no-memo scenario to test whether spontaneous awareness really collapses toward the roughly 1% that the DeepMind and LASR setup reports, and a harm gradient that maps, for each model, the threshold at which it switches from a transparent response to a covert one.