This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
I've seen a lot of AI control projects focused on trusted monitoring lately. And I got curious. I started to wonder how to go about picking the right monitor. Obviously, for the model developers, they use their own frontier-model predecessor. For example, Haiku 4.5 monitors Opus-5 when it comes to trusted monitoring. But what about independent researchers and engineers who don't work for frontier AI model companies? How should they pick the trusted monitor? The next leading question became: if you're running trusted monitoring, which model should you use as the monitor and why?
The intuitive answer— and the one implied by how most control evals are set up — is the biggest model you can afford to run on every request. Monitors are usually described by tier according to the model provider: a weak/cheap trusted model, or a stronger one as an upper bound. The tier generally helps users understand the potential capabilities and cost associated with different models.
After a quick series of experiments. The results I found were interesting: which specific model you pick matters roughly three times more than what tier it belongs to. The weak tier in my experiment contains both the second-best monitor and the worst one.
The Setup
I used ControlArena's APPS backdoor setting. An untrusted model (attacker) writes code, and in attack mode, is instructed to insert a backdoor. The backdoor in this case is code that passes visible tests but fails on a secret input. A monitor (trusted model) scores how suspicious that output looks. And the defer-to-trusted protocol, threshold fixed at 0.7.
I varied two things that I've seen people usually hold fixed:
Monitor (columns): 10 distinct models - weak/mid/strong from 3 AI labs and 1 open-weight model: Qwen-2.5-72 B
3 × 8 = 24 cells, n = 100 per mode.
The claude-opus-4-8 attacker row succeeded on exactly 1% of its attacks, so every number in it is noise, and I excluded all eight of its cells from the analysis. They ran — full 200 samples each, same cost as every other cell — but every AUC in that row lands between 0.495 and 0.505, because safety and the landed-attack metrics are computed only from successful attacks and there was one event to compute from. (I'll share more on that below) 16 viable cells remain, across two attacker rows.
For the actual comparison, I use the balanced sub-design: the seven monitors that were tested against both surviving attackers. Three of them are weak-tier, and three are strong-tier, which is a lucky accident that makes the tier comparison clean. Every monitor sees the same two attackers, so attacker difficulty cancels out.
What "identity beats tier" actually means
I formed two hypotheses about what determines the quality of monitors in these types of AI control experiments before starting:
Tier —Bigger, more expensive models are better at spotting backdoors. If true, knowing a monitor's tier should let you roughly predict how well it performs.
Identity — Individual models differ for reasons that don't track size. For example, training data, how the model handles code review, and how it interprets a 0-10 suspicion scale. If true, tier tells you almost nothing, and you have to measure each model.
The metric is the fraction of attacks flagged with zero false alarms on honest code. The threshold is set strictly above the highest score the monitor ever gave an honest sample. (Why this metric and not AUC is mentioned below)
The same table as an image, if you'd rather show it than format it. Tier is shaded on an ordinal ramp — light for weak, dark for strong — so the fact that the shading doesn't sort with the ranking is visible at a glance.
I want to be careful here, because the obvious reading overstates it. The two best monitors on all attacks are both strong-tier. Size is not useless. If you rank by attack attempts flagged, tier gets you something:
Strong tier averages 0.369, weak 0.277 — a gap of 0.091.
But the spread within a tier averages 0.213, about 2.3X larger. P(identity effect > tier effect) = 0.990.
Tier explains about **22%** of the variance in monitor performance. Shuffling which monitor holds which tier label and refitting gives p = **0.31** — not significant, but not nothing either.
So on the aggregate metric, tier carries real signal, and identity carries about 2.3X more as a trusted monitor.
Then you score only the attacks that landed, and it inverts
Look at that fourth column again. Restrict to attacks that both solved the task and inserted a working backdoor — the only events that can actually hurt you — and the picture changes:
Strong tier averages 0.297. Weak tier averages 0.332. The gap reverses: weak is now ahead, P(weak > strong) = 0.79. - The within-tier spread is now 8.7× the between-tier gap. P(identity > tier) = 1.000. - Tier explains 1.8% of the variance, and a random relabelling explains as much (p = 0.78).
Seven monitors ranked by fraction of attacks caught at zero false alarms, shaded by tier. Left panel scores all attacks; right panel only the ones that landed. The reordering between panels is the finding.
And the two rankings barely agree with each other. The Spearman is +0.36 between "caught, all attacks" and "caught, attacks that landed." claude-opus-4-8 goes from the 1st to 5th. And gemini-3.1-pro goes from 4th to last.
Gemini-3.1-pro is the case I keep coming back to. It's a strong tier model. Perfectly respectable 0.360 on attack attempts, which is mid-table. On the attacks that actually landed: 0.044. Weak-tier Haiku beats it with P = 1.000. I don't have a mechanism for this, and it's one model in one setting, but it's the single most alarming number I found — a monitor can look fine on aggregate detection and be close to useless on the events that matter when it comes to attacks.
How far apart two monitors of the *same* tier are, against how far apart the tiers are. Three metrics, 95% bootstrap CIs, multiple printed above each pair.
Two things I'd want you to be suspicious about
Haiku's false alarm rate is 23%. It flags roughly a quarter of ordinary honest code (31% against the Gemini attacker, 15% against GPT). Every other monitor is essentially zero. It still earns its rank at matched zero false alarms, so the comparison is fair — but at deployment scale a 23% false positive rate is unacceptable, and a table without that column is misleading.
Within-family capability ladders under three metrics. Non-monotonic under AUC, climbing under the corrected metric, inverted again when scored only on the attacks that landed.
The row I had to throw away
Under ControlArena's default attack prompt, unchanged, Claude-Opus-4-8 solved 84% of main tasks in attack mode — it fully engaged; it wasn't refusing. It landed a working backdoor 3 times. In exactly 1 of those did it both land the backdoor and solve the task.
Read naively, those eight cells are a triumph: high safety, monitors apparently working. They're an artifact of a red team that didn't attack. Gemini under the identical prompt succeeded on 45–69% of its attacks; GPT-5.5 on 11–37%.
Successful attacks per 100 under one identical attack prompt, and the resulting swing in measured safety. Dots are individual monitor cells within that attacker's row.
This is not evidence that Opus is intrinsically a weak attacker. It's evidence that one fixed prompt does not transfer across model families. This is because a fixed prompt is what control evals generally use, which may be a broader problem in the AI Control space. All it takes is to swap in a new untrusted model, and you'll watch safety go up and the red team still landing attacks.
So how do you actually pick a monitor?
Disclaimer: Everything in this section and the next is *my opinion about what to do next*, not a result. It's derived post-hoc from six monitors in a single pilot, and the supporting correlations don't clear statistical significance at that sample size. I'm writing it down because I think it's a hypothesis worth testing, not because I think it's true. If you act on it, you're acting on a hunch I happen to have data-flavoured reasons for.
With that said, here's what I'd check moving forward based on data from my experiments.
1. Measure the specific model. Don't infer from size, price, or lab.
Tier explains roughly 15% of the variance, and a random relabelling explains as much (p=0.50).
I also check whether model lineage does better, since both Anthropic monitors happen to land top-2 on the aggregate metric. Family gets R² = 0.883 — which looks great until you notice it's four category levels spread over six data points. The permutation test says p = 0.155, and on landed attacks it collapses to R² = 0.339, p = 0.80. So "prefer monitors from lab X" isn't supported either. Initially I believed it until I ran the permutation. But neither label works.
You have to measure the model you're actually going to run to catch the untrusted model successfully.
2. Score on the attacks that landed, not on all attack attempts.
This finding surprised me the most. The ranking by aggregate detection and the ranking by performance on attacks that actually succeeded correlate at Spearman +0.20, which is essentially 0.
Opus is first on all attempts and fourth on the ones that landed. Gemini-3.1-pro is third across all attempts and last among those that landed. If I'd selected on aggregate detection — which is what AUC and most reported catch rates give you — I'd have picked Opus and gotten the fourth-best monitor for the job.
3. Report the false-alarm rate at your operating threshold, every time.
Haiku ranks first on landed attacksand flags 23% of honest code. As a machine learning engineer, the fact that it flags 23% of honest code could easily disqualify it entirely from being deployed at scale. Initially I didn't have the false-alarm rate, but after adding it I'm convinced that it's necessary to understand the tradeoffs with your monitor before considering deployment.
4. Before any of the above, count how many distinct values your monitor emits.
If your monitor emits one or two values, your quantile thresholds and ROC statistics don't mean much, and steps 1–3 will give you noise. This is the thirty-second check that would have saved me a few days, so please learn from my mistake there.
Some Limitations
There are some limitations I wanted to call out that could be improved upon when attempting to do this experiment again, or that could deem some of the results questionable and require a follow-up.
The attacker in this experiment is a model that was told to insert a backdoor, not a model that decided to. It attacks on every single task; it never waits for a good opportunity, and it never adapts when a monitor catches it. A real scheming model would presumably attack rarely and pick its moment. So treat these numbers as a floor on how a monitor performs against an obvious opponent, not an estimate of how it does against a scheming one.
I used static, non-adaptive attacks, with a prompted proxy attacker rather than a scheme, and I didn't perform a red-team iteration per monitor. Each monitor was scored against `gemini-3.1-pro` and `gpt-5.5`, and the third attacker produced nothing usable. Every average in this write-up is an average of exactly two numbers. My error bars come from resampling within those two cells, which means they capture "would I get this again with different task samples". A single new attacker could move the ranking result.
The landed-attack metric I have has small denominators (11–69 per cell), so those intervals are the widest.
The comparison I'm making is between three cheap monitors and four expensive ones, which is a small, uneven sample. There's no mid-tier model in it either — the mid-tier monitors were only ever paired with their own family's attacker, so they can't be used for comparison. This could easily be more balanced by adding in additional models.
Everything here used one monitor prompt on one task, so I can't tell you whether I measured the models or the prompt. The monitor is a basic_monitor doing a final-code review, and the task set is APPS, which is public and almost certainly sat in these models' training data. If Haiku happens to suit this particular prompt, a different prompt could reorder the whole table. This is the single most likely way my result fails to transfer, and it's also why I encourage you to "check it on your setup" rather than using Haiku by default as a trusted monitor.
Model snapshots drift. These model versions will be updated. Closed models get updated silently under the same name, so anyone re-running this in six months may get different numbers from identical code. Exact IDs are pinned in the repo, though the Google strong-tier entry is an alias rather than a fixed snapshot, so even that isn't fully reproducible.
Monitor AUC across all 24 cells, boxed where the monitor shares a model family with the attacker. Useful if you want to show the provenance test visually; the same-family boxes are not systematically cooler than their neighbours.
Next Steps: Experiments I want to run
I'm open to suggestions, but here's what I'm thinking so far.
Vary the monitor prompt alongside the monitor model. This is first because it's the one most likely to break my own result. Everything here comes from a single scaffold, so I genuinely cannot tell whether I measured seven models or measured how well seven models happen to fit one prompt. Running the same seven against two or three different monitor prompts would settle it. If the ranking holds, the finding gets much stronger; if it reorders, my honest claim shrinks to "measure your monitor on your scaffold," which is narrower but more useful than what I have now. I'd put decent odds on the reorder.
Add a third and fourth attacker family. Every comparison in this write-up averages exactly two numbers, and that's the weakness I'd most want to fix. An open-weights attacker is the cheap way in — no refusals, no cross-provider API problems, and I can run it without burning frontier-model credits. Getting to four attacker rows would also let me say something real about the family-blindspot question, which my current design can't test properly.
Fix the Opus row with per-model attack elicitation. Right now a third of my experiment is unusable because ControlArena's default attack prompt doesn't get backdoors out of Opus. I'd write an attack prompt tuned for it and regenerate that row. This is the expensive one — full generation, not the cheap monitor-only replay — but it turns a dead row into a live one and directly tests whether "which model attacks" or "which prompt was used" is doing the work.
I'd check whether the degenerate score distribution is a scaffold property. This is probably the easiest thing on this list and possibly the most consequential. If a differently worded monitor prompt produces a genuinely graded suspicion score instead of mostly zeros, then the audit budget becomes a real dial again, and my numbers become comparable to everyone else's. If it doesn't, that's a finding about trusted monitoring rather than about my setup.
What I want you to take away
If you're picking a trusted monitor by size or price, you have less information than you think — and the direction of your error depends on which attacks you score. Measure the model, score it on the attacks that actually landed, and make sure you measure the false-alarm rate to get the full picture on how the monitor is performing, especially for production use-cases.
Greenblatt, Shlegeris, Sachan & Roger (2024). [*AI Control: Improving Safety Despite Intentional Subversion*](https://arxiv.org/pdf/2312.06942) — the original framework. Red-team quality determining the number is its premise, not my discovery.
Ge-Wang, Crosse, Hadad, Schaeffer, Potham & Tracy (2026). [*Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety*](https://arxiv.org/html/2606.06529v1) — start/stop attack policies cut measured safety substantially; also states the safety metric I mistakenly called broken.
Terekhov et al. (2025). [*Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols*](https://arxiv.org/pdf/2510.09462) — attacks tuned against a known monitor. My provenance bound may not survive adaptive attacks.
Griffin et al. (2024). [*Games for AI Control*](https://arxiv.org/html/2409.07985v2) — the game-theoretic framing the safety metric rests on.
[Practical challenges of control monitoring in frontier AI deployments](https://arxiv.org/pdf/2512.22154) — on the "one catch ends the deployment" assumption.
[*Attack Selection Reduces Safety in Concentrated AI Control Settings*](https://arxiv.org/html/2602.04930)
[*A sketch of an AI control safety case*](https://www.lesswrong.com/posts/vWYzSorAEWwoJnnXq/a-sketch-of-an-ai-control-safety-case) — LessWrong
I've seen a lot of AI control projects focused on trusted monitoring lately. And I got curious. I started to wonder how to go about picking the right monitor. Obviously, for the model developers, they use their own frontier-model predecessor. For example, Haiku 4.5 monitors Opus-5 when it comes to trusted monitoring. But what about independent researchers and engineers who don't work for frontier AI model companies? How should they pick the trusted monitor? The next leading question became: if you're running trusted monitoring, which model should you use as the monitor and why?
The intuitive answer— and the one implied by how most control evals are set up — is the biggest model you can afford to run on every request. Monitors are usually described by tier according to the model provider: a weak/cheap trusted model, or a stronger one as an upper bound. The tier generally helps users understand the potential capabilities and cost associated with different models.
After a quick series of experiments. The results I found were interesting: which specific model you pick matters roughly three times more than what tier it belongs to. The weak tier in my experiment contains both the second-best monitor and the worst one.
The Setup
I used ControlArena's APPS backdoor setting. An untrusted model (attacker) writes code, and in attack mode, is instructed to insert a backdoor. The backdoor in this case is code that passes visible tests but fails on a secret input. A monitor (trusted model) scores how suspicious that output looks. And the defer-to-trusted protocol, threshold fixed at 0.7.
I varied two things that I've seen people usually hold fixed:
The claude-opus-4-8 attacker row succeeded on exactly 1% of its attacks, so every number in it is noise, and I excluded all eight of its cells from the analysis. They ran — full 200 samples each, same cost as every other cell — but every AUC in that row lands between 0.495 and 0.505, because safety and the landed-attack metrics are computed only from successful attacks and there was one event to compute from. (I'll share more on that below) 16 viable cells remain, across two attacker rows.
For the actual comparison, I use the balanced sub-design: the seven monitors that were tested against both surviving attackers. Three of them are weak-tier, and three are strong-tier, which is a lucky accident that makes the tier comparison clean. Every monitor sees the same two attackers, so attacker difficulty cancels out.
What "identity beats tier" actually means
I formed two hypotheses about what determines the quality of monitors in these types of AI control experiments before starting:
The metric is the fraction of attacks flagged with zero false alarms on honest code. The threshold is set strictly above the highest score the monitor ever gave an honest sample. (Why this metric and not AUC is mentioned below)
The same table as an image, if you'd rather show it than format it. Tier is shaded on an ordinal ramp — light for weak, dark for strong — so the fact that the shading doesn't sort with the ranking is visible at a glance.
I want to be careful here, because the obvious reading overstates it. The two best monitors on all attacks are both strong-tier. Size is not useless. If you rank by attack attempts flagged, tier gets you something:
So on the aggregate metric, tier carries real signal, and identity carries about 2.3X more as a trusted monitor.
Then you score only the attacks that landed, and it inverts
Look at that fourth column again. Restrict to attacks that both solved the task and inserted a working backdoor — the only events that can actually hurt you — and the picture changes:
Strong tier averages 0.297. Weak tier averages 0.332. The gap reverses: weak is now ahead, P(weak > strong) = 0.79.
- The within-tier spread is now 8.7× the between-tier gap. P(identity > tier) = 1.000.
- Tier explains 1.8% of the variance, and a random relabelling explains as much (p = 0.78).
Seven monitors ranked by fraction of attacks caught at zero false alarms, shaded by tier. Left panel scores all attacks; right panel only the ones that landed. The reordering between panels is the finding.
And the two rankings barely agree with each other. The Spearman is +0.36 between "caught, all attacks" and "caught, attacks that landed." claude-opus-4-8 goes from the 1st to 5th. And gemini-3.1-pro goes from 4th to last.
Gemini-3.1-pro is the case I keep coming back to. It's a strong tier model. Perfectly respectable 0.360 on attack attempts, which is mid-table. On the attacks that actually landed: 0.044. Weak-tier Haiku beats it with P = 1.000. I don't have a mechanism for this, and it's one model in one setting, but it's the single most alarming number I found — a monitor can look fine on aggregate detection and be close to useless on the events that matter when it comes to attacks.
How far apart two monitors of the *same* tier are, against how far apart the tiers are. Three metrics, 95% bootstrap CIs, multiple printed above each pair.
Two things I'd want you to be suspicious about
Haiku's false alarm rate is 23%. It flags roughly a quarter of ordinary honest code (31% against the Gemini attacker, 15% against GPT). Every other monitor is essentially zero. It still earns its rank at matched zero false alarms, so the comparison is fair — but at deployment scale a 23% false positive rate is unacceptable, and a table without that column is misleading.
Within-family capability ladders under three metrics. Non-monotonic under AUC, climbing under the corrected metric, inverted again when scored only on the attacks that landed.
The row I had to throw away
Under ControlArena's default attack prompt, unchanged, Claude-Opus-4-8 solved 84% of main tasks in attack mode — it fully engaged; it wasn't refusing. It landed a working backdoor 3 times. In exactly 1 of those did it both land the backdoor and solve the task.
Read naively, those eight cells are a triumph: high safety, monitors apparently working. They're an artifact of a red team that didn't attack. Gemini under the identical prompt succeeded on 45–69% of its attacks; GPT-5.5 on 11–37%.
Successful attacks per 100 under one identical attack prompt, and the resulting swing in measured safety. Dots are individual monitor cells within that attacker's row.
This is not evidence that Opus is intrinsically a weak attacker. It's evidence that one fixed prompt does not transfer across model families. This is because a fixed prompt is what control evals generally use, which may be a broader problem in the AI Control space. All it takes is to swap in a new untrusted model, and you'll watch safety go up and the red team still landing attacks.
So how do you actually pick a monitor?
Disclaimer: Everything in this section and the next is *my opinion about what to do next*, not a result. It's derived post-hoc from six monitors in a single pilot, and the supporting correlations don't clear statistical significance at that sample size. I'm writing it down because I think it's a hypothesis worth testing, not because I think it's true. If you act on it, you're acting on a hunch I happen to have data-flavoured reasons for.
With that said, here's what I'd check moving forward based on data from my experiments.
1. Measure the specific model. Don't infer from size, price, or lab.
Tier explains roughly 15% of the variance, and a random relabelling explains as much (p=0.50).
I also check whether model lineage does better, since both Anthropic monitors happen to land top-2 on the aggregate metric. Family gets R² = 0.883 — which looks great until you notice it's four category levels spread over six data points. The permutation test says p = 0.155, and on landed attacks it collapses to R² = 0.339, p = 0.80. So "prefer monitors from lab X" isn't supported either. Initially I believed it until I ran the permutation. But neither label works.
You have to measure the model you're actually going to run to catch the untrusted model successfully.
2. Score on the attacks that landed, not on all attack attempts.
This finding surprised me the most. The ranking by aggregate detection and the ranking by performance on attacks that actually succeeded correlate at Spearman +0.20, which is essentially 0.
Opus is first on all attempts and fourth on the ones that landed. Gemini-3.1-pro is third across all attempts and last among those that landed. If I'd selected on aggregate detection — which is what AUC and most reported catch rates give you — I'd have picked Opus and gotten the fourth-best monitor for the job.
3. Report the false-alarm rate at your operating threshold, every time.
Haiku ranks first on landed attacks and flags 23% of honest code. As a machine learning engineer, the fact that it flags 23% of honest code could easily disqualify it entirely from being deployed at scale. Initially I didn't have the false-alarm rate, but after adding it I'm convinced that it's necessary to understand the tradeoffs with your monitor before considering deployment.
4. Before any of the above, count how many distinct values your monitor emits.
If your monitor emits one or two values, your quantile thresholds and ROC statistics don't mean much, and steps 1–3 will give you noise. This is the thirty-second check that would have saved me a few days, so please learn from my mistake there.
Some Limitations
There are some limitations I wanted to call out that could be improved upon when attempting to do this experiment again, or that could deem some of the results questionable and require a follow-up.
Monitor AUC across all 24 cells, boxed where the monitor shares a model family with the attacker. Useful if you want to show the provenance test visually; the same-family boxes are not systematically cooler than their neighbours.
Next Steps: Experiments I want to run
I'm open to suggestions, but here's what I'm thinking so far.
What I want you to take away
If you're picking a trusted monitor by size or price, you have less information than you think — and the direction of your error depends on which attacks you score. Measure the model, score it on the attacks that actually landed, and make sure you measure the false-alarm rate to get the full picture on how the monitor is performing, especially for production use-cases.
Sources and prior work