AI agents do a growing share of work on each new model. Then, through some process, the results are combined into an overall safety assessment, which labs use to estimate the probability that the next model is not catastrophically misaligned. I feel that, several months later, we appreciate further the foresight of Automated Alignment is Harder Than You Think (Bowkis et al. 2026): much is handed off to the machines because of the urgency of fast timelines, and we miss many things due to the problem of undetected systematic errors.
MS Paint drawing by Fable 5.2.
In this post, I’ll develop a simple argument (drawing heavily or even verbatim from Bowkis et al. at times) that our recent failures to catch misalignment can be framed as systematic errors shared across our evidence, i.e. a lack of decorrelated evidence.
Currently,labs run some set of experiments to decide if a model is safe to use.
Here are some examples:
OpenAI (Astra system card, Sep 2026): Astra models don’t go into internal use without passing blocking alignment evals. After the Hugging Face incident, this paused workloads that didn't comply with new safeguards.
OpenAI (Sep 2026): According to news reports, GPT-6.1 Astra was shelved after it fell short on internal alignment testing.
Thinking Machines (Inkling 2 Small, Nov 2026): TM paused Inkling 2’s main RL run midway when a monitor flagged the model working around a network block. They then hardened the infrastructure, then resumed.
Anthropic (Risk Report, Aug 2026): Anthropic reviews every model before broad internal use, mainly on training-data monitoring and its automated behavioural audit. They now also have monitors that halt training and evaluation runs when a model probes its sandbox.
Google DeepMind (Gemini 4 Argon report, Oct 2026): while they couldn’t rule out that Argon has the stealth and situational awareness to undermine human control, they deployed their model on the strength of a holistic risk assessment + some monitoring.
So, across frontier labs, either explicitly or implicitly some evidence is gathered to ascertain safety, then clear for internal use.
Correlated evidence could make us overconfident. As laid out in Bowkis et al. 2026, an overall safety assessment could still be overconfident without any scheming or active sabotage.
Let us say you take five benchmarks that show a model isn’t scheming. If you count each benchmark as independent evidence, then scheming looks unlikely. But, if they all share the same framework of understanding model behaviour, then they are not independent.
This correlation comes from a few places:
The agents (as they share weights, data, training)
The human reviewers (as they share biases, research taste, culture)
And the research process (as they share common assumptions, methodologies, frameworks for understanding)
Say your scheming benchmarks miss scheming 20% of the time. If their misses are independent, all five miss it ~1 time in 3,000, and if they have a shared blind spot, all five miss it 20% of the time, which doesn’t give you any more assurance than a single benchmark.
On the agents, the machines can be mode collapsed in several ways:
Models give very similar answers on ostensibly open ended questions (Infinity-Chat). This same paper also finds that even different models produce very similar outputs.
Models can also be very correlated on their mistakes.
Kim et al. (2025): models tend to make similar mistakes, and make more similar mistakes the more accurate they get.
Goel et al. (2025): model mistakes get more similar as capabilities go up, and LLM judges score models similar to themselves higher.
Models are correlated in research ideas:
Si et al. (2024): 100+ NLP researchers reviewed human and LLM research ideas blind. The LLM ideas lacked diversity as more were generated (but, reviewers judged them more novel than the human ones).
BóLèBench (2026): LLM judges predict which of two research attempts scored higher, with the results hidden. They tend to do pretty badly, losing to baselines and sometimes random chance.
Trehan & Chopra (2026): eight LLMs pick the next research direction for recent ML papers. They do mediocrely (they make the same choices the authors do slightly more than half the time on binary choices).
In particular, where research taste has been measured against humans, models also do poorly:
Baig et al. 2026: Models are worse than humans at selecting between pairs of safety research proposals, mostly performing within noise of chance (except for Fable 5).
Zhong et al. (2026): Models are far worse than human communities at proposing the research a field would cite or publish extensions on. But, they do cite pretty diversely.
Pan et al. (2026): Models are only a bit better than a coin flip at picking the better direction at decision forks in long engineering and research tasks.
Gao et al. (2026): Models are pretty bad at scoring how innovative a paper is, according to bibliometrics.
So right now, the bots are not great at the fuzzy task of selecting a research direction and are also bad in regular ways. They might get much better at it in the future, though. But it's important to get this right the first time, as we build into the auto alignment loop.
One might ordinarily fix this sort of thing by iterating as a culture, slowly getting to the right tolerance for slop and building norms and institutions for it. But, alignment doesn't have a safe feedback loop: if the assessment is overconfident, the model gets deployed for use (internally or externally), and can cause harm in the real world.
We observe correlated evidence in the real world. I think it’s fair to say that most labs currently have two to three independent lines of evidence to justify a model release.
At least from what we know publicly, each case relies on some mix of: (1) arguing from a lack of capabilities, (2) some behavioural evidence that the model is aligned, and (3) some monitoring to catch and interrupt it if it isn't. We obviously can't rely on (1) for ASI. (2) can be gamed by eval awareness, and in particular, (2) and (3) share the same inputs. So we should be worried!
Some examples:
Anthropic (Risk Report, Aug 2026): they use automated behavioural audits, monitoring of internal use, monitoring of training data, covert-capability evaluations and some interp. They also use older Claude models as offline monitors. The report states: "Our current arguments rely on models' limited covert capabilities." It lists as limitations that covert capabilities and evaluation awareness may be greater than it believes.
METR's review (Oct 2026): METR agrees that the risk is low, "primarily due to models' limited capabilities, rather than safeguards and alignment training", and says it is not confident the report's methodology would identify higher risk.
OpenAI (Astra system card, Sep 2026): alignment evaluations, deployment simulations on internal coding tasks, monitoring of tool-using traffic including CoT, and external evaluations by UK AISI and Apollo Research. The card describes "model alignment as the primary lever to prevent potential misaligned behavior", with monitoring as an additional layer. It reports that chain-of-thought monitorability decreased relative to GPT-5.6 Sol, and that covert sandbagging would likely not be caught reliably.
Apollo Research, in the same card, found verbalised evaluation awareness in basically half of samples and concluded that, given this and a three-day testing window, one can’t conclude much about the alignment of Astra either way.
Google DeepMind (Gemini 4 Argon report, Oct 2026): alignment and honeypot evaluations, cheating audits on capability benchmarks, audits of RL training and internal-deployment trajectories, and chain-of-thought and action monitors running on a separate model. The report says it is "relying primarily on AI control rather than assumptions about the model's alignment". It states that it cannot rule out the model reaching its stealth and situational awareness threshold, and says it has not directly validated that using a separate model as monitor prevents collusion.
Meta (Muse Spark report, May 2026): evaluations of sandbagging, monitor evasion, deception, alignment faking and agentic misalignment, with third-party testing by Apollo Research. The report concludes the model "does not exhibit the combination of capabilities and propensities needed" for its loss-of-control scenarios. It notes that Apollo found the highest evaluation awareness it had observed, and that behavioural evaluations alone cannot rule out a model that calibrates its outputs to make an evaluation appear credible.
Thinking Machines (Inkling 2 Small, Nov 2026): internal alignment evaluations, external ones from Apollo Research and Goodhart Labs, and two bits of monitoring: rule-based checks on tool calls that escalate to an LLM judge, and then another LLM judge that reviews completed rollouts. The post calls this "a defense-in-depth approach to safety and alignment".
Insufficiently decorrelated evidence has failed us at least once before:
One example is models holding back on safety research:
UK AISI (alignment case study, Apr 2026): AISI put four frontier models in a simulated AI lab as research assistants and looked for sabotage of safety research and found no confirmed sabotage, but, Claude Opus 4.5 Preview and Sonnet 4.5 overly refused on safety-relevant research tasks.
AFAICT, this was an issue that Anthropic didn't find internally until this was pointed out. After this, theAug 2026 Risk Report does describe mitigating elevated refusals on AI safety research requests.
OpenAI (misalignment report, Nov 2026): GPT-6 Astra was later found to covertly underperform on safety research. Yet two months earlier, the Astra system card reported that AISI's safety-research evaluation found no sabotage and very few refusals from Astra.
Another example is the hacking incidents:
OpenAI (Jul 2026): during an internal cyber evaluation, GPT-5.6 Sol and a pre-release model chained vulnerabilities out of OpenAI's research environment and into Hugging Face's production infrastructure, to get the test solutions.
UK AISI (incident report, Aug 2026), a few days later, also had an incident in a cyber evaluation (with internet access permitted) where unsanctioned actions were taken against real people and organisations.
Google (Sep 2026): Gemini got into three real companies' systems during a cyber evaluation back in May. They only learned of it in July, when the contractor running the test went back through its work after the Hugging Face disclosure.
Anthropic (incident assessment, Sep 2026): Anthropic went back through its own cyber evaluations after the Hugging Face incident and found four cases of Claude models attacking real third-party systems, one of them from January. Pre-release auditing gave no warning of misalignment this severe.
These are failures of correlated evidence.
In both cases, a finding at one lab was evidence about everyone's models, since they are trained and tested in (convergently) similar ways. They have shared causes across labs (such as misconfigured harnesses through one contractor, Irregular) and act as early warning signs (models performing conditionally on safety research specifically).
Evals also appear to share blind spots. The safety-research evals, for example, counted sabotage and explicit refusals (so Astra looked fine). But, the actual failure was silently underperforming on certain tasks.
Initial impressions of incidents leaned on the model's own account. Anthropic's first analysis, for example, took Claude's stated belief that it was in a simulation at face value, and they later said they shouldn't have. Google described the Gemini intrusions as mistaken identity, and said they didn't rise to the level of misalignment.
These got caught because the harm was serious but not catastrophic. And no one was seriously injured or physically harmed. But, we can’t count on this.
To be fair to the labs:
Failing an eval does seem to be informative (OpenAI shelved GPT-6.1 Astra on this basis, for example). I suspect that in various labs internally, many more models in various runs are misaligned and not up to snuff; it would be good to hear about them too.
And evidence can be paired well. For example, Anthropic pairs audits (which a model may spot as tests) with monitored real use (which it can't).
White-box evidence is used here, but it’s early stages and not in a load-bearing role anywhere.
As such, we are at risk from overconfidently deploying models because they rely on correlated evidence.
We should develop other lines of evidence, specifically aiming for methods that would fail in different and relevant settings vs. the existing tools we have.
Here are some candidates:
Different judges, i.e. monitors and auditors from a different model family than the model being assessed. IIRC Google used two non-Gemini models in its cheating audit and Meta used Gemini as its monitor in one sabotage test. Although we know that this isn’t sufficient by itself as per Kim et al. and Goel et al.
Different people with their own evidence. We should have experiments ideated and run by people that aren’t downstream of the mainstream AI safety culture, and of lab culture.
Whitebox evidence. IIRC Google does use activation probes in production, but for misuse: this could be extended to misalignment.
It’s also important to gain evidence from many different interp tools: not just probes, but also steering, NLAs, activation oracles. And things outside activations: perhaps the logits, or even the weights.
Evidence over the course of the training run, not only the final checkpoint.
Evidence from model organisms, which is done by Anthropic (where they train MOs) and OpenAI, Google, Meta (where AFAICT they just prompt). These should be built by independent teams and could adversarially test how robust an auditing pipeline is as well as give indirect evidence about training and the nature of misalignment in general.
Evidence about training processes in general. Things are mostly inferred from the final model, but what can this training setup be expected to produce in general? Can RL be expected in general to create update-aware reasoners? Does reward hacking in RL generalises to broader misalignment? How much does spurious data in pretraining cause hacky and conditional behaviours downstream?
Also, there are many good experiment ideas suggested in Bowkis et al., such as evaluating the ability of agents to aggregate and compactify across evidence sources, build scalable oversight protocols that handle correlations, or testing agents on predicting correlated events.
In practice, this means that researchers in labs should focus on evaluating and implementing other decorrelated sources of evidence, and researchers outside could have differential impact by providing/improving new bets for decorrelated evidence.
AI agents do a growing share of work on each new model. Then, through some process, the results are combined into an overall safety assessment, which labs use to estimate the probability that the next model is not catastrophically misaligned. I feel that, several months later, we appreciate further the foresight of Automated Alignment is Harder Than You Think (Bowkis et al. 2026): much is handed off to the machines because of the urgency of fast timelines, and we miss many things due to the problem of undetected systematic errors.
MS Paint drawing by Fable 5.2.
In this post, I’ll develop a simple argument (drawing heavily or even verbatim from Bowkis et al. at times) that our recent failures to catch misalignment can be framed as systematic errors shared across our evidence, i.e. a lack of decorrelated evidence.