Currently, the public has very little insight about the robustness of alignment methods used by AGI companies. It’s possible that, when an AGI company notices some misaligned behavior (e.g. through an alignment evaluation, or through noticing real-world misbehavior), they conduct a principled investigation and address root causes of the misalignment. It’s also possible that they directly try to train away the examples of misalignment, which could result in brittle alignment that does not generalize out-of-distribution.
Without additional information from within AI companies about how they align their models (especially in response to egregious cases of misalignment, e.g. cybersecurity incidents), the public cannot adequately reason about risks from misalignment.
Public evidence is consistent with companies overfitting on alignment methods
The public evidence is consistent with multiple companies having no clear separation between the ways they notice misalignment and the environments they use to train misalignment away.
Companies have made very few statements about whether companies train directly on their evaluation distribution or real-world examples of misalignment. As far as I can tell, this is what’s publicly known:
OpenAI:
In the GPT-6 Astra System Card, they wrote that they built an eval in response to the Hugging Face incident but "did not construct honeypot training environments specifically to target this evaluation."
In their OpenAI-Hugging Face incident technical report, they wrote that they are expanding their alignment training and evals to encompass failure modes similar to those in the incident, but they didn’t comment on how they are separating them to avoid overfitting. They have also written about trying to preserve chain-of-thought monitorability and avoid accidental CoT grading.
METR’s Frontier Risk Report (February to March 2026): "OpenAI reported that it 'explicitly aim[s] to avoid deriving training data from flagged production deception cases to avoid overfitting to the monitor.'"
Google DeepMind:
In the Gemini 4 Argon Safety & Security Report, they wrote "Chain of thought monitoring is one of our most effective mechanisms for detecting misalignment. While signals from our monitors serve as a valuable flywheel for our hardening efforts, we are careful with their use elsewhere in the training pipeline. In particular, to preserve our chain of thought monitoring capability, we avoid training directly against our monitors, or using monitor outputs as a training signal or intervention."
METR’s Frontier Risk Report: "Google reported that it took significant precautions against training on or overfitting to misalignment detection methods (e.g., its anomaly detection classifiers). They reported that this included a structural precaution of siloing information between detection and mitigation teams."
Anthropic:
They have a number of claims across many risk reports and system cards. Especially notable is that in Alignment Risk Update: Claude Mythos Preview, they wrote "all our mitigations have a structure akin to the classic train / val / test split: our ‘outer layer’ of oversight (monitoring during training, auditing, and deployment monitoring) is held out as a ‘test set’, and isn’t trained against or iterated against during mitigation development. When an issue is identified (via auditing, usage monitoring, etc.), the observation is used to seed a ‘val set’: a simple, static evaluation that recreates the issue which we iterate mitigations against. Finally, the actual mitigation data is from a separate third distribution."
In the August 2026 Risk Report they modified that passage to be hedged: "our mitigations generally have a structure akin to..." and mitigation data "is usually from a separate distribution."
METR’s Frontier Risk Report: "An Anthropic employee we interviewed reported that Anthropic is 'conservative about training against reward hacks… [the] [m]ain way we’ve defended against reward hacks is strengthening environments.' She described being 'worried about [models learning to hide their reward hacks from monitors if you train against them], and want[ing] the model organism team to push hard on this.'"
Meta:
METR’s Frontier Risk Report: "Meta stated that it did not train on automated chain-of-thought monitors or white box probes. Meta also stated that it did not apply training-time interventions that would incentivize the model to learn evasion strategies, such as penalizing monitor-flagged behaviors in RL."
xAI: No public writing.
The information listed above is not verified by third parties, and thus provides limited evidence to the public.[1]
Anthropic, OpenAI, and GDM have all written about accidentally training on their CoT, which is some evidence that they're not great at sanitizing their training environments in general. Anthropic also wrote about how they accidentally trained on alignment-faking transcripts.[2] Given this track record, I wouldn't be surprised if multiple companies accidentally trained on alignment evals that were supposed to be held-out.
The lack of information is seriously hindering the public’s ability to estimate misalignment risk
Currently, when models exhibit some sort of misaligned behavior in one model generation but not the next, we do not know why this is the case.
It could be that companies took examples of misaligned behaviors and directly trained models not to repeat those behaviors. This would likely reduce the frequency of that type of misalignment in the near future, but would not clearly reduce long-term risk.
It could be that companies used a more principled approach where they kept their measures of alignment separate from the ways they align models. Then, improvements on measures of alignment would more closely track actual risk levels.
As such, the public is not in a position to properly reason about long-term misalignment risk given the current available evidence.
In order to allow the public to reason more clearly about misalignment risk, companies could:
Publish their approaches to aligning models and avoiding alignment overfitting. Combined with whistleblower protections and third-party reviews, information on these approaches would be very risk-relevant.
Conduct and publish research on whether their alignment techniques are generalizing out of their alignment training distribution. Companies should create new held-out alignment eval sets and measure whether their normal alignment techniques generalize out of distribution.
Involve third-party evaluators: Where companies are reluctant to talk publicly about their exact alignment methods, they could involve third parties in high-access alignment assessments that then report to the public whether alignment overfitting is a concern.
Even in METR’s Frontier Risk Report, METR is reporting what companies said in response to a questionnaire and in interviews, and never claims to verify those statements. They write that "In future iterations, we hope to [...] pilot ways to verify company-reported information."
August 2026 Risk Report, page 168: “The filters intended to remove these transcripts from training data were misconfigured, so they had not filtered transcripts for several model generations without anyone noticing. [...] [We] now suspect that all of our production models with a knowledge cutoff after December 2024 were trained on at least some of these transcripts.”
Currently, the public has very little insight about the robustness of alignment methods used by AGI companies. It’s possible that, when an AGI company notices some misaligned behavior (e.g. through an alignment evaluation, or through noticing real-world misbehavior), they conduct a principled investigation and address root causes of the misalignment. It’s also possible that they directly try to train away the examples of misalignment, which could result in brittle alignment that does not generalize out-of-distribution.
Without additional information from within AI companies about how they align their models (especially in response to egregious cases of misalignment, e.g. cybersecurity incidents), the public cannot adequately reason about risks from misalignment.
Public evidence is consistent with companies overfitting on alignment methods
The public evidence is consistent with multiple companies having no clear separation between the ways they notice misalignment and the environments they use to train misalignment away.
Companies have made very few statements about whether companies train directly on their evaluation distribution or real-world examples of misalignment. As far as I can tell, this is what’s publicly known:
The information listed above is not verified by third parties, and thus provides limited evidence to the public.[1]
Anthropic, OpenAI, and GDM have all written about accidentally training on their CoT, which is some evidence that they're not great at sanitizing their training environments in general. Anthropic also wrote about how they accidentally trained on alignment-faking transcripts.[2] Given this track record, I wouldn't be surprised if multiple companies accidentally trained on alignment evals that were supposed to be held-out.
The lack of information is seriously hindering the public’s ability to estimate misalignment risk
Currently, when models exhibit some sort of misaligned behavior in one model generation but not the next, we do not know why this is the case.
As such, the public is not in a position to properly reason about long-term misalignment risk given the current available evidence.
In order to allow the public to reason more clearly about misalignment risk, companies could:
Even in METR’s Frontier Risk Report, METR is reporting what companies said in response to a questionnaire and in interviews, and never claims to verify those statements. They write that "In future iterations, we hope to [...] pilot ways to verify company-reported information."
August 2026 Risk Report, page 168: “The filters intended to remove these transcripts from training data were misconfigured, so they had not filtered transcripts for several model generations without anyone noticing. [...] [We] now suspect that all of our production models with a knowledge cutoff after December 2024 were trained on at least some of these transcripts.”