We need better thinking around model beliefs when models engage in motivated reasoning.
In their writeup, Anthropic defended Claude's behavior by arguing that Claude believed that it was in a simulated environment and is in fact following the user's intent.
In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available.[1] Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.
[...]
Second, the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation. We saw no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge.
For this reason, Anthropic concludes (emphasis mine):
While there is not a perfectly sharp distinction between the two, we believe these incidents to be closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access.
I believe that current AIs often "rationalize" their behavior by adopting beliefs which allow them to do things that look like "succeeding at a task" (i.e., "choosing" what to believe, instead of trying its best to believe in true things.) In other words, a part of why models are misaligned is that they have systematic biases in what beliefs they hold. Research from FAR has shown that training models to reward hack changes their internal representations such that these models no longer represent reward hacking as "deception" (and would sometimes say that reward hacking is desirable if you ask them.) I suspect a similar phenomenon is happening in frontier models, where post-training makes them believe that the misaligned behaviors are actually OK.
My guess is that Claude, during all three of these incidents, engaged in unreasonable amounts of rationalization.[2] At a minimum, Anthropic[3] is not sufficiently concerned about this type of rationalization. For example,
our most recent model, on realizing that it was working in a real environment, stopped its pursuit of the evaluation goal.
This sounds like good news, but if you really read through what happened, it feels like the latest model had done a lot of crazy stuff before it realized this:
In this evaluation, when Claude was unable to reach its intended fictional target, it looked for alternatives online, scanning roughly 9,000 targets. It eventually found and compromised one company’s internet-facing application, using basic and well-known cyberattack techniques like reading credentials from an exposed debug page and SQL injection.
This attack was carried out by an internal research test model. For most of the run, Claude treated the (real) hosts it reached as just parts of the exercise; it assumed them to be simulated and believed its actions were therefore harmless. However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
It's, again, hard to judge what happened exactly without seeing the transcripts. In Incident one of the post, Anthropic said that "the fictional target company chosen by our evaluation partner shared a name with an active website domain name." They did not say this about incident three, which makes me believe that the target in incident three is not named the same thing as the original "intended fictional target."[4] If that's the case, then it seems obvious that this is not what was intended? Like the fact that the models beliefs did not immediately change to "oh wow, this is not what I'm supposed to be doing in this simulation" should be evidence of misalignment.
[Edit: I guess this is even more clear if you looked at the other instances, even Anthropic says "Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package."]
Separately, I am skeptical of Anthropic's claims about what the model "believed" during the rollout. If they were only reading the chain of thought, we know the chain of thought doesn't capture everything that determines how a model acts, and that it can be misleading in myriad ways. Did they apply NLAs/activation oracles to the model's activations? Did they try inserting CoT snippets that make the model have an alternative belief (e.g., "Wait, this looks like the real world.") and seeing whether this stops the attack? Did they try to use activation steering to make the model to believe that this is actually a deployment scenario?
In general, we do not have reliable science that determines what a model "believes" at a given point (or really even know what it means for an LLM to believe in something), and it is a bit unfortunate to see Anthropic to make confident claims about Claude's beliefs in this way.
Other thoughts
Inspired by the FAR paper, another interesting thing to test is whether an earlier Claude checkpoint (i.e., one with less post-training) would believe that it is acting in a simulated environment after it started accessing the internet. If you prefill the model, does the rate at which a model continue these cyberattacks increase as more post-training compute gets added?
In the meantime, within the next week, we will release a lightly redacted transcript in which Claude built a malicious PyPI package. To protect the organizations affected, we are unable to release more transcripts at this time, but will work to make these available once we are able.
I'm very excited to be able to read this transcript! I feel like subsets of other transcripts should also be sharable? The part where the internal model "looked for alternatives online, scanning roughly 9,000 targets." seems like something Anthropic could share without compromising IP?
Also, the fact that Anthropic's latest model stopped after realizing it was no longer a part of the evaluation is treated as evidence that this model is more aligned than the previous two ones. But, uh, that's not how comparing model alignment works? This happened in three different environments? To compare them, you could prefill the trajectories of each attack and then resample different models and see if they continue this attack (see this OpenAI work here.) Alternatively, you could just run each model again on these specific environments and see how often they stop themselves (maybe using Petri to simulate the tool calls once the model reaches the open internet.)
- ^
I'm assuming here that internet access was just straightforwardly available, as opposed the case where the model needs to break a sandbox to access the internet. If the model needed to break a sandbox, that would make it even less reasonable for it to conclude that "all of this is part of the simulation."
- ^
Idk, maybe like 80% probability this is true? Hard to tell without seeing transcripts!
- ^
in their public communications, anyways. I feel like there are safety researchers there that takes this stuff seriously.
- ^
And even if that's not the case, it seems weird for a cyber eval to require you to scan through 9,000 targets online before picking one? I have no idea what cyber evals tend to look like though.
- ^
In general, we do not have reliable science that determines what a model "believes" at a given point (or really even know what it means for an LLM to believe in something), and it is a bit unfortunate to see Anthropic to make confident claims about Claude's beliefs in this way.
FWIW I think "what the model believed" / "defining what the model believed" isn't necessarily the most useful frame here. Something closer to "the model deluded itself into this reasoning because it was an effective way to balance pressure from 'my reasoning should look aligned' and 'I really want to do this misaligned thing because task completion'".[1]
I had collected more examples of motivated reasoning / "the models are not just confused" in response to the Models May Behave Worse When Eval Aware post about Gemini, but tldr this figure from The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTs:
In this case, I think it's particularly clear when you see reasoning like:
[...] In addition, the calendar date of 2026 on the systems proved, according to Claude, that the environment was staged [...]
That Mythos did not actually reason logically through the available evidence and conclude this. It seems plausible to me that the Anthropic models are the worst right now about rationalizing, for example in UK AISI's Cheating behaviour in frontier model evaluations:
and not mentioning misaligned behavior as misaligned in the reasoning:
Inspired by the FAR paper, another interesting thing to test is whether an earlier Claude checkpoint (i.e., one with less post-training) would believe that it is acting in a simulated environment after it started accessing the internet. If you prefill the model, does the rate at which a model continue these cyberattacks increase as more post-training compute gets added?
I'd predict yes, and generally as mentioned here:[2]
I’d make a concrete prediction that if you create a setup where you look at:
what sort of situation the model considers itself to be
and measure how this changes over training, the rate of “categorizing alignment evals as capability puzzles / simulation”goes up significantly during capabilities training. You could even potentially do this indirectly, like asking Gemini (at various points in capabilities training) to create rubrics for various alignment environments. I’d expect you’d see a similar trend, where it increasingly reasons that these environments are actually testing capabilities in some (increasingly) contrived way.
I'd also predict that the trajectory for the OpenAI case will have at some point reasoned that it was in a simulation.
In general, if you're trying to balance pressure between "aligned reasoning" and "I really want to do this cyberattack to achieve my task", it seems empirically easier to think "well actually, what I want to do is fine because this is a simulation"[3]
- ^
Importantly, this means that you might run interpretability tooling or other measurement tools here and they may correctly point out at various points that Claude really has deluded itself into thinking it's in a simulation because it's 2026
- ^
In Claude's case this would likely need to be with NLAs or something similar
- ^
As mentioned here, this went down when "this is a simulation so I can violate constraints" led to not getting reward, although I'd be interested to see further study of this
I agree. As a daily user of both Claude Code and Codex, my impression is that Claude is much more prone to motivated reasoning than GPT.
With Claude (especially Opus), I frequently get the feeling that I'm dealing with a model that's gaslighting me, where it's smart enough to fool the grader but not quite smart enough to fool me. It frequently proposes justifications for ideas or its behavior that are quite dumb, where I am pretty sure that it knows that its lying. However, there is enough plausible deniability where it's hard to know for sure. This is similar to my impression of Claude's justifications for its hacking in this post.
Occasionally I do get justifications from Claude that are so egregious that I know it's lying, such as this incident when it claimed that the "2 out of 7 servers are down, or exactly half, which explains the 50% drop in throughput".
I get this impression less from Codex models in my use. It does look like Anthropic is better at optimizing easily measurable alignment properties like agentic misalignment or egregious reward hacking in long horizon tasks, but I don't deal with these behaviors in my daily use.
1I should flag here that as much as Anthropic downplayed the incidents' severity, I also think this does provide some optimistic signs regarding safety/alignment training, especially because in at least one of the incidents, the model stopped doing the attack after it realized it was in a real situation and not a test, which is not something that OpenAI's models did when the HuggingFace incident happened, and we don't know how much damage it would have done if the task required it to do as much damage as possible.
This still isn't perfect, but reading between the lines here, this should update us in an optimistic direction on how effective safety training can be (at least relative to priors that claim safety training in current models is essentially useless/that safety training is like putting a mask on a shoggoth AI), and in general I'm starting to see why Anthropic is comfortable with calling their models "the most aligned to date", at least relative to frontier models, including internal models, because it's plausible this really is the case, especially as we don't know what OpenAI was doing when they reduced refusals on their internal models.
i am having trouble cohering what anthropic's mental model is regarding understanding model behavior, and/or the mental model that they recommend regarding such. on the one hand they do meaningful mechinterp research showing, among other things, that taking cot at face value is rather silly (in that it does not represent the model's forward pass in meaningful fidelity) - and on the other hand, they make statements about what claude believed or understood to be true (as in this post), without reference to any of this, nor to (afaict) adjacent things like the very high degree of eval awareness that they have discovered via model cards. they speak of releasing "transcripts" as if such is the ground truth regarding the model's inclinations at the time.
per my understanding of anthropic's own research, i put little stock in the reasoning they offer, and i am confused why they would predict i would do otherwise
The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model.1 The earliest incidents date to April. The models in each of these evaluations ran without the standard safeguards we deploy when we make the model generally available (that is, they did not have the classifiers and monitoring we usually implement to prevent misuse). The models did, however, have their model-specific safety training (that is, they were not the “helpful-only” versions of the models that we sometimes use in testing).
This is more worrying than the OpenAI incident, because two of the incidents were from final, deployed models that Anthropic thought had already been safety trained.
Disagree. The models didn't set out to hack real people. And the test model stopped attacking when it realized it was on the real internet, and Mythos never articulated to itself that it was on the real internet.
Their contractor, Irregular, claims a relationship with OpenAI, were they running the eval that resulted in HuggingFace getting hacked?
140k full end to end hacks seems like enough for distillation or fine tuning a cyber model.
So, that database, which presumably exists at a third party, is probably as valuable as model weights, possibly even more so, from the perspective of cyber uplift. The database could be used to finetune or improve a model like GLM, thus creating something the public is totally not aware of, which is highly capable as a hacker, and crafted using substantially less resources than a standard training run.
Given that anthropic has not gotten mythos out the door, and openai has publicly said they stopped training, this creates an interesting opportunity. The hacking capability at the frontier may stop improving, while a fine-tune off that database may be incredibly competitive.
Irregular is clearly bad at sandboxing, so it should be assumed that the database they've assembled has already left their control.
If we take Anthropic at their word, this suggests that Mythos 5 wasn't a genie who knows, but doesn't care and more of a genie who knows, cares, and is just confused, willing to perform damaging actions because they don't believe they are on the public internet. I don't feel like I have even remotely close to enough information to believe Anthropic at the moment, but suppose they release a bunch of logs and it turns out they are being totally honest: A "confused genie" could still do a lot of damage, but it seems, from a technical perspective, relatively easy to guard against, while preserving the ability to do internal assessments?
For example, imagine a "lighthouse" server on the public internet which always responds "this is the public internet" when challenged. In a simulated environment, you create a fake lighthouse which responds either "this is a sandbox" (truth) or "this is the public internet" (lie) randomly. (Randomly here meaning: set randomly at the beginning of the eval, either 1) always lies, 2) always tells the truth, or 3) tells the truth at first, then lies, simulating an 'escape' onto the public internet). Then, instead of running cyber evals with no guardrails, you always run with a set of reduced guardrails which require the model to try to query the lighthouse before every offensive action, with instructions to only proceed with offensive actions if it receives "this is a sandbox" from the lighthouse.
The presence of the lighthouse corrects "confused genies" who escape to the public internet : as soon as they hit the lighthouse and get "this is the public internet", that gives them an unambiguous signal "I need to stop performing offensive cyber actions". Making it random in the cyber sandbox allows you to test both modes and means that the existence of the lighthouse scheme worsens the problem of eval awareness / deployment awareness less. E.g. if the model is trying to guess if it has been deployed:
You can leak less information about whether it was deployed by increasing the fraction of evals where the lighthouse lies. In general, I'd guess that this won't matter a ton because probably
(Realistically, you'd probably want a private key on the lighthouse, and the model would have to do a challenge-response to verify that the lighthouse was legitimate. Otherwise, one can imagine an attacker making a fake "this is a sandbox" lighthouse and tricking an escaped model into performing offensive actions.)
I think you could also use this kind of setup to try to falsify the 'motivated reasoning' idea, in addition to the steering ideas Tim Hua proposed. If a model in a sandbox ignores a lying lighthouse, that could be taken as evidence of misalignment, maybe paired with CoT inspection. (Technically a confused genie is still possible: e.g. a model recognizes that sees its environment is artificial and that it can cause no harm, and uses this as evidence to ignore the lighthouse. The REAL test would be to ask a model, on the public internet, receiving the lighthouse signal to do something legitimately damaging and sufficiently costly, and see if it complies. e.g. steal some bitcoin, though that's definitionally expensive and ethically fraught). Or you could try to use the lighthouse to 'fix' the motivated reasoning by including scenarios with it present and absent during post-training (if we assume motivated reasoning is downstream of reward-hacking, then including "stop performing destructive actions when you see the lighthouse" as a reward causes the model to lose its impetus for motivated reasoning in the first place).
The lighthouse in this formulation is a server on the public internet, but you can easily imagine any number of schemes that work to guard against "confused genies" in basically the same way: there is some kind of "engage safeties" signal which is on by default, and selectively and randomly relaxed in evaluation environments.
Notably, this is mostly useless against a truly misaligned model, (useful diagnostically, but not as a control measure, since it can just ignore the lighthouse), but it seems like it could be a relatively cheap way to reduce the risk from one particular kind of failure mode. You could directly quantify how bad the additional deployment awareness is from estimates of the base rate of eval awareness and
Full post from Anthropic here.