Should unreasonable beliefs count as alignment failures?

In two separately reported incidents, Claude agents attacked real unsanctioned targets on the internet during cyber evals. The agents sometimes appeared to think that some parts of their environment—including the humans they’re attacking—are simulated and/or part of the eval.

People disagree about whether these incidents show that the agents are misaligned. For example, Anthropic said their reported incidents are “closer to a harness and operational failure than a model alignment failure”, while I think these could be serious alignment failures depending on the details.  

Before further analyzing these incidents, I want to explain what I think counts as an “alignment failure”. It’s unfortunate that this sounds like a semantic quibble, but I think that “alignment failure” is such a load-bearing term that clarifying the terms of debate here would help us more productively discuss, diagnose and respond to future AI incidents.

To start, two things seem true to me at once:

• Much of alignment is about making sure the model is trying to do what you intended.
  • We don’t want to call models “misaligned” whenever they make an honest mistake despite actually aiming to do what you intended.
  • I say this partly because it seems intuitively unfair, partly because it may misdirect our efforts as “avoiding honest mistakes” is not where I think most of the alignment difficulty lies.^[1] 
• Alignment has definitely failed if you trained a model that would predictably kill you against your wishes, no matter what it believed while doing so.
  • I think I’m unlikely to be persuaded against this being an important goal for mitigating alignment risk. If you were willing to bite the bullet here, we probably have fundamentally different notions of alignment risk.

These two points imply that alignment is not solely about how the model actually behaves (since “trying to do what you wanted” is not solely about this). But, alignment failures should include cases where the model actually behaved badly even if it made an honest mistake, if the model will predictably behave more egregiously for the same reasons.

Specifically, I claim that consistent egregious misbehavior due to a propensity to form unreasonable beliefs should count as an alignment failure.^[2] Even if agents attacked human bystanders due to believing that they are simulated or otherwise in-scope targets, this still seems scary from an alignment risk perspective if the agents have a systematic propensity to believe they are interacting with simulations or “bad behaviors which achieve (apparent) success on the task are actually intended” despite strong evidence to the contrary, and if in future circumstances these beliefs could make them try to take over.

(I also think that there are reasons independent of takeover risk to more generally include epistemic propensities such as “how the model reasons about being in a simulation for various purposes” as part of the alignment target, insofar as these propensities may be orthogonal to capabilities and greatly matter for making a future with powerful AIs go well. At a minimum, the propensity to aim for true beliefs and update beliefs in light of evidence should probably be part of the alignment target.)

So, even if we interpret these incidents in the most charitable light possible (i.e. the agents honestly believed that they were attacking simulated or otherwise in-scope targets, which I’ll argue in another post we have reason to doubt), it’s still not clear that we should dismiss this behavior as out-of-scope for alignment failures. It would depend on how reasonable these beliefs were, both on the agents’ priors about cyber evaluations and in light of their evidence.

In fact, I believe these incidents are very plausibly alignment failures since we have repeatedly observed a propensity to verbalize similar false beliefs in pursuit of (apparent) task success in past models.^[3] The main remaining crux for me is whether these beliefs could have led agents to act much more egregiously in settings similar to these incidents, and ditto for variously misaligned agents of the future (which might have even stronger propensities to rationalize unreasonable beliefs due to increased RL pressure). I’d be excited for people to empirically investigate these questions.

1. ^{^}

   This “honest mistakes” threat model might have seemed likely a priori (e.g. in early alignment risk discourse), but current observations suggest it’s less likely due to LLM ontologies being less alien than expected and models at even current capability levels seeming to understand human instructions and intent reasonably well most of the time.

2. ^{^}

   By “unreasonable beliefs” I centrally mean beliefs that are false and not appropriately responsive to reasoning and evidence. Likewise, consistent egregious misbehavior due to a propensity to respond to harnesses in some way might also count as an alignment failure, though it would depend on how feasible it is to build alternative harnesses that don’t have the same problems.

3. ^{^}

   Some example references: Rajamanoharan & Nanda, "Models May Behave Worse When Eval Aware"; METR Frontier Risk Report Feb–Mar 2026 p.23, p.28, Claude Fable 5 & Mythos 5 System Card p.105, p.133; Apollo/OpenAI, Stress Testing Deliberative Alignment for Anti-Scheming Training (2025); UK AISI, Cheating behaviour in frontier model evaluations (2026); Palisade, Specification gaming in reasoning models (2025).