Thanks for your thoughts and concerns! We’ve updated the blog post to address some of them. The most important changes involve adding nuance to language and being explicit everywhere that our measure of incoherence refers to the relative contribution of variance to error, and is separate from how overall error changes with model capability, or how self-consistent successful trajectories are.
The blog post was put together much more hastily than the paper. Probably a lot more cumulative effort will go into reading the blog than the paper, so this is the opposite of how we should have distributed our effort. Reading and editing our post again, I agree parts of it were sloppy. Hopefully it is now at least improved (though certainly still not perfect!). This was my fault – I’m sorry. And I regret that we underutilized an opportunity to communicate these ideas to a larger audience.
That being said … I think your take on the paper is incorrect.
in most experiments, model coherence remained unchanged or increased with size.
This is actually quite difficult to judge, and depends on how you weigh different experiments. e.g. if you look at Figure 4, you will see that in two out of three experimental settings we see a robust positive correlation between measures of intelligence and error incoherence (panes b and c), and in only one configuration (pane a) do we see a mixed relationship.
My takeaway from the experiments across model scale is that the observed relationship between intelligence and incoherence in errors is complex, but that increasing model intelligence does not consistently reduce the incoherence of errors on fixed tasks. In contrast, the positive relationship between task length and error incoherence is extremely robust. So in net this suggests that as smarter models perform more complex tasks, their errors will be more incoherent.
But bias stemming from a lack of ability is not the same as bias stemming from a lack of propensity.
I’m confused by the intended meaning here. Propensity is a tendency to behave in a particular way. Lack of ability could be a cause of an observed propensity, but I don’t think that’s what you intend to say? (Do you mean to use a word like “intention” rather than “propensity”?)
I think this result provides approximately no evidence that can be used to extrapolate to superintelligent AIs where misalignment might pose actual risks.
The current AI revolution is driven by the observation that if you find scaling laws for losses, those scaling laws often extrapolate reliably from toy systems all the way up to intelligent models.
I expect empirical relationships between different error components to predict the types of error we see in superintelligent systems better than I expect thought experiments to generalize to the behavior of superintelligent systems.
I don’t expect to convince you of this stance. You should understand though that from my perspective, simple operationalizable definitions, and empirical scaling law-like measurements, are some of the strongest predictors we can build for how superintelligent systems will behave.
Ok, now let's consider a model with variance of 1e-3 and bias of 1e-6. Huge "incoherence"! Am I supposed to be reassured that this model will therefore not coherently pursue goals contrary to my interests?
This does seem like a much better position to be in. It would suggest that when the model pursues a goal it is not trained to pursue, that goal will not be self consistent across instances, or across time (see incoherence vs. sequence length). There are still ways this can go wrong … but at least the bad behavior is less likely to be coherent across time and space.
(unless of course the model was directly trained to pursue a goal contrary to your interests. See discussion of goal misspecification and reward hacking.)
an extremely dumb, broken model which always outputs the same answer regardless of input is extremely "coherent". A rock is also extremely "coherent", by this definition.
That is correct, about the dumb broken model. For the rock it’s undefined. You will need to define a training objective, and a way to compare outcomes produced by the rock to oracular outcomes under the training objective.
The paper basically assumes away the possibility of deceptive schemers
I don’t think this is correct, unless you mean something different than I understand by deceptive schemer. A deceptive schemer maps cleanly onto bias-dominated failures. It will consistently achieve a specific outcome, which is not the outcome we want it to achieve.
> The set of dynamical systems that act as optimizers of a fixed loss is measure zero in the space of all dynamical systems.
This seems to me like a vacuous attempt at defining away the possibility of building superintelligence (or perhaps "coherent optimizers"). I will spend no effort on its refutation, Claude 4.5 Opus being capable of doing a credible job:
Claude’s response here is correct, and is the point. Essentially every interesting property that you would like to imbue into a dynamical system is measure zero in the space of dynamical systems. You almost always have to work hard for all of them.
The broader (and weaker) argument - that we "shouldn't expect AI to act as coherent optimizers without considerable effort" - might be trivially true. Unfortunately Anthropic (and OpenAI, and Google Deepmind, etc) are putting forth considerable effort to build systems that can reliably solve extremely difficult problems over long time horizons ("coherent optimizers").
I think your argument is confusing overall error rate, and the contribution of variance to the error. The frontier labs are working very hard to reduce the overall error rate of their systems. They are not applying meaningful optimization pressure on the decomposition of those errors into bias or variance.
As models get smarter, we expect them to act more like optimizers targeting the outcomes they were trained to achieve. The question here isn’t whether the models will get better at optimizing a target. The question is, when they fail to optimize the right target, what will that failure look like?
The authors also say that we shouldn't "expect this to be easier than training other properties into their dynamics", but there are reasons to think this is false, which renders the bare assertion to the contrary kind of strange.
The article you link to doesn’t imply that models will become optimizers faster than they will settle on the right goal. It doesn’t speak to that question. It implies that as the model is trained to a loss of 0 (e.g. on some RL task), it will become a perfect optimizer. As the model is trained to a loss of 0 though, it will also come to perfectly align with the training goal.
The question we are probing is what the deviations from perfection look like. If we have an extremely powerful but still imperfect model, will its deviations from perfection tend to be dominated by it pursuing the wrong outcome or by it not acting in pursuit of a consistent outcome.
Author's note: this is somewhat more rushed than ideal, but I think getting this out sooner is pretty important. Ideally, it would be a bit less snarky. I've made a few edits in response to David Johnston's comment here, mostly about the paper's reporting of its own results.
Anthropic[1] recently published a new piece of research: The Hot Mess of AI: How Does Misalignment Scale with Model Intelligence and Task Complexity? (arXiv, Twitter thread).
I have some complaints about both the paper and the accompanying blog post.
tl;dr
Paper
The paper's abstract says:
This is a selective emphasis of the results, where in most experiments, model coherence remained unchanged or increased with size. There are a few[3] obvious exceptions.
The first is the Synthetic Optimizer setting, where they trained "models to literally mimic the trajectory of a hand-coded optimizer descending a loss function". They say:
But bias stemming from a lack of ability is not the same as bias stemming from a lack of propensity. The smaller models here are clearly not misaligned in the propensity sense, which is the conceptual link the paper tries to establish in the description of Figure 1 to motivate its definition of "incoherence":
So I think this result provides approximately no evidence that can be used to extrapolate to superintelligent AIs where misalignment might pose actual risks.
The next two are Gemma3 (1b, 4b, 12b, 27b) on MMLU and GPQA, respectively.
There are some other positive slopes, but frankly they look like noise to me (Qwen3 on both MMLU and GPQA).
Anyways, notice that on four of the five groups of questions, Gemma3's incoherence drops with increasing model size; only on the hardest group of questions does it trend (slightly) upward.
I think that particular headline claim is basically false. But even if it were true, it would be uninteresting, because they define incoherence as "the fraction of model error caused by variance".
Ok, now let's consider a model with variance of 1e-3 and bias of 1e-6. Huge "incoherence"! Am I supposed to be reassured that this model will therefore not coherently pursue goals contrary to my interests? Whence this conclusion? (Similarly, an extremely dumb, broken model which always outputs the same answer regardless of input is extremely "coherent". A rock is also extremely "coherent", by this definition.)
A couple other random complaints:
Blog
First of all, the blog post seems to be substantially the output of an LLM. In context, this is not that surprising, but it is annoying to read, and I also think this might have contributed to some of the more significant exaggerations or unjustified inferences.
Let me quibble with a couple sections. First, "Why Should We Expect Incoherence? LLMs as Dynamical Systems":
The paper has a similar section, with an even zanier claim:
This seems to me like a vacuous attempt at defining away the possibility of building superintelligence (or perhaps "coherent optimizers"). I will spend no effort on its refutation, Claude 4.5 Opus being capable of doing a credible job:
Claude Opus 4.5 on the "measure zero" argument.
Yes, optimizers of a fixed loss are measure zero in the space of all dynamical systems. But so is essentially every interesting property. The set of dynamical systems that produce grammatical English is measure zero. The set that can do arithmetic is measure zero. The set that do anything resembling cognition is measure zero. If you took this argument seriously, you'd conclude we shouldn't expect LLMs to produce coherent text at all—which they obviously do.
The implicit reasoning is something like: "We're unlikely to land on an optimizer if we're wandering around the space of dynamical systems." But we're not wandering randomly. We're running a highly directed training process specifically designed to push systems toward useful, goal-directed behavior. The uniform prior over all dynamical systems is the wrong reference class entirely.
The broader (and weaker) argument - that we "shouldn't expect AI to act as coherent optimizers without considerable effort" - might be trivially true. Unfortunately Anthropic (and OpenAI, and Google Deepmind, etc) are putting forth considerable effort to build systems that can reliably solve extremely difficult problems over long time horizons ("coherent optimizers"). The authors also say that we shouldn't "expect this to be easier than training other properties into their dynamics", but there are reasons to think this is false, which renders the bare assertion to the contrary kind of strange.
Then there's the "Implications for AI Safety" section:
1 is uninteresting in the context of future superintelligences (unless you're trying to define them out of existence).
2 is actively contradicted by the evidence in the paper, relies on a definition of "incoherence" that could easily classify a fully-human-dominating superintelligence as more "incoherent" than humans, and is attempting to both extrapolate trend lines from experiments on tiny models to superintelligence, and then extrapolate from those trend lines to the underlying cognitive properties of those systems!
3 relies on 2.
4 is slop.
I think this paper could have honestly reported a result on incoherence increasing with task length. As it is, I think the surrounding communications misreport the paper's results re: incoherence scaling with model size, the paper itself performs an implicit motte-and-bailey with its definition of "incoherence", and it tries to draw conclusions about the likelihood of future alignment difficulties that would be unjustified by any plausible reading of the experiment results.
From their Anthropic Fellows program, but published on both their Alignment blog and on their Twitter.
Expanded on later in this post.
One of which is Figure 2a's "MCQ Format: Self-Reported Survival Instinct" with Opus 4 and Sonnet 4, which I'm ignoring because reasoning-length half of the paper isn't the part that I take issue with.
Figure 1: "AI can fail because it is misaligned, and produces consistent but undesired outcomes, or because it is incoherent, and does not produce consistent outcomes at all. These failures correspond to bias and variance respectively. As we extrapolate risks from AI, it is important to understand whether failures from more capable models performing more complex tasks will be bias or variance dominated. Bias dominated failures will look like model misalignment, while variance dominated failures will resemble industrial accidents."