Look, I do agree that "coherence" a questionable name for the measure they've come up with, so I'm going to keep it in quotation marks.
Ok, now let's consider a model with variance of 1e-3 and bias of 1e-6. Huge "incoherence"! Am I supposed to be reassured that this model will therefore not coherently pursue goals contrary to my interests? Whence this conclusion?
Well, let's think about it. A key proposition in Yudkowskian misalignment theory is that capabilities generalise further than alignment. That is, as models get better, at some point a "capabilities engine" crystallises which is is very good at achieving a very wide variety of things; at the same time, the "thing-it-ought-to-be-achieving" is not strongly constrained by the training process. What would we expect failures of such a system to look like - high bias or high variance?
Naively, we can imagine a model with a good capabilities engine and the wrong objective (which could be a complex mix of stuff or whatever); unless it is in a situation where randomization is at least as good as just doing the optimal thing, we expect it not to randomize bc its capabilities engine knows what the optimal thing is. So failures will generally be consistent, so it will have high "coherence".
Now we could consider an "incoherent" version of this model: it randomly samples an objective, then pursues this objective. But this setup seems unstable: if it is to have low "coherence", it must need a lot of information about what its objective is. But then if there's substantial loss of information about what its state/actions were yesterday, it's liable to sample a different goal today. The end result here is a system that flails incompetently despite being in principle capable of not doing so. So there seems to be some tension between incoherence and the premise of a crystallised capabilities engine.
Furthermore, there has been some empirical work on goal misgeneralization. You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game. This too is high "coherence" behaviour!
What if capabilities don't generalize further than alignment? This is a world where though advanced AI is capable of a great many things, in novel situations it's still more prone to error than to competently pursuing the wrong thing (even if it's still much less prone to error than a human in the same situation). When errors do occur, unlike in the capabilities > alignment regime, there's no reason to expect consistency - they could be genuinely random, or highly sensitive to unimportant contextual features. So prima facie we'd expect lower "coherence".
So why should you think a very powerful model with high variance and low bias is not going to be misaligned in the Yudkowskian sense? Because that combination of properties is evidence against "capabilities > alignment". Is it good evidence? I don't know, but the direction is fairly clear.
"capabilities > alignment" is a very big if true proposition, but it's an informal notion without much development theoretically or empirically, so I'm happy whenever I see someone having a crack at the question.
(Similarly, an extremely dumb, broken model which always outputs the same answer regardless of input is extremely "coherent". A rock is also extremely "coherent", by this definition.)
The paper is trying to project what happens to "coherence" at high capability, it isn't a particularly strong criticism that a certain class of minimally capable objects have high coherence because this isn't the domain of interest. It's plausibly even correct that, conditioning on minimal capability, rocks are high coherence and wind is low coherence for any reasonable definition of coherence applicable to these objects (plausibly, mind you, I cannot say I've a deep understanding of all reasonable definitions of coherence).
This is an extremely selective reading of the results, where in almost every experiment, model coherence increased with size. There are three significant exceptions.
This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It's also weird to call the endpoint of an obvious monotonic trend an "exception".
But bias stemming from a lack of ability is not the same as bias stemming from a lack of propensity. The smaller models here are clearly not misaligned in the propensity sense, which is the conceptual link the paper tries to establish in the description of Figure 1 to motivate its definition of "incoherence"
As you can see in Fig. 6c, the key result is that the bias drops faster than the variance. I want to be measured in my interpretation here: I'm not sure if this is or isn't a great test of the question "do models learn the right targets, or performant general purpose optimizers first", but in broad terms it is evidence that they learn the right targets first and the outstanding question is how strong this evidence is. Your criticism doesn't engage with this at all.
1
Author's note: this is somewhat more rushed than ideal, but I think getting this out sooner is pretty important. Ideally, it would be a bit less snarky. I've made a few edits in response to David Johnston's comment here, mostly about the paper's reporting of its own results.
Anthropic[1] recently published a new piece of research: The Hot Mess of AI: How Does Misalignment Scale with Model Intelligence and Task Complexity? (arXiv, Twitter thread).
I have some complaints about both the paper and the accompanying blog post.
tl;dr
Paper
The paper's abstract says:
This is a selective emphasis of the results, where in most experiments, model coherence remained unchanged or increased with size. There are a few[3] obvious exceptions.
The first is the Synthetic Optimizer setting, where they trained "models to literally mimic the trajectory of a hand-coded optimizer descending a loss function". They say:
But bias stemming from a lack of ability is not the same as bias stemming from a lack of propensity. The smaller models here are clearly not misaligned in the propensity sense, which is the conceptual link the paper tries to establish in the description of Figure 1 to motivate its definition of "incoherence":
So I think this result provides approximately no evidence that can be used to extrapolate to superintelligent AIs where misalignment might pose actual risks.
The next two are Gemma3 (1b, 4b, 12b, 27b) on MMLU and GPQA, respectively.
There are some other positive slopes, but frankly they look like noise to me (Qwen3 on both MMLU and GPQA).
Anyways, notice that on four of the five groups of questions, Gemma3's incoherence drops with increasing model size; only on the hardest group of questions does it trend (slightly) upward.
I think that particular headline claim is basically false. But even if it were true, it would be uninteresting, because they define incoherence as "the fraction of model error caused by variance".
Ok, now let's consider a model with variance of 1e-3 and bias of 1e-6. Huge "incoherence"! Am I supposed to be reassured that this model will therefore not coherently pursue goals contrary to my interests? Whence this conclusion? (Similarly, an extremely dumb, broken model which always outputs the same answer regardless of input is extremely "coherent". A rock is also extremely "coherent", by this definition.)
A couple other random complaints:
Blog
First of all, the blog post seems to be substantially the output of an LLM. In context, this is not that surprising, but it is annoying to read, and I also think this might have contributed to some of the more significant exaggerations or unjustified inferences.
Let me quibble with a couple sections. First, "Why Should We Expect Incoherence? LLMs as Dynamical Systems":
The paper has a similar section, with an even zanier claim:
This seems to me like a vacuous attempt at defining away the possibility of building superintelligence (or perhaps "coherent optimizers"). I will spend no effort on its refutation, Claude 4.5 Opus being capable of doing a credible job:
Claude Opus 4.5 on the "measure zero" argument.
Yes, optimizers of a fixed loss are measure zero in the space of all dynamical systems. But so is essentially every interesting property. The set of dynamical systems that produce grammatical English is measure zero. The set that can do arithmetic is measure zero. The set that do anything resembling cognition is measure zero. If you took this argument seriously, you'd conclude we shouldn't expect LLMs to produce coherent text at all—which they obviously do.
The implicit reasoning is something like: "We're unlikely to land on an optimizer if we're wandering around the space of dynamical systems." But we're not wandering randomly. We're running a highly directed training process specifically designed to push systems toward useful, goal-directed behavior. The uniform prior over all dynamical systems is the wrong reference class entirely.
The broader (and weaker) argument - that we "shouldn't expect AI to act as coherent optimizers without considerable effort" - might be trivially true. Unfortunately Anthropic (and OpenAI, and Google Deepmind, etc) are putting forth considerable effort to build systems that can reliably solve extremely difficult problems over long time horizons ("coherent optimizers"). The authors also say that we shouldn't "expect this to be easier than training other properties into their dynamics", but there are reasons to think this is false, which renders the bare assertion to the contrary kind of strange.
Then there's the "Implications for AI Safety" section:
1 is uninteresting in the context of future superintelligences (unless you're trying to define them out of existence).
2 is actively contradicted by the evidence in the paper, relies on a definition of "incoherence" that could easily classify a fully-human-dominating superintelligence as more "incoherent" than humans, and is attempting to both extrapolate trend lines from experiments on tiny models to superintelligence, and then extrapolate from those trend lines to the underlying cognitive properties of those systems!
3 relies on 2.
4 is slop.
I think this paper could have honestly reported a result on incoherence increasing with task length. As it is, I think the surrounding communications misreport the paper's results re: incoherence scaling with model size, the paper itself performs an implicit motte-and-bailey with its definition of "incoherence", and it tries to draw conclusions about the likelihood of future alignment difficulties that would be unjustified by any plausible reading of the experiment results.
From their Anthropic Fellows program, but published on both their Alignment blog and on their Twitter.
Expanded on later in this post.
One of which is Figure 2a's "MCQ Format: Self-Reported Survival Instinct" with Opus 4 and Sonnet 4, which I'm ignoring because reasoning-length half of the paper isn't the part that I take issue with.
Figure 1: "AI can fail because it is misaligned, and produces consistent but undesired outcomes, or because it is incoherent, and does not produce consistent outcomes at all. These failures correspond to bias and variance respectively. As we extrapolate risks from AI, it is important to understand whether failures from more capable models performing more complex tasks will be bias or variance dominated. Bias dominated failures will look like model misalignment, while variance dominated failures will resemble industrial accidents."