A week ago, OpenAI released their long-awaited blog post on the effect of model architecture and training pipeline on CoT monitorability. First off, I'm glad that they're sharing this information! That said, I dislike the way they frame their takeaways about how model architecture influences no-CoT capability. Here's how they describe the effect in three different places:
The fact that architecture has an influence is framed as a small caveat only observed in some evals, while the much more vocal takeaway is that no-CoT performance scales with base model perplexity. I don't think this is the right framing. Here is the figure they present to support these claims:

Presumably, Architecture 2 here is a looped transformer, while Architecture 1 and Architecture 3 are not. Grouping things together, Academic/Professional QA and World Knowledge both seem to be factual recall tasks where I wouldn't expect additional serial computation to help much. Same goes for language understanding. It's somewhat surprising that performance on Graduate-Level Science and Grade School Mathematics doesn't benefit much from looping, given that Astra's no-CoT math performance is clearly off-trend, but we don't know the composition of these benchmarks and it's plausible these also involve a substantial amount of factual recall.
But then, in Serial Multihop Computation and Planning Tasks, the architecture change does clearly improve the performance! Unfortunately, OpenAI doesn't provide a very clean comparison since the lowest perplexity achieved by Architecture 2 is lower than that of architectures 1 and 3, but it seems that the gap between architectures increases as perplexity gets lower.
And the two benchmarks where the architecture change has an effect are exactly the ones most relevant to scheming and covert reasoning capabilities! If a model shows a large jump in its ability to form competent plans without CoT, that's much scarier to me than a large jump in the number of facts it knows, although the latter can also matter for scheming. Thus, I think that the observation in the body of their post that "for multi-hop planning and serial computation questions, certain architectures perform better than base model perplexity predicts” should have been at least on par with “perplexity is predictive across architectures” as a top-level takeaway.
Mostly compatible, I think. My subjective probabilities for the dominant explanation of the apparent disagreement:
I agree with OP’s safety-weighting critique: planning matters more than counting benchmark wins, and “smooth” doesn’t mean safe. Also, conditioning on perplexity excludes architecture’s effects mediated through better perplexity—it doesn’t measure... (58 more words)