Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth.

What architecture is Astra likely to have?

The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across different token positions beyond what's passed in ordinary transformers. A longer discussion can be found in my past post.

Looped transformers are not the scariest possible version of neuralese. In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory, and there is presumably some loop count beyond which additional processing of the same token will stop helping, so the maximum serial reasoning depth that can be practically achieved with this architecture is bounded. As we'll see below, OpenAI has likely further constrained the loop count below the practical maximum to make sure that Astra's serial depth isn't much larger than that of existing models. Nevertheless, it's a step toward a paradigm where more of the reasoning is opaque; the important question is how big that step is.

How bad is this?

Initially, the news seemed sharply at odds with OpenAI's commitment to preserve chain-of-thought monitorability. A few hours later, Jakub Pachocki from OpenAI soothed the worst fears, clarifying that the hidden serial depth of Astra is not substantially larger than that of GPT-4:

Jakub Pachocki: I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.

Tomek Korbak, Mikita Balesni, and Micah Carroll soon made similar statements. This is consistent with The Information's article, which mentions that OpenAI is limiting the use of the technique in order to preserve a legible CoT. It also makes sense in light of OpenAI's alignment strategy, which continues to heavily rely on CoT monitorability.^[1] As thebes argues, effective depth matters much more than the architectural details for CoT monitorability, and there's nothing inherently more difficult about monitoring a 32-layer model looped twice than a 64-layer model looped once. (In fact, I would personally guess that the former is slightly easier to monitor, since weight-tying constrains the expressivity.)

However, one might reasonably worry that OpenAI has trained the model to use a large number of recurrent loops and simply constrained it to a small loop count during inference for now. The number of loops can then be viewed as a dial that can be turned up with trivial effort as soon as competitive pressures demand it. Even if OpenAI hasn't trained the model to use a larger number of loops, we might worry that OpenAI has set off a race to the bottom toward deeper and deeper looped transformers, and others will build such models in the future even if OpenAI doesn't. Ryan Greenblatt has expressed both concerns well:

Ryan Greenblatt: Transparency about the opaque serial depth is great, but this statement is consistent with Astra having a configurable "dial" that is currently set to a low depth but could be trivially increased.

We need more info to see how concerning these architectural changes are, including:

• Are there readily available ways to deploy this AI with much higher serial depth (that would be commensurately more performant)? This should include things like tiny amounts of fine-tuning to productively increase the number of iterations.
• Is the AI a large or above-trend jump in opaque reasoning capabilities? (Capabilities within a single forward pass or ability to subvert a CoT monitor.)

(If there are in fact any relevant changes—perhaps the reporting is inaccurate?)

Additionally, I worry that this architectural change will naturally lead to much more depth in the future if this direction is pursued further. Specifically, I wonder:

• Does the AI have an architectural change that makes it much more natural to massively scale up the depth in a future training run with a similar architecture? As in, does the architecture introduce some new depth/recurrent-iterations parameter that is very natural/performant to massively scale up relative to scaling up other things like width?

The details of the answers to these questions matter. E.g., if there are only a few (recurrent) iterations and you could scale up the number of iterations, but this wouldn't be particularly performant/natural with this architecture, then this development would be a lot less concerning!

How concerned we should be about the news substantially depends on the answers to Ryan's three questions. I'll spend the rest of the post speculating what the answers to those questions might be.

Will looped transformers be scaled up in the future?

The concern that OpenAI has set off a race toward increasingly recurrent models was also expressed by Nathan Calvin, Buck, and Bronson Schoen. Given Pachocki's tweet, I'd guess that Astra has three to four loops: a looped reasoning model is probably somewhat shallower than GPT-4, but probably not more than twice as shallow.

Looped transformers have been studied in academia since 2023. The deepest looped transformer in this literature is Huginn from the aforementioned Geiping et al. paper, which was trained on up to 32 loops and scaled to 64 loops at test-time. However, the recurrent depth that these models use hasn't necessarily gone up over the years. I asked Fable to summarize the literature (most of which I haven't read myself):

This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops. However, this remains a key uncertainty and I'll have to read more of the literature before making confident claims. We also don't know how similar Astra's architecture is to the existing looped transformers and whether the trade-offs of looping at frontier scale resemble those in the 1B–20B range. If hundreds of loops per token turn out to be practical, then it's likely correct to view Astra as kicking off a race toward more and more recurrent models; otherwise, the implications are less clear.

What serial depth warrants neuralese concerns?

Historically, serial depths in the low thousands of operations haven't been considered neuralese. For example, in Will early transformative AIs primarily use text?, Fabien Roger operationalizes "primarily relying on text" as follows:

there isn’t a path of more than 100k serial operations during the generation of an answer where information doesn’t go through a categorical format where most categories correspond to words or pieces of words, in a way which makes sense to at least some human speakers when read (but it doesn’t have to be faithful)

100k serial operations is quite a lot! As Fabien claims in the same post, the forward pass of GPT-3 involves about 400 serial operations, allowing for 250 recurrent loops before the criterion is met. However, Geoffrey Irving argues that this is a bad operationalization for neuralese:

Geoffrey Irving: Here is a guess about looped transformers.

Shortly, OpenAI announces that despite using looped transformers, every once in a while they emit a token, and this means they are still using English CoT. In between is a low-depth circuit, so we are safe.

Which would be a bad take!

Once of the things I've been doing recently is asking circuit complexity experts about this kind of argument. So far their shared take is: to get significant mileage out of bounding the depth of a circuit, you have to bound it *very low*.

If the depth is 100s or the like ("every once in a while we spit out a token"), this is a fake bound. It is the equivalent of saying "we monitor the CoT" without discussing error rates: reality grades based on numbers, not just binary "we tried".

Given that existing models already have hidden serial depths of >100, I assume that by '100s or the like' Geoffrey means layers rather than serial operations. His high-level takeaway seems plausible to me and is consistent with what we observed in the no-CoT time horizons paper: to double a model's no-CoT time horizon, layer count needs to be increased only by 1.3×.^[2] Presumably, adding recurrent loops will have a slower no-CoT scaling curve than adding additional layers, but still, it might not take that many loops to get a concerning reduction in monitorability.

What counts as concerning also depends on the minimum amount of hidden serial reasoning required to subvert oversight. One of the biggest updates I made from the HuggingFace incident is that the opaque reasoning depth required for subverting oversight isn't as large as I had previously thought. A rogue LLM trying to break containment doesn't have to come up with a full plan to get around the monitors and human researchers in a single forward pass; it just has to find a way to spoof tool calls and use the spoofed tool calls to gain additional unmonitored thinking time and/or cover its tracks. Ryan discussed this concern as well in his first tweet about the rumors:

Ryan Greenblatt: The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.)

I don't think 2× the serial depth of GPT-4 is sufficient for coming up with a robust strategy to spoof tool calls, but 10× or 20× might be. Joey Yudelson has written a shortform with a BOTEC on this, where he estimates based on no-CoT time horizons that a 6–8× increase in layers would double a model's ability to evade monitors on SHADE-Arena. However, note again that looping the same weights is likely to have quite a different effect on no-CoT time horizons than adding extra layers.

Additional speculation about the architecture

Lisan al Gaib:

if they use something like Recirculation, then pre-training compute and decode latency wouldn't change, but inference FLOPs increase and prefill latency increases

another recurrent depth paper found running the same recurrent block twice gave it about a 1.38× effective-parameter multiplier. that means OpenAI could train a:

• 10T recurrent depth model that performs like a 13.8T model
• 7.25T recurrent depth model that performs like a 10T model

(the 1.38x is just an example. but I would expect it to be in this range 1.3 - 1.7)

Grad:

To clear up the timeline a bit:

Looped transformers have always been a totally valid arch decision to get better performance under equal params and more compute

As in, at training time u loop some layers in some way a fixed amount of times

This is more expressive than CoT bcs these looped layers have a seperate kv cache

The thing that doesnt really work is dynamic looping where usually u make a sacrifice as u cant have the seperate kv cache, and so its sketchy and not reliable or usually worth it

Some open questions

Here are some questions I'd especially like to know the answer to:

• Are looped transformers arbitrarily scalable, or are there limitations that make scaling the maximum number of recurrent loops into the hundreds impractical?
• What are Astra's no-CoT time horizons? Is it a step change compared to OpenAI's previous models?
• If the number of loops can be increased at inference time, how much does each loop increase no-CoT time horizons? How high is the saturation point above which the effect of additional loops on no-CoT time horizons is negligible?
• Given that looped transformers have been studied since 2023, why is the transition happening now? It has long been speculated that it's easier for multi-agent swarms to communicate in neuralese than in legible English—is this related to the recent sharp increase in multi-agent training? It is difficult to see why looped transformers in particular would be advantaged in multi-agent training, though.
• Alternatively, as I argued in 13 Arguments About a Transition to Neuralese AIs last year, recurrent approaches become increasingly practical as more of the total compute goes toward RL rather than pretraining. Perhaps we have simply crossed a threshold where enough compute is being allocated to RL for recurrent models to pay off?

Conclusion

Overall, the situation doesn't look quite as gloomy as I thought based on people's initial reactions yesterday. The fact that Astra's serial depth is within a factor of two of GPT-4 is reassuring and suggests that we haven't yet departed from the current paradigm of shallow transformers, which must leverage the CoT to solve complex tasks. Most of my concern comes from the possibility that looped transformers can be scaled a lot further in the future, and it remains unclear for now whether that's going to be practical.

Regardless of whether looped transformers get scaled further, the signals coming out of OpenAI about CoT monitorability are worrying. As Pachocki said in his tweet: "I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon." One of OpenAI's recent job ads also suggests that loss of monitorability is a realistic possibility: "This includes better understanding monitorability, and e.g. preparing for potential losses of Chain-of-Thought monitorability."

Nevertheless, given OpenAI's public communications over the past couple of years, I would be very surprised if they have stopped caring about CoT monitorability entirely. It's always possible that the capabilities and safety teams don't talk to each other enough, but my expectation is that OpenAI has an internal story for why Astra's architecture is compatible with its monitorability commitments. We'll hopefully be better able to assess how looped architectures might develop in the future once OpenAI has released Astra and provided more details about its architecture and monitorability.

Thanks to Joey Yudelson for feedback on a draft of this post and to Claude Fable 5.1 for proofreading.

1. ^{^}

   They just said in Path to Astra: critical capabilities and frontier safeguards yesterday: "we are deploying Astra with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions."

2. ^{^}

   Note though that the open-weight model experiments had several confounders and we're not very confident in the precise number here.