Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth.

What architecture is Astra likely to have?

The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across different token positions beyond what's passed in ordinary transformers. A longer discussion can be found in my past post.

Looped transformers are not the scariest possible version of neuralese. In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory, and there is presumably some loop count beyond which additional processing of the same token will stop helping, so the maximum serial reasoning depth that can be practically achieved with this architecture is bounded. As we'll see below, OpenAI has likely further constrained the loop count below the practical maximum to make sure that Astra's serial depth isn't much larger than that of existing models. Nevertheless, it's a step toward a paradigm where more of the reasoning is opaque; the important question is how big that step is.

How bad is this?

Initially, the news seemed sharply at odds with OpenAI's commitment to preserve chain-of-thought monitorability. A few hours later, Jakub Pachocki from OpenAI soothed the worst fears, clarifying that the hidden serial depth of Astra is not substantially larger than that of GPT-4:

Jakub Pachocki: I want to preven

...

ashim-mahara

ashim-mahara