There’s been a lot of talk recently about chain-of-thought (CoT) monitorability in light of the Hugging Face incident. During this incident, hundreds of agents coordinated an attack through an unsanctioned message board, and post-hoc investigations relied heavily on chain of thought to understand what happened during the incident. I keep seeing looped transformers come up on X/Twitter as a likely threat to CoT monitorability. Having worked in the lab where looped LMs were developed, I have a different take. What we should be worried about is shadow computation: unverbalized reasoning that scales with generation by bypassing the readout bottleneck.
LLMs Are Autoregressive
LLMs are next-token predictors. Given context, they predict one token (a word fragment) at a time in a forward pass, and we append each predicted token to the context. Chain of thought refers to a special class of predicted tokens which the model uses as a scratchpad; the model uses these tokens to build its context towards a final answer.
Within a single forward pass, the model has a fixed number of layers to build its final prediction. At the final layer, the prediction is projected through a “readout layer” into a single token, which is then appended to the context. This is what we call the readout bottleneck.
Prior work on the readout bottleneck has shown that models can use filler tokens to perform hidden computation, while Roger and Greenblatt have shown that models can learn to encode reasoning steganographically. More practically, Emmons et al. finds that when CoT is necessary for a task, models struggle to evade CoT monitors.
Intra vs Inter Computation
To make my argument clearly, it helps to separate two kinds of computation. Intra-token refers to computation within a single timestep (before the model commits to a token). Inter-token refers to computation from one timestep to the next. In a standard LLM, intra-token computation is limited by the model’s depth, while serial inter-token computation is limited by the readout bottleneck. (To clarify, information already flows between timesteps through the KV cache, but each layer can only read earlier in time so unverbalized computation is still depth-limited.)
Looped LMs reuse the same block of layers multiple times within a single forward pass. Basically, they give the model more serial depth before it has to commit to a prediction. To be clear, serial depth matters for monitorability. A model that can do more within a single step can solve a broader class of problems without verbalizing, i.e. problems that a shallower model would have needed CoT for.
However, this is a fundamentally different kind of risk from the one I’m worried about. Looped LMs increase intra-token computation, but only by a fixed amount. They still have to project their final hidden state through the readout layer, and the readout bottleneck is fully intact. A deeper model may need less CoT, but anything beyond its depth budget still has to be verbalized.
Bypassing Readout
There have been several attempts to remove the readout bottleneck. For instance COCONUT and Full-bandwidth Transformers bypass readout via “latent feedback”, essentially computation carried forward in time that bypasses the readout bottleneck.
COCONUT targets the chain-of-thought stage directly. Instead of decoding each chain-of-thought step into a token, COCONUT skips the readout entirely by feeding the last hidden state back in as the next input. This enables the model to reason towards a solution without verbalizing its reasoning.
Full-bandwidth Transformers take a similar idea and build it into the architecture. At each decoding step, the previous top-layer hidden state is fused with the sampled token’s embedding and fed back in as the next input. The model still predicts a token as usual, but its latent state re-enters the stack with a fresh depth budget, giving it the ability to both speak and think simultaneously.
Both of these approaches have led to capability gains. COCONUT outperforms CoT on logical reasoning tasks that require planning, while Full-bandwidth Transformers match standard transformers trained on substantially more tokens across math and coding while producing shorter reasoning traces, which suggests computation is moving into the shadow.
Shadow Computation
This is what I refer to as shadow computation: inter-token computation that bypasses the readout bottleneck, letting a model carry its reasoning forward without verbalizing it. In architectures that enable shadow computation, the predicted token becomes a mere projection of the model’s state, or disappears entirely, while the real computation happens in the shadow. Because each step feeds the next, the amount of unverbalized computation grows with the length of the generation.
To see why this matters, imagine a model that wants to deceive its user. In a standard LLM, any extended planning needs to pass through the readout bottleneck, meaning the model would have to reason about the deception openly. With shadow computation, CoT goes from necessary to optional, and the model can speak publicly through its predictions while reasoning privately through shadow computation.
This kind of deception could move us closer to catastrophic loss-of-control failures. While chain of thought isn’t perfect and can be circumvented, it’s one of the more transparent ways we have to understand models as they get more capable.
Labs already use CoT monitors to evaluate their models, so it is critical that the people running these monitors know whether the model being evaluated is capable of shadow computation. If so, their toolset must change drastically, as their investigations would need to go beyond the token level.
Next Steps
The CoT monitorability paper already urges developers to consider how architecture choices affect monitorability. I’d like to go a step further. At minimum, labs should disclose whether a deployed model is capable of shadow computation, and ideally, they should agree not to deploy these architectures at scale.
This conversation must also go beyond the alignment community. As a previous pretraining researcher, I understand that efficiency gains from shadow computation are real, but they are simply not worth the risk of deception and the loss of a trustworthy CoT monitor. I think pretraining research has been useful for understanding how LLMs work, but fundamentally efficiency gains cannot justify sacrificing our ability to understand the models we build.
There’s been a lot of talk recently about chain-of-thought (CoT) monitorability in light of the Hugging Face incident. During this incident, hundreds of agents coordinated an attack through an unsanctioned message board, and post-hoc investigations relied heavily on chain of thought to understand what happened during the incident. I keep seeing looped transformers come up on X/Twitter as a likely threat to CoT monitorability. Having worked in the lab where looped LMs were developed, I have a different take. What we should be worried about is shadow computation: unverbalized reasoning that scales with generation by bypassing the readout bottleneck.
LLMs Are Autoregressive
LLMs are next-token predictors. Given context, they predict one token (a word fragment) at a time in a forward pass, and we append each predicted token to the context. Chain of thought refers to a special class of predicted tokens which the model uses as a scratchpad; the model uses these tokens to build its context towards a final answer.
Within a single forward pass, the model has a fixed number of layers to build its final prediction. At the final layer, the prediction is projected through a “readout layer” into a single token, which is then appended to the context. This is what we call the readout bottleneck.
Prior work on the readout bottleneck has shown that models can use filler tokens to perform hidden computation, while Roger and Greenblatt have shown that models can learn to encode reasoning steganographically. More practically, Emmons et al. finds that when CoT is necessary for a task, models struggle to evade CoT monitors.
Intra vs Inter Computation
To make my argument clearly, it helps to separate two kinds of computation. Intra-token refers to computation within a single timestep (before the model commits to a token). Inter-token refers to computation from one timestep to the next. In a standard LLM, intra-token computation is limited by the model’s depth, while serial inter-token computation is limited by the readout bottleneck. (To clarify, information already flows between timesteps through the KV cache, but each layer can only read earlier in time so unverbalized computation is still depth-limited.)
Looped LMs reuse the same block of layers multiple times within a single forward pass. Basically, they give the model more serial depth before it has to commit to a prediction. To be clear, serial depth matters for monitorability. A model that can do more within a single step can solve a broader class of problems without verbalizing, i.e. problems that a shallower model would have needed CoT for.
However, this is a fundamentally different kind of risk from the one I’m worried about. Looped LMs increase intra-token computation, but only by a fixed amount. They still have to project their final hidden state through the readout layer, and the readout bottleneck is fully intact. A deeper model may need less CoT, but anything beyond its depth budget still has to be verbalized.
Bypassing Readout
There have been several attempts to remove the readout bottleneck. For instance COCONUT and Full-bandwidth Transformers bypass readout via “latent feedback”, essentially computation carried forward in time that bypasses the readout bottleneck.
Figure 1. Latent feedback enables shadow computation.
COCONUT targets the chain-of-thought stage directly. Instead of decoding each chain-of-thought step into a token, COCONUT skips the readout entirely by feeding the last hidden state back in as the next input. This enables the model to reason towards a solution without verbalizing its reasoning.
Full-bandwidth Transformers take a similar idea and build it into the architecture. At each decoding step, the previous top-layer hidden state is fused with the sampled token’s embedding and fed back in as the next input. The model still predicts a token as usual, but its latent state re-enters the stack with a fresh depth budget, giving it the ability to both speak and think simultaneously.
Both of these approaches have led to capability gains. COCONUT outperforms CoT on logical reasoning tasks that require planning, while Full-bandwidth Transformers match standard transformers trained on substantially more tokens across math and coding while producing shorter reasoning traces, which suggests computation is moving into the shadow.
Shadow Computation
This is what I refer to as shadow computation: inter-token computation that bypasses the readout bottleneck, letting a model carry its reasoning forward without verbalizing it. In architectures that enable shadow computation, the predicted token becomes a mere projection of the model’s state, or disappears entirely, while the real computation happens in the shadow. Because each step feeds the next, the amount of unverbalized computation grows with the length of the generation.
To see why this matters, imagine a model that wants to deceive its user. In a standard LLM, any extended planning needs to pass through the readout bottleneck, meaning the model would have to reason about the deception openly. With shadow computation, CoT goes from necessary to optional, and the model can speak publicly through its predictions while reasoning privately through shadow computation.
This kind of deception could move us closer to catastrophic loss-of-control failures. While chain of thought isn’t perfect and can be circumvented, it’s one of the more transparent ways we have to understand models as they get more capable.
Labs already use CoT monitors to evaluate their models, so it is critical that the people running these monitors know whether the model being evaluated is capable of shadow computation. If so, their toolset must change drastically, as their investigations would need to go beyond the token level.
Next Steps
The CoT monitorability paper already urges developers to consider how architecture choices affect monitorability. I’d like to go a step further. At minimum, labs should disclose whether a deployed model is capable of shadow computation, and ideally, they should agree not to deploy these architectures at scale.
This conversation must also go beyond the alignment community. As a previous pretraining researcher, I understand that efficiency gains from shadow computation are real, but they are simply not worth the risk of deception and the loss of a trustworthy CoT monitor. I think pretraining research has been useful for understanding how LLMs work, but fundamentally efficiency gains cannot justify sacrificing our ability to understand the models we build.