On NLAs and Neuralese/Recurrence

For some background, Anthropic recently published work on Natural Language Autoencoders (NLAs), a new interpretability method for understanding LLM activations. The idea is that given a hidden state which we would like to understand, we have

1. An Activation Verbalizer , which takes the hidden state as input and outputs a natural language explanation
2. An Activation Reconstructor , which takes this natural languge explanation as input and outputs a reconstruction

Like in other work on autoencoders, the goal is to train and to minimize the reconstruction loss over some dataset. To minimize this loss, a sufficiently good NLA will be forced to map all of the information in into natural language . A priori, there's not any reason to expect to be particularly interpretable. To ensure that it is, Anthropic employs various methods; for example, they initialize based on Opus 4.5 summaries of text, and have a KL divergence term in their loss function to encourage it to stay close to this initialization during training. Overall, the work has been positively received, and I think it is a notable advancement in interpretability research.

However, I think this work has some unfortunate capabilities externalities which I haven't seen discussed. In particular, I think that if NLA research goes well, we should expect it to be easier to be able to build models which use neuralese/recurrence. There are many ways this could happen, but a simple baseline is the following:

1. Train a CoT model using existing methods, and collect reasoning traces
2. For each reasoning trace, you can break down into steps and the model's final answer
3. Train a new model to predict the next encoding of these CoT steps conditional on the previous ones; that is, M predicts . Training can proceed in parallel using a transformer.
4. During inference, use to reason recurrently, generating hidden states rather than sampling tokens to simulate CoT. The final answer can be decoded using the Activation Verbalizer.

If your and models are sufficiently good^[1], this bypasses CoT reasoning entirely. It does inference more efficiently using hidden vectors^[2] instead of text, since it compresses many tokens of CoT into a single hidden vector, generated using a single forward pass of .

To be clear, I think that this exact idea doesn't work for current NLA models, for reasons I won't discuss. This is because there are many advantages to CoT, and I think we should try to preserve it for as long as possible! However, it does seem plausible to me that better methods encoding/decoding text to hidden vectors may be a helpful training signal for capabilities researchers, and that a future version may end up working. To the extent that they agree, I think researchers working on NLAs and similar methods should keep this in mind.

1. ^{^}

   At base NLA objective, ignoring the KL divergence term etc.

2. ^{^}

   One advantage over directly training for recurrence using something like COCONUT is that your NLA gives you more precise feedback on intermediate steps of reasoning.