Thanks to Brendan Halstead for feedback on an early draft of this piece. Any mistakes here are my own.

[Epistemic status: I've looked at the relevant code enough to be moderately sure I understand what's going on. Predictions about the future, including about what facts will turn out to be relevant, are uncertain as always.]

Introduction

With the recent breakthroughs taking advantage of extensive Chain of Thought (CoT) reasoning in LLMs, there have been many attempts to modify the technique to be even more powerful. One of the natural ideas for improving CoT is to have LLMs perform CoT reasoning in the same latent space that they use for reasoning within a single forward pass, rather than being constrained to the space of possible tokens.

However, as people working on AI safety, it makes sense to ask how this changes the game for LLM interpretability. After all, we are able to catch a large fraction of current LLM deception by monitoring their natural-language CoT, since right now CoT is primarily faithful to the LLM's true reasoning and is legible to us given the right techniques. In order to understand this strategic situation, it's important to understand this new "language" (which people often refer to as Neuralese) that is created by reasoning in latent spaces instead of using tokens.

Understanding Neuralese

To refresh, a language transformer starts by embedding input tokens as vectors in some high-dimensional latent space, and runs each of these embeddings through a series of repeated computational layers. Then, of the resulting modified vectors in latent space, the vector that previously corresponded to the final input token is projected and normalized to create a probability distribution over what the next token could be. Then, to actually get the next token, you sample from the distribution.

Chain of Thought reasoning works so well because the model does some computation, outputs a token, and then all future instances of that model have access to that information as well. In essence, this technique for storing information between different forward passes greatly increases the serial depth of computation that is possible for the model. Because there is a computation in latent space corresponding to every input token, the computation also gets wider as the model reasons more, allowing for more parallelized reasoning^[1].

The recent Neuralese paper takes this p...

tatellos

tatellos