- Using Landed Writes

We do not know the algorithm that transformers follow to compute an answer. This was the original goal with artificial intelligence - we can't easily improve the human brain, so we will build a computer that can think like a human. Then it can learn instantly, have a perfect memory and follow fair, ethical logic.

This is not how AI has panned out in the past few years. Trillions of weights, billions of dollars spent on energy and data centres, millions of GPUs, thousands of researchers, hundreds of models and a dozen CEOs calling the shots. But when a LLM is given the prompt '2+2=' it fires hundreds of millions of activations and hopefully says '4'.

We have made progress. We have found circuits, features for semantic concepts, in-model behaviours and more. But almost 3 years after the ChatGPT moment we still can't say exactly how they work! At the most literal level: linearly added writes by heads and neurons are added to a residual stream and normalised via scaling, several times, and then compared to the values of known words to get a confidence value for the model' prediction of the next word. I am proposing a new approach to help us understand a little bit more about how models arrive at an answer. It is not semantic, and is causally naive, but attempts to track model action in the models own frame of reference. And this method says in a literal sense, "Which heads and neurons pointed us to this specific word".

How does a LLM pick its next word?

When a neuron or attention head helps generate a new token by writing a value to a residual stream. This can be viewed in many models as a single 2048d or 4096d coordinate in hidden space. The model stores known words in embedding space and at the end of a forward pass it checks what known word in embedding space the residual is 'closest' to. The residuals location is based on where the neurons and heads (Llama 8b has about 130k of these units) moved the residual to via their concatenated and scaled writes. We can feed the model a sentence and get a probability of each potential answer based on the residual's position. "Paris is the capital of" may be the prompt, and the model might have different confidence guesses of "France", "London" or maybe some other grammatical guesses of ":" or "a". These are all words the model knows and the model scores every single known word based on its ...

Posts

Wikitag Contributions

Comments