Mis-Understandings's Shortform

Mis-Understandings

This is a special post for quick takes by Mis-Understandings. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I noticed a possible training artifact that might exist in LLMs, but am not sure what the testable prediction is. That is to say I would think that the lowest loss model for the training tasks will be doing things in the residual stream for the benefit of future tokens, not the collumn aligned token.

the residuals are translation invariant
The gradient is the gradient of the overall loss

3. therefore when taking the gradient of the attention heads, we also take the gradient of the residuals of past tokens of the total loss, not just the loss for the gradient of the loss for the activation column aligned for them

Thus we would expect to see some computation being donated to tokens further ahead in the residual stream (if it was efficient).

This explains why we see lookahead in autoregressive models

This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so.

from anthropics most recent release, mainly was the thought.

I was trying to fit that into how that behaviour shows up.

3Vladimir_Nesov2y

Hence "next token predictor" is a bit of a misnomer, as computation on any given token will also try to contribute to prediction of distant future tokens, not just the next one.

4nostalgebraist2y

If I understand what you're saying here, it's true but fairly well-known? See e.g. footnote 26 of the post "Simulators." My favorite way of looking at this is: The usual intuitive view of causal attention is that it's an operation that "looks back" at earlier positions. At each position i, it computes a "query" based on information from position i, and this query is used to search over "keys and values" computed at positions i-1, i-2, etc. (as well as i itself). OK, so at each position, attention computes a query. What makes a query "good"? Well, a good query is one that will "do something useful" in conjunction with keys and values computed at earlier positions. But attention is also computing keys and values at each position. What makes a key or value "good"? Precisely that it will "do something useful" in conjunction with the queries computed at later positions! The latter observation is just the flipside of the former. Queries at position i are encouraged to do useful lookback, on average over the "pasts" (i-1, ...) encountered in training; keys and values at position i are encouraged to be useful for the lookbacks performed by later queries, on average over the "futures" (i+1, ...) encountered in training. This is complicated slightly by the fact that causal attention lets positions attend to themselves, but it's easy to see that this is not a huge deal in practice. Consider that the keys and values computed at position i get used by... 1. ...the attention operation at position i, when it attends to itself (along with all earlier positions) 2. ...the attention operation at positions i+1, i+2, ..., when they "look back" to position i The K and V weights get gradients from all of these positions. So for a context window of size N, on average the gradient will be a sum over ~N/2 terms from future positions, plus just a single term from the current position. Since N >> 2 in practice, all else being equal we should expect this sum to be dominated