Most of the residual stream forgets within a token. A compact subspace doesn't.
Preface Fix some data corpus, then take the set of all information that persists across tokens in the residual stream. We can categorize it along two axes: (1) how quickly the content value changes token to token, from near constant to rapidly updating (e.g. document topic vs. current token part...
Jun 630