Tracing the Thoughts of a Large Language Model
35nostalgebraist
28Julian Bradshaw
23Joseph Miller
5Mateusz Bagiński
1Alex Gibson
2Mateusz Bagiński
4Alex Gibson
3Ann
14Archimedes
8Adam Jermyn
11Daniel Kokotajlo
7Nicholas Goldowsky-Dill
10Viliam
6Ram Potham
4Ruby
3Kaj_Sotala
3Daniil Strizhov
2Fabien Roger
1Sylvain Ribes
1Annapurna
3Adam Jermyn
1Annapurna
1derek shiller
7Adam Jermyn
New Comment
Very impressive! At least on a first read, to me this felt closer than any past work to realizing the SAE dream of actually understanding the computations occurring in the model, as opposed to just describing various "cool-looking patterns" that can be extracted from activations.
I'm curious about what would happen if you studied cases similar to the examples you present, except that the recruitment of a particular capability (such as arithmetic or knowledge about a named entity) occurs through in-context learning.
For example, you discuss an "obscured arithmetic" task involving publication dates. In that case, the model seems to have learned in training that the correct prediction can be done by doing arithmetic. But we could imagine obscured arithmetic tasks that are novel to the model, in which the mapping between the text and a "latent arithmetic problem" has to be learned in-context[1].
We might then ask ourselves: how does the model's approach to these problems relate to its approach to problems which it "can immediately tell" are arithmetic problems?
A naively obvious "algorithm" would look like
- Try out various mappings between the observed text and (among other things) arithmetic problems
- Notice that one particular mapping to arithmetic always yields the right answer on previous example cases
- Based on the observation in (2), map the current example to arithmetic, solve the arithmetic problem, and map back to predict the answer
However, due to the feedforward and causal structure of transformer LMs, they can't re-use the same mechanism twice to "verify that arithmetic works" in 1+2 and then "do arithmetic" in 3.[2]
It's possible that LLMs actually solve cases like this in some qualitatively different way than the "algorithm" above, in which case it would be interesting to learn what that is[3].
Alternatively, if the model is doing something like this "algorithm," it must be recruiting multiple "copies" of the same capability, and we could study how many "copies" exist and to what extent they use identical albeit duplicated circuitry. (See fn2 of this comment for more)
It would be particularly interesting if feature circuit analysis could be used to make quantitative predictions about things like "the model can perform computations of depth D or lower when not obscured in a novel way, but it this depth lowers to some D' < D when it must identify the required computation through few-shot learning."
(A related line of investigation would be looking into how the model solves problems that are obscured by transformations like base64, where the model has learned the mapping in training, yet the mapping is sufficiently complicated that its capabilities typically degrade significantly relative to those it displays on "plaintext" problems.)
- ^
One could quantify the extent to which this is true by looking at how much the model benefits from examples. In an "ideal" case of this kind, the model would do very poorly when given no examples (equivalently, when predicting the first answer in a few-shot sequence), yet it would do perfectly when given many examples.
- ^
For instance, suppose that the current example maps to an addition problem where one operand has 9 in the ones place. So we might imagine that an "add _9" add function feature is involved in successfully computing the answer, here.
But for this feature to be active at all, the model needs to know (by this point in the list of layers) that it should do addition with such an operand in the first place. If it's figuring that out by trying mappings to arithmetic and noticing that they work, the implementations of arithmetic used to "try and verify" must appear in layers before the one in which the "add _9" feature under discussion occurs, since the final outputs of the entire "try and verify" process are responsible for activating that feature. And then we could ask: how does this earlier implementation of arithmetic work? And how many times does the model "re-implement" a capability across the layer list?
- ^
Perhaps it is something like "try-and-check many different possible approaches at every answer-to-example position, then use induction heads to move info about try-and-check outputs that matched the associated answer position to later positions, and finally use this info to amplify the output of the 'right' computation and suppress everything else."
2
1Copying over a comment from Chris Olah of Anthropic on Hacker News I thought was good: (along with parent comment)
fpgaminer
> This is powerful evidence that even though models are trained to output one word at a time
I find this oversimplification of LLMs to be frequently poisonous to discussions surrounding them. No user facing LLM today is trained on next token prediction.
olah3
Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy (https://colah.github.io/). I think this post and its children have some important questions about modern deep learning and how it relates to our present research, and wanted to take the opportunity to try and clarify a few things.
When people talk about models "just predicting the next word", this is a popularization of the fact that modern LLMs are "autoregressive" models. This actually has two components: an architectural component (the model generates words one at a time), and a loss component (it maximizes probability).As the parent says, modern LLMs are finetuned with a different loss function after pretraining. This means that in some strict sense they're no longer autoregressive models – but they do still generate text one word at a time. I think this really is the heart of the "just predicting the next word" critique.
This brings us to a debate which goes back many, many years: what does it mean to predict the next word? Many researchers, including myself, have believed that if you want to predict the next word really well, you need to do a lot more. (And with this paper, we're able to see this mechanistically!)
Here's an example, which we didn't put in the paper: How does Claude answer "What do you call someone who studies the stars?" with "An astronomer"? In order to predict "An" instead of “A”, you need to know that you're going to say something that starts with a vowel next. So you're incentivized to figure out one word ahead, and indeed, Claude realizes it's going to say astronomer and works backwards. This is a kind of very, very small scale planning – but you can see how even just a pure autoregressive model is incentivized to do it.
DeepMind says boo SAEs, now Anthropic says yay SAEs![1]
Reading this paper pushed me a fair amount in the yay direction. We may still be at the unsatisfying level where we can only say "this cluster of features seems to roughly correlate with this type of thing" and "the interaction between this cluster and this cluster seems to mostly explain this loose group of behaviors". But it looks like we're actually pointing at real things in the model. And therefore we are beginning to be able to decompose the computation of LLMs in meaningful ways. The Addition Case Study is seriously cool and feels like a true insight into the model's internal algorithms.
Maybe we will further decompose these explanations until we can get down to satisfying low-level descriptions like "this mathematical object is computed by this function and is used in this algorithm". Even if we could still interpret circuits at this level of abstraction, humans probably couldn't hold in their heads all the relevant parts of a single forward pass at once. But AIs could or maybe that won't be required for useful applications.
The prominent error terms and simplifying assumptions are worrying, but maybe throwing enough compute and hill-climbing research at the problem will eventually shrink them to acceptable sizes. It's notable that this paper contains very few novel conceptual ideas and is mostly just a triumph of engineering schlep, massive compute and painstaking manual analysis.
- ^
This is obviously a straw man of both sides. They seem to be thinking about it from pretty different perspectives. DeepMind is roughly judging them by their immediate usefulness in applications, while Anthropic is looking at them as a stepping stone towards ambitious moonshot interp.
DeepMind says boo SAEs, now Anthropic says yay SAEs!
The most straightforward synthesis[1] of these two reports is that SAEs find some sensible decomposition of the model's internals into computational elements (concepts, features, etc.), which circuits then operate on. It's just that these computational elements don't align with human thinking as nicely as humans would like. E.g. SAE-based concept probes don't work well OOD because the models were not optimized to have concepts that generalize OOD. This is perfectly consistent with linear probes being able to detect the concept from model activations (the model retains enough information about the concept such as "harmful intent" for the probe to latch onto it, even if the concept itself (or rather, its OOD-generalizing version) is not priviledged in the model's ontology).
ETA: I think this would (weakly?) predict that SAE generalization failures should align with model performance dropping on some tasks. Or at least that the model would need to have some other features that get engaged OOD so that the performance doesn't drop? Investigating this is not my priority, but I'd be curious to know if something like this is the case.
- ^
not to say that I'm believing it's strongly; it's just a tentative/provisional synthesis/conclusion
But models are incentivized to have concepts that generalize OOD because models hardly ever see the same training data more than once.
Seeing some training data more than once would make the incentive to [have concepts that generalize OOD] weaker than if [they saw every possible training datapoint at most once], but this doesn't mean that the latter is an incentive towards concepts that generalize OOD.
Though admittedly, we are getting into the discussion of where to place the zero point of "null OOD generalization incentive".
Also, I haven't looked into it, but it's plausible to me that models actually do see some data more than once because there are a lot of duplicates on the internet. If your training data contains the entire English Wikipedia, nlab, and some math textbooks, then surely there's a lot of duplicated theorems and exercises (not necessarily word-by-word, but it doesn't have to be word-by-word).
But I realized there might be another flaw in my comment, so I'm going to add an ETA.
(If I'm misunderstanding you, feel free to elaborate, ofc.)
My model of why SAEs work well for the Anthropic analysis is that the concepts discussed are genuinely 'sparse' features. Like predicting 'Rabbit' on the next line is a discrete decision, and so is of the form SAEs model for. We expect these SAE features to generalize OOD, because the model probably genuinely has these sparse directions.
Whereas for 'contextual / vibes' based features, the ground truth is not a sparse sum of discrete features. It's a continuous summary of the text obtained by averaging representations over the sequence. In this case, SAEs exhibit feature splitting where they are able to model the continuous summary with sparser and sparser features by clustering texts from the dataset together in finer and finer divisions. This starts off canonical, but eventually the clusters you choose are not features of the model, but features of the dataset. And at this point the features are no longer robust OOD because they aren't genuine internal model features, they are tiny clusters that emerge from the interaction between the model and the dataset.
So in theory the model might have a direction corresponding to 'harmful intent', but the SAEs split the dataset into so many chunks that to recover 'harmful intent' you need to combine lots of SAE latents together. And the OOD behaviour arises from the SAE latents being unfaithful to the ground truth, not the model having poor OOD behaviour. Like SAE latents might be sufficiently fine that you can patch together chunks of the dataset to fit the train dataset, in a non-robust way.
As for concepts that generalize OOD - I suppose it depends what is meant by OOD? Like is looking at a dataset the model wasn't exposed to, but that it reasonably could have been, OOD? If so, the incentive for learning OOD robust concepts is that assuming that most text the model receives is novel, this text is OOD for the model, so if its concepts are only relevant to the text it has seen so far, it will perform poorly. You can also argue regularisation drives short description lengths, and thus generalising concepts. Whether a chunk of the training set is duplicates / similar is kind of irrelevant, because even if only 50% of text is novel, the novel text still provides the incentive for robust concepts.
Models do see data more than once. Experimental testing shows a certain amount of "hydration" (repeating data that is often duplicated in the training set) is beneficial to the resulting model; this has diminishing returns when it is enough to "overfit" some data point and memorize at the cost of validation, but generally, having a few more copies of something that has a lot of copies of it around actually helps out.
(Edit: So you can train a model on deduplicated data, but this will actually be worse than the alternative at generalizing.)
In the poetry case study, we had set out to show that the model didn't plan ahead, and found instead that it did.
I found it shocking they didn't think the model plans ahead. The poetry ability of LLMs since at least GPT2 is well beyond what feels possible without anticipating a rhyme by planning at least a handful of tokens in advance.
It's not so much that we didn't think models plan ahead in general, as that we had various hypotheses (including "unknown unknowns") and this kind of planning in poetry wasn't obviously the best one until we saw the evidence.
[More generally: in Interpretability we often have the experience of being surprised by the specific mechanism a model is using, even though with the benefit of hindsight it seems obvious. E.g. when we did the work for Towards Monosemanticity we were initially quite surprised to see the "the in <context>" features, thought they were indicative of a bug in our setup, and had to spend a while thinking about them and poking around before we realized why the model wanted them (which now feels obvious).]
1Awesome work!
In this section, you describe what seems at first glance to be an example of a model playing the training game and/or optimizing for reward. I'm curious if you agree with that assessment.
So the model learns to behave in ways that it thinks the RM will reinforce, not just ways they actually reinforce. Right? This seems at least fairly conceptually similar to playing the training game and at least some evidence that reward can sometimes become the optimization target?
FYI, See this paper for a more detailed explanation of the model in question and how it's trained.
A few months ago I complained that automatic translation sucks when you translate between two languages which are not English, and that the result is the same as if you translated through English. When translating between two Slavic languages, even sentences where you practically just had to transcribe Cyrillic to Latin and change a few vowels, both Google Translate and DeepL succeeded to randomize the word order, misgender every noun, and mistranslate concepts that happen to be translated to English as the same word.
I tried some translation today, and from my perspective, Claude is almost perfect. Google Translate sucks the same way it did before (maybe I am not fair here, I did not try exactly the same text), and DeepL is somewhere in between. You can give Claude a book and tell to translate it to another language, and the result is pleasant to read. (I haven't tried other LLMs.)
This seems to support the hypothesis that Claude is multilingual in some deep sense.
I assume that Google Translate does not prioritize English on purpose, it's just that it has way more English texts that any other language, so if e.g. two Russian words map to the same English word, it treats it as strong evidence that those two words are the same. Claude can see that the two words are used differently, and can match them correctly to corresponding two words in a different language. (This is just a guess; I don't really understand how these things work under the hood.)
I found it really interesting how you mentioned that Claude’s explanation of its reasoning is different than the actual reasoning paths it took.
For exploring its actual reasoning pathways, what are the most promising approaches?
Curated. There's a lot of interesting stuff in here and definitely bolsters my feeling that with enough years (could be quite a few) that eventually we'd figure out artificial minds and build powerful ones safely.
The language learning research feels in the direction of natural abstraction. The arithmetic experiment suggests a lack of introspection on Claude's part (more so than deception), the features pressuring it to maintain coherence and outweighing safety training is perhaps amusing and disturbing. The lack of faithful reasoning is disturbing and should make us fear what'll happen in cases where we can't investigate. And of course, evidence of planning ahead is big.
I feel like I hear it less often these days, but the "just a next-token predictor" used to get invoked often as a limitation that meant LLMs could only be so dangerous. Evidence of planning ahead suggests a much more typical, and potentially agentic mind. I also liked the surfaced quote from Chris Olah that simply to give the name of someone who studies the starts, you have to think ahead to choose "an" instead of "a", as a trivial case. The rabbit rhyme case is even stronger.
Subtract the existential concerns, and I'd relate to all this with fascination and excitement.
Fascinating results!
Interestingly, it sounds like faking the chain of thought emerges as a special case of planning ahead. With the rhyming, Claude decides on the word that the line should end with, and then figures out the sentence that gets it there. With the math example, Claude decides on the number that the calculation should end up at, and then figures out the steps that get there.
The poetry case really stuck with me. Claude’s clearly planning rhymes ahead, which already cracks the “just next-token” intuition about autoregressive models. But maybe it’s more than a neat trick. What if this spatial planning is a core capability—like the model’s not just unrolling a string, but navigating a conceptual space toward a target? One could test this by checking how often similar planning circuits pop up in multi-step reasoning tasks. If it’s building a rough "mental map" of where it wants to land, that might explain why bigger context windows boost reasoning so much. Not just more data—more room to plan. Has anyone tried prompting or tracing for this directly?
Cool work!
There are 2 findings about that I found surprising and that I’d be interested in seeing explored through other methods:
- Some LLMs are computing the last digit in addition mostly in base 10 (even when all numbers are a single token?)
- LLMs trained to not hallucinate sometimes decide whether to give an answer or not based on the familiarity of the entity, rather than based on the familiarity of the actual answer (they don’t try to first recover the answer and then refuse to answer if they only recall a low-confidence guess?)
The second one may imply that LLMs are less able to reason about what they are about to say than I thought.
I also find it cool that you measured how good the explanations for your new features are. I find it slightly concerning how bad the numbers are. In particular, I would have expected a sort eval error much below 2% (which is the sort eval error you would get if you perfectly assigned each dataset example to one of 5 balanced categories of features [Edit: My math was wrong. 2% is what you get with 25 categories]), but you find a sort eval error around 10%. Some of that is probably Claude being dumb, but I guess you would also struggle to get below 2% with human labels?
But I also see how very predictive feature explanations might not be necessary. I am looking forward to seeing how Circuit Tracing performs in cases where there are more sources of external validations (e.g. hard auditing games)!
Re: Multi-lingual capabilities
I have not reviewed the recent literature but last I checked (summer 2024), it seemed to me that the literature overwhelmingly drew the conclusion that English for the most part acted as a sort of mentalese, with translation neurons layered across the network, but mostly on the outermost layers on both sides (close to input and output)
It seems in this way that Haiku might be distinct from the heretofore studied models?
Do you think this is just a natural consequence of scale, with the understanding of a sort of a "universal grammar" naturally emerging, or is there anything to Claude's training that'd make it qualitatively different from other models in this regard?
Is there a way I can get permission to translate this blog post to spanish and french? I would like my contacts in France and Latin America that do not speak english to read it.
In the poetry case study, we had set out to show that the model didn't plan ahead, and found instead that it did.
This is a very interesting change (?) from earlier models. I wonder if this is a poetry-specific mechanism given the amount of poetry in the training set, or the application of a more general capability. Do you have any thoughts either way?
I would guess that models plan in this style much more generally. It's just useful in so many contexts. For instance, if you're trying to choose what article goes in front of a word, and that word is fixed by other constraints, you need a plan of what that word is ("an astronomer" not "a astronomer"). Or you might be writing code and have to know the type of the return value of a function before you've written the body of the function, since Python type annotations come at the start of the function in the signature. Etc. This sort of thing just comes up all over the place.
Curated and popular this week
