This post was written as part of the Iliad Fellowship. Inspired by conversations with Richard Ngo, Dmitry Vaintrob, and Brianna Grado-White. To all of these, my thanks.
Preface: I'm confused about how neural networks do and learn computations. In response to a friend's challenge, I'm writing up some interim thoughts. This essay has four parts: the first tries to track what I call the 'default ontology' of the mechinterp community over the years. The second part is about 'representational drift' as an important obstacle to weights-based approaches to circuits. The third part reflects on how 'universality' should shape our explanations of LLM function. The fourth part is a sketch of a 'co-selectionist' view of circuits I have been thinking about. These parts share a common theme but should be readable separately.
I want to understand how neural networks, LLMs in particular, work. In my research I've spent a lot of time trying to think through what kinds of explanatory accounts are best suited to this. In thinking about comparisons between evolution, neuroscience, and deep learning, I've ended up with an intuition like the following:
Large-scale learning processes like deep learning or the brain are different in kind from computers or assemblies of circuits. 'Circuits' and 'computation' are not the natural ontology for understanding what is happening inside LLMs.[1][2]
This post is an attempt to communicate what I mean (part I), why I have this intuition (parts II and III). Part IV is a sketch of a perspective on circuits as emergent 'units of selection'.
Definitions are important - here I'll lay out what I think are the core components of the circuit ontology. The central thesis is something like:
Trained LLMs are well thought of as programs - they consist of a collection of algorithms and heuristics, implemented in a way that is functionally modular and likely sparse in an appropriate sense. These functional modules are circuits; they turn a semantically meaningful input into a semantically meaningful output by an in-principle-understandable algorithm. Given enough effort, we could decompile an LLM into readable Python code.
There are a lot of live variants. The most important axes of variation that come to mind are:
Sparsity: Is sparsity (in any sense) a property we expect the model to 'have', or a pragmatic interpretive lens?
Semantic Bootstrap: How do we bootstrap 'semantic meaning' from the inputs or outputs? Or is there some better, mainly functional criterion for identifying circuits?
Regime Dependence: Should our explanations be allowed to depend on the training data distribution? That is, must the explanation hold on arbitrary inputs?
Metric of Faithfulness: If causal intervention is the gold standard for 'explaining' a circuit, against what should we measure shortfalls of such an explanation? E.g. loss, downstream behaviour.
Localizability: Should the 'circuit' be spatially localizable within the model - to, e.g., a handful of attention heads? If so, must those units' behaviour be completely described by their action within that circuit?
Modularity / Isolability: How do we distinguish between 'two circuits composed' and 'one big circuit'?
First, note that the question of circuits is in principle distinct from theories of representation - the latter being 'feature directions', the 'linear representation hypothesis', etc.
Second, note that these questions are definitional as much as they are pragmatic - we're jointly unsure of what a 'circuit' should be and how to look for them.
I.B. Circuits, Features, and MechInterp
Here I'll give a rough account of how I see the interp community's 'default ontology' as having changed over time. I think this is useful because it relates what kinds of structure we thought neural networks have to how we went looking for them.
I think of things as proceeding in roughly three phases, which I'll call the 'exploratory', the 'mechanistic', and the 'prosaic'. Alternatively, we might call these 'worldviews'. The sketch below is more intellectual autobiography than history, has no pretension to completeness etc., so cum grano salis.
By 'exploratory', I essentially mean 'look at the neural network, and poke it and see if you can make sense of what it's doing'. This work had rather little conceptual baggage attached to it - it was less about 'how can we understand the model-qua-gestalt' but instead 'can we find anything understandable at all'. It seems important that during this era, 'neural networks are just inherently uninterpretable messes' felt like a live hypothesis. I think the best examples are things like Chris Olah's pre-transformer-circuits work (see his blog here) and also the Distill circuits thread.[3]
I would also tentatively put the earliest mechinterp work in this category. E.g., the early induction heads work essentially avoided the 'semantic bootstrap' problem I mentioned above - that of needing to jointly assign semantic meaning to circuits' operations and their inputs/outputs. Their induction head metric is essentially operational - an eigenvalue statistic[4] - and their analysis would have worked as well on transformers which predict sheet music instead of text.
With the advent of the LLM era, we moved towards what I'll call the 'mechanistic' era - we finally got models that fucking have to be 'thinking' for some definition of that word, and we wanted to figure out how. I don't think anyone seriously thought they had the theory of understanding neural networks, but I think the spirit of this era was that we were looking for it. This was the era of sparsity, mechinterp, and SAEs.
The most important organizing principle people posited was 'sparsity' - the idea that for most datasets, the most natural way to compress them was to learn a big conceptual dictionary, such that a sparse subset of these would be active at any given time. The best articulation of this idea I'm aware of is this note in the transformer circuits thread, which does a very good job tying together the intuitions from neuroscience and sparse sensing. This line of thinking leads to SAEs and descendants. The SAE vision focused less on circuits or computation, than on 'feature directions' in the residual stream.
Essentially all[5] of the later transformer-circuits thread research built on the conceptual foundation of linear representation, superposition, sparsity, etc. The apotheosis of the SAE vision is 'scaling monosemanticity'. Following this, Anthropic's interp team tried harder to focus on computation and what gets 'done' with features, resulting in the transcoders and 'biology' work. These are extremely cool, but also, in my opinion, go quite a bit further than SAEs in 'imposing' an ontology on a model - they train, layer-by-layer, a sparse replacement for the base model, replacing each MLP layer by a much larger layer of sparse neurons with more permissive (cross-layer) connectivity.
Separate from the SAE-driven cluster of things, there is also an ongoing tradition of very gears-level mechinterp. Good examples of what I have in mind are things like 'logit lens', automated circuit discovery, etc. This 'gears-level' mechinterp is certainly 'mechanistic', but I would feel disingenuous slotting it into one of my three arbitrary worldviews/eras here - I think looking for circuits assumes less about what's going on inside than do SAEs. Redwood's causal scrubbing was an early negative result on this approach.[6]
This 'mechanistic' era gave way to the 'prosaic' around the time that the GDM mechinterp team published 'negative results for SAEs'. Anthropic's interp work since 'biology' has largely focused on pragmatic interpretability for really large language models - training various interpretable surrogates, interpreters, or empirical studies of LLM behaviour. I think it's fair to say that the field (at least at the frontier) drifted away from work on a "fundamental" science of interpretability - from neuroscience to biology and psychology.[7]
'Prosaic' interpretability is based on the idea that LLMs are fucking complicated, possibly unstructured in important ways, and 'fundamental' interpretability either can't or won't get us to useful tools before we need them for alignment. I think of this era as crystallizing with Neel and co.'s 'pragmatic vision', which gave us the 'pragmatic/ambitious interp' dichotomy. Things like natural language autoencoders (NLAs) are especially characteristic of this era - I recall them being described as 'bitter-lesson-pilled interp'. The spirit of the era is training big interpretable ML models on top of big ML models.
To the extent this era has a 'default ontology', it's that interpretability is essentially machine-learning complete, or perhaps that there aren't (or we can't find) faithful useful accounts of model internals at scale - at least not fast enough to be useful for alignment. On a slightly different reading, the default ontology could also be that 'interpretability' is essentially orthogonal to functionality - that LLMs are a big ol' mess in the way we once feared was true of AlexNet, and that interpretability is thus not a problem to be solved, but an additional desideratum.
So in summary, there are a handful of different ways of thinking about what structures neural networks learn. There are exploratory methods which are reasonably structure-agnostic, circuit-discovery methods, feature-interaction methods, looping back around to structure-agnostic surrogate-model methods. These all come from different ideas of 'where the structure is' in neural networks. Next I'll talk about representational drift and why I think this suggests the need to 'look somewhere else' for the fundamental units of computation/structure.
II. The Central Problems of Noise and Representational Drift
The main idea from neuroscience that updates me against thinking of 'circuits' as residing in a fixed set of weights is that of 'representational drift'. Disclaimer: I'm acting as a synthesist here, not a neuroscientist; I still have much to learn here.
There is in neuroscience a notion of something called an 'engram' - roughly, a localizable 'trace' of having learned some particular thing; the mouse learns the maze, and you circle some neurons where the knowledge of the maze is stored. This is intuitive and I'd be surprised if it was uniformly wrong, but there's more to it.
Instead there is something called 'representational drift' - you run the mouse through the maze, and you circle the neurons where that maze knowledge is etc. - but then you find that that localization is not stable over time. The population of neurons which encode that knowledge seems to migrate around over the timescale of days to weeks. Iiuc this is fairly random motion, rather than 'memories migrate predictably as they age', though that probably happens somewhere too.
This feels fucky because if stuff just moves around at random, how does the brain track it to know how to access it? But I claim this is actually rather natural once you think about the design problem the brain faces - in particular, the problem of plasticity.
Plasticity management is one of the deep problems the brain has to solve - you have to manage activity on many different space- and time-scales in order to preserve bothrepresentational capacityand the ability to continually learn. Too much overall activity in a region means you lose information in the noise, and too much rigidity means you can't learn things anymore, nor can you readily adapt your firing rules to the population.
Imagine you wanted to inscribe some fact in the synapse weights of a group of neurons - you have two problems. One: since there needs to be at least short-timescale regulation of each neuron (if the community's average firing rate isn't regulated, you lose signal, and also maybe have a seizure), your encoding can't depend on very precise values of the weights - it'll have to be encoded diffusely, and since there's always noise, probably redundantly. Two: even once you've done that encoding, you do still need to leave those synapses open for long-term plasticity in case new, more important information comes in, or some of the neurons in your circuit are part of another circuit with interfering updates, so you need to jiggle some stuff around. So by engineering fiat, the brain mostly can't be acting like a von Neumann computer with variables stored at some address with particular access wires. The dynamics of operating a functional brain means the system needs to let 'addresses' of 'facts' move, and once they're allowed to 'move', it's natural to let them move by default.[8] Your learning algorithm will just adapt.
II.B. Representational Drift in Neural Networks
I expect this picture to apply to neural networks at both the micro and macro levels. Here I'll give some concrete ways we know it holds at the micro level. I'll note one could assent to these microscopic phenomena and still expect macroscopically fixed functional units.
Fixing some low level of loss, there remain a drift and a diffusion component causing nonzero motion in the parameters.
When you add noise, you get an effective loss like (see this paper for GD and this one for SGD)
where is the loss on a minibatch . You can imagine doing descent until is minimized, at which point you'll start doing descent on the noise term alone - i.e. even if you've minimized the loss already, you'll keep moving in a deterministic way in order to reduce noise. That to say even once you've learned to implement a particular function, your parameters still have reason to be moving about - not just in a random way, but also in a concerted overall way. If you get to zero on that effective loss too, still there will be merely random motion in the weights.
(here is the stationary average, is the second moment of the minibatch gradient, and is the Hessian), i.e., that at equilibrium, you'll still have nontrivial motion due to batch noise etc. This is an imprecise argument, but the point is that by default, you do not get zero motion in parameter space.
So neural networks have some analogue of this representational drift problem, at least by default. You can hypothesise that the various effective penalties will cause the network to enter the regime where it can just encode stuff at the weights level without needing to worry about it moving around - it definitely does get progressively flatter! But this adds an extra hypothesis which I've found it hard to justify - that 'more noise robust' should for some fundamental reason correspond to circuits in any functional sense.
So to summarize, the brain can't plop information down at known addresses and re-access it like a von Neumann computer, because noisiness favors diffuse encoding and plasticity favors gradually-migratable encodings. There are similar-ish phenomena in neural networks that come from effective loss terms caused by noise, so we can be relatively sure that parameters will always be moving, at least a bit - which means that probably neural networks should want to distribute functionality and representation as broadly as possible. If you're thinking about features, this leads you to think about superposition, which I think everyone is on board with.
It's less clear what this looks like if computation is itself 'diffuse'. There is 'computation in superposition', which is what it says on the tin. But by 'diffuse computation', I really mean a paradigm of computation where the computation is designed under comparable constraints. The difference is like 'writing precise algorithms to operate on noisy data' vs 'noisily writing algorithms to noisily operate on noisy data'. That there should be some kind of spiritually different 'theory of diffuse computation' is I think my central intuition. I expect this 'diffuse computation' to converge to 'normal computation' in some regimes some of the time - I don't doubt that GPT learns some fairly precise algorithms some of the time.
Regardless, I think one promising direction is to think a lot about denoising operations; Dmitry Vaintrob's recent work on this is cool.[9] An example of something that would be borderline between either of these two views would be something like voting among a bag of heuristics.
III. Developmental Motifs Circumscribe Notions of 'Circuits'
III.A. Neurotrophins and Microstructure
I think it's useful to look now at the problem the brain has to solve in order to reliably get useful neural circuitry.
There's some level of large-scale connectivity in the brain which is genetically hardcoded - see this article for a good exposition. For the sake of argument, we'll pretend that there are brain regions and that your brain roughly hardcodes connections between those regions, and that it encodes what it wants each of those regions to do by specifying a distribution of neurotrophins[10] for each of those regions. So there's a global connectivity rule and a local development rule.
Figure: the global connectivity rule says which areas talk to which; the local rule says what each area should become, and is implemented as any one of many microstructures. The local rule is sufficient but not interpretable, and the microstructure is irrelevant by design - so we need to understand both separately.
For example, let's imagine that the neurons are smart enough that the population will always evolve to be one of 8 circuits, and the neurotrophin distribution just sets which of the 8 it becomes. We'll assume we don't know this in advance. To understand that that's what's happening, you need to know how the development rules work and how the neurotrophins select between those behaviours.
The problem roughly factors - we can pretty reasonably map out the connections between these regions, and we can plausibly map out these neurotrophin distributions as well.
But if we want to do interp we have something of a problem - we have no idea why this works reliably - no idea how the th area's connectivity + neurotrophin distribution reliably produces the right functional behaviour. We do know for sure that there's some process by which specifying these things, along with the normal rules for neural development, results in reliably having some particular function.
But by the same token we know that the particular configuration of weights and neurons cannot be the right way to explain what area is doing, because that configuration is only one of a bajillion completely different configurations those neurons could have taken under that local development environment and still produced equivalent functionality.
On this picture there's no need to ask about weights, just as there is no need to ask about the locations of atoms in a gas.
I'm sure there are functional units which implement different computations, but I'm not sure that these are in weight-space - that's why I think it's important to think about how these top-down explanations circumscribe the kinds of bottom-up explanations we will need.
III.B. Back to Neural Networks
This of course isn't directly analogous to neural networks, because we do much less hardcoding - but there still are meaningful structural motifs - layers specialize differently, there are characteristic eNTK spectra,[11] etc.
Here I mean to argue that, conditioning on your neural network + training procedure predictably working at all, most of a good account of this is a combination of generic neural network developmental motifs plus some theory of how this comes to reflect different properties of data.
Put another way, suppose arguendo an LSTM and a transformer learn essentially the same function on some given data, and that this is essentially true for a wide range of tasks we're considering. Then our explanation of what they have learned should be symmetric between them. If we want to know something deep about why training these networks on given data produces a particular outcome, we needn't explain things in terms of QK circuits if we know that the phenomenon occurs in architectures without attention heads. Instead we explain why both architectures have neuro-magic-property-X, then explain why that joint neuro-magic-property-X results in that behaviour.
I don't expect that strat to work irl - instead I argue only that we should focus a lot of effort on these local development rules, and maybe this will help us understand what kinds of structures to look for and where - I don't expect that those rules alone will be useful. But consider 'superposition' - knowing about superposition, any interp methods that don't assume superposition appear confused. Once you understand superposition, you realize you shouldn't be looking at the neurons - you should be looking at feature directions. Once you know about feature directions, you realize you were looking in the 'wrong place'.
I think superposition is one of a few such ontology-changes, some of which we still need to figure out. Personally I expect 'circuits' will become some theory of 'denoising computation', and this will tell us to look for 'interpretable computations' in a different place.
IV. A Co-Selectionist View: Circuits Move Together?
Here I'll talk about one particular operationalization of circuits which I have found useful to think about, based on the idea that circuits should receive correlated gradients - 'live together, die together' - 'neural Darwinism'. tl;dr is that I think there's a principled mathematical description of circuits as 'emergent units of selection', but there's still ambiguity in what the 'type signature' of a circuit is.[12]
The guiding intuition is that circuits, as functional units, should receive correlated gradients - that they are 'emergent units of selection'. Another way to put this is that we should be able to say 'these weights are in the circuit' and the individual gradients should mostly be predicted by how well the whole circuit is reinforced. This is also the natural response to representational drift: if the weights are always moving, then the thing to look for is not a fixed set of weights but whatever moves together under that motion.
It turns out that there's a distantly analogous problem in fluid mechanics - looking for so-called 'coherent structures' in a flow. There's also the related problem of 'community detection' on graphs, where you try to find regions with lots of internal connections and minimal external connections. Turns out you can write these in a very general way so that they become special cases of the same problem.
Figure: codex tried hard to illustrate the analogy.
A 'coherent structure' in fluid mechanics is some bounded volume of fluid such that the boundary doesn't move wildly and that there isn't too much mass flow across it over time. Morally, you look for
Figure: Lagrangian coherent structures in a periodically forced flow, from Wikipedia. The coherent structures are those little eyes - they mostly 'move together' and mostly stay looking like eyes. Reminder that we're talking about persistence selection, so we want to find the right notion of 'persistent' structures in a neural network.
The reason you need this functional is because presumably nothing is ever perfectly coherent, so you look for structures which are as coherent as possible.
The generalization of this is to understand these as coming from some manifold and use the Laplacian on that manifold to write an energy function. Basically, once you tell me what you think the 'type signature' of a circuit is, there's a corresponding coherence functional.
For circuits, this is cool because it lets us swap in different incoherence functions corresponding to different accounts of 'what a circuit is'. For example, circuits as directions in parameter space (in ), as collections of weights (in ), as -dimensional subspaces of parameter space (), etc. You do this by choosing either a Laplacian, a Dirichlet energy, or a manifold to work with - once you specify the 'type signature' of the circuit, this is determined for you.
I'm consciously stopping myself from writing a lot of math. I'm doing some experiments with this with a friend so the math'll be in the writeup. If you want a preview, you can look at my claudeslop notes.
This formulation is really nice because it ports right into dynamical systems, so you get useful tools like Lyapunov exponents for analyzing stability etc.
The point is that you can write down a pretty principled selectionist view of circuits, and this has very nice natural connections to dynamical systems, community detection, etc. It gives us lots of nice ways to say how stable a circuit is over time, how 'not-quite-circuit-y' it is.
But! Look at the degrees of freedom we still have! We still need to tell the theory what the type signature of the circuit is! I'm still working on exploring this, so I'll just note that this type-signature freedom makes things a bit difficult.
Overall I'm optimistic about 'circuits as co-selected units', but I don't think those units will be in weight-space. I'm less sure about activation space. Lord knows there has to be structure somewhere in there...
"Wir müssen wissen. Wir werden wissen." - Hilbert
LLM Usage Statement: I wrote everything - yes sadly the emdashes really are all mine and I really am that insufferable. I had Claude make minor changes, link references, fix figures, and accepted some small corrections. Codex made one of the figures.
There still likely are crisp traditional circuits, but they are likely in the minority. More importantly, I have a hard time visualizing evidence that would settle the question 'how much of Claude is circuits' satisfactorily. ↩︎
A tentative reason for this: humans design circuits to be modular and predictable - see 'static discipline' for digital circuit design. Evolution typically learns to operate in spite of noise, or even use it as in e.g. kinetic proofreading. ↩︎
Having not read all of them, I understand that most(?) of this earlier work is focused on image processing instead of text. Unsure if this is significant, but perhaps natural visualization and not having to handle the complexity of sequence models lends itself to more methodological flexibility? Speculative. ↩︎
Claude tells me that different induction head works have used different operationalizations, some less principled than the OV eigenvalue definition. I'm unsure if this matters for my point here. ↩︎
I'll note that I am probably overindexing on Anthropic's work here, even within the LW-adjacent research community. I am unsure if this reflects their impact on the community or is just my own sample bias. ↩︎
Causal scrubbing predates the SAE life-cycle - 'Scaling Monosemanticity' came out mid-2024, while causal scrubbing was end of 2022. Just be aware this is a bit garbled as a timeline. ↩︎
This is my read of overall field vibes; I expect this is distorted by overweighting the transformer-circuits thread, but I expect a similar effect to hold for the overall community. ↩︎
Global changes in plasticity I think are moderate evidence against this argument, but long-term plasticity is structurally different than short-term - idk more thinking needed. ↩︎
See this video, which talks about a method of training transformers inspired by thinking of them as doing denoising diffusion. ↩︎
Strictly, the linked article is about cell types and transcription factors rather than neurotrophins. The distinction isn't the point for now - read 'neurotrophins' as 'whatever local chemical signal sets a region's developmental program'. ↩︎
The empirical neural tangent kernel: the Gram matrix of per-example gradients at the current parameters. Its spectrum tells you which directions in function space the network can currently move in easily, and it has a characteristic shape (a few large outliers plus a power-law-ish bulk) across architectures. ↩︎
Note here I mean persistence selection rather than real Darwinism - circuits don't have offspring. ↩︎
This post was written as part of the Iliad Fellowship. Inspired by conversations with Richard Ngo, Dmitry Vaintrob, and Brianna Grado-White. To all of these, my thanks.
Preface: I'm confused about how neural networks do and learn computations. In response to a friend's challenge, I'm writing up some interim thoughts. This essay has four parts: the first tries to track what I call the 'default ontology' of the mechinterp community over the years. The second part is about 'representational drift' as an important obstacle to weights-based approaches to circuits. The third part reflects on how 'universality' should shape our explanations of LLM function. The fourth part is a sketch of a 'co-selectionist' view of circuits I have been thinking about. These parts share a common theme but should be readable separately.
I want to understand how neural networks, LLMs in particular, work. In my research I've spent a lot of time trying to think through what kinds of explanatory accounts are best suited to this. In thinking about comparisons between evolution, neuroscience, and deep learning, I've ended up with an intuition like the following:
This post is an attempt to communicate what I mean (part I), why I have this intuition (parts II and III). Part IV is a sketch of a perspective on circuits as emergent 'units of selection'.
Contents
I. What Might We Mean By "Circuits"?
I.A. Definitions
Definitions are important - here I'll lay out what I think are the core components of the circuit ontology. The central thesis is something like:
There are a lot of live variants. The most important axes of variation that come to mind are:
First, note that the question of circuits is in principle distinct from theories of representation - the latter being 'feature directions', the 'linear representation hypothesis', etc.
Second, note that these questions are definitional as much as they are pragmatic - we're jointly unsure of what a 'circuit' should be and how to look for them.
I.B. Circuits, Features, and MechInterp
Here I'll give a rough account of how I see the interp community's 'default ontology' as having changed over time. I think this is useful because it relates what kinds of structure we thought neural networks have to how we went looking for them.
I think of things as proceeding in roughly three phases, which I'll call the 'exploratory', the 'mechanistic', and the 'prosaic'. Alternatively, we might call these 'worldviews'. The sketch below is more intellectual autobiography than history, has no pretension to completeness etc., so cum grano salis.
By 'exploratory', I essentially mean 'look at the neural network, and poke it and see if you can make sense of what it's doing'. This work had rather little conceptual baggage attached to it - it was less about 'how can we understand the model-qua-gestalt' but instead 'can we find anything understandable at all'. It seems important that during this era, 'neural networks are just inherently uninterpretable messes' felt like a live hypothesis. I think the best examples are things like Chris Olah's pre-transformer-circuits work (see his blog here) and also the Distill circuits thread. [3]
I would also tentatively put the earliest mechinterp work in this category. E.g., the early induction heads work essentially avoided the 'semantic bootstrap' problem I mentioned above - that of needing to jointly assign semantic meaning to circuits' operations and their inputs/outputs. Their induction head metric is essentially operational - an eigenvalue statistic [4] - and their analysis would have worked as well on transformers which predict sheet music instead of text.
With the advent of the LLM era, we moved towards what I'll call the 'mechanistic' era - we finally got models that fucking have to be 'thinking' for some definition of that word, and we wanted to figure out how. I don't think anyone seriously thought they had the theory of understanding neural networks, but I think the spirit of this era was that we were looking for it. This was the era of sparsity, mechinterp, and SAEs.
The most important organizing principle people posited was 'sparsity' - the idea that for most datasets, the most natural way to compress them was to learn a big conceptual dictionary, such that a sparse subset of these would be active at any given time. The best articulation of this idea I'm aware of is this note in the transformer circuits thread, which does a very good job tying together the intuitions from neuroscience and sparse sensing. This line of thinking leads to SAEs and descendants. The SAE vision focused less on circuits or computation, than on 'feature directions' in the residual stream.
Essentially all [5] of the later transformer-circuits thread research built on the conceptual foundation of linear representation, superposition, sparsity, etc. The apotheosis of the SAE vision is 'scaling monosemanticity'. Following this, Anthropic's interp team tried harder to focus on computation and what gets 'done' with features, resulting in the transcoders and 'biology' work. These are extremely cool, but also, in my opinion, go quite a bit further than SAEs in 'imposing' an ontology on a model - they train, layer-by-layer, a sparse replacement for the base model, replacing each MLP layer by a much larger layer of sparse neurons with more permissive (cross-layer) connectivity.
Separate from the SAE-driven cluster of things, there is also an ongoing tradition of very gears-level mechinterp. Good examples of what I have in mind are things like 'logit lens', automated circuit discovery, etc. This 'gears-level' mechinterp is certainly 'mechanistic', but I would feel disingenuous slotting it into one of my three arbitrary worldviews/eras here - I think looking for circuits assumes less about what's going on inside than do SAEs. Redwood's causal scrubbing was an early negative result on this approach. [6]
This 'mechanistic' era gave way to the 'prosaic' around the time that the GDM mechinterp team published 'negative results for SAEs'. Anthropic's interp work since 'biology' has largely focused on pragmatic interpretability for really large language models - training various interpretable surrogates, interpreters, or empirical studies of LLM behaviour. I think it's fair to say that the field (at least at the frontier) drifted away from work on a "fundamental" science of interpretability - from neuroscience to biology and psychology. [7]
'Prosaic' interpretability is based on the idea that LLMs are fucking complicated, possibly unstructured in important ways, and 'fundamental' interpretability either can't or won't get us to useful tools before we need them for alignment. I think of this era as crystallizing with Neel and co.'s 'pragmatic vision', which gave us the 'pragmatic/ambitious interp' dichotomy. Things like natural language autoencoders (NLAs) are especially characteristic of this era - I recall them being described as 'bitter-lesson-pilled interp'. The spirit of the era is training big interpretable ML models on top of big ML models.
To the extent this era has a 'default ontology', it's that interpretability is essentially machine-learning complete, or perhaps that there aren't (or we can't find) faithful useful accounts of model internals at scale - at least not fast enough to be useful for alignment. On a slightly different reading, the default ontology could also be that 'interpretability' is essentially orthogonal to functionality - that LLMs are a big ol' mess in the way we once feared was true of AlexNet, and that interpretability is thus not a problem to be solved, but an additional desideratum.
So in summary, there are a handful of different ways of thinking about what structures neural networks learn. There are exploratory methods which are reasonably structure-agnostic, circuit-discovery methods, feature-interaction methods, looping back around to structure-agnostic surrogate-model methods. These all come from different ideas of 'where the structure is' in neural networks. Next I'll talk about representational drift and why I think this suggests the need to 'look somewhere else' for the fundamental units of computation/structure.
II. The Central Problems of Noise and Representational Drift
Figure: representational drift, and also cats. Sure. From 'Statistical learning and representational drift: a dynamic substrate for memories'.
II.A. Representational Drift in the Brain
The main idea from neuroscience that updates me against thinking of 'circuits' as residing in a fixed set of weights is that of 'representational drift'. Disclaimer: I'm acting as a synthesist here, not a neuroscientist; I still have much to learn here.
There is in neuroscience a notion of something called an 'engram' - roughly, a localizable 'trace' of having learned some particular thing; the mouse learns the maze, and you circle some neurons where the knowledge of the maze is stored. This is intuitive and I'd be surprised if it was uniformly wrong, but there's more to it.
Instead there is something called 'representational drift' - you run the mouse through the maze, and you circle the neurons where that maze knowledge is etc. - but then you find that that localization is not stable over time. The population of neurons which encode that knowledge seems to migrate around over the timescale of days to weeks. Iiuc this is fairly random motion, rather than 'memories migrate predictably as they age', though that probably happens somewhere too.
This feels fucky because if stuff just moves around at random, how does the brain track it to know how to access it? But I claim this is actually rather natural once you think about the design problem the brain faces - in particular, the problem of plasticity.
Plasticity management is one of the deep problems the brain has to solve - you have to manage activity on many different space- and time-scales in order to preserve both representational capacity and the ability to continually learn. Too much overall activity in a region means you lose information in the noise, and too much rigidity means you can't learn things anymore, nor can you readily adapt your firing rules to the population.
Imagine you wanted to inscribe some fact in the synapse weights of a group of neurons - you have two problems. One: since there needs to be at least short-timescale regulation of each neuron (if the community's average firing rate isn't regulated, you lose signal, and also maybe have a seizure), your encoding can't depend on very precise values of the weights - it'll have to be encoded diffusely, and since there's always noise, probably redundantly. Two: even once you've done that encoding, you do still need to leave those synapses open for long-term plasticity in case new, more important information comes in, or some of the neurons in your circuit are part of another circuit with interfering updates, so you need to jiggle some stuff around. So by engineering fiat, the brain mostly can't be acting like a von Neumann computer with variables stored at some address with particular access wires. The dynamics of operating a functional brain means the system needs to let 'addresses' of 'facts' move, and once they're allowed to 'move', it's natural to let them move by default. [8] Your learning algorithm will just adapt.
II.B. Representational Drift in Neural Networks
I expect this picture to apply to neural networks at both the micro and macro levels. Here I'll give some concrete ways we know it holds at the micro level. I'll note one could assent to these microscopic phenomena and still expect macroscopically fixed functional units.
Fixing some low level of loss, there remain a drift and a diffusion component causing nonzero motion in the parameters.
When you add noise, you get an effective loss like (see this paper for GD and this one for SGD)
where
Even at equilibrium, you can use the 'fluctuation-dissipation relations for SGD' to show
and
(here
So neural networks have some analogue of this representational drift problem, at least by default. You can hypothesise that the various effective penalties will cause the network to enter the regime where it can just encode stuff at the weights level without needing to worry about it moving around - it definitely does get progressively flatter! But this adds an extra hypothesis which I've found it hard to justify - that 'more noise robust' should for some fundamental reason correspond to circuits in any functional sense.
So to summarize, the brain can't plop information down at known addresses and re-access it like a von Neumann computer, because noisiness favors diffuse encoding and plasticity favors gradually-migratable encodings. There are similar-ish phenomena in neural networks that come from effective loss terms caused by noise, so we can be relatively sure that parameters will always be moving, at least a bit - which means that probably neural networks should want to distribute functionality and representation as broadly as possible. If you're thinking about features, this leads you to think about superposition, which I think everyone is on board with.
It's less clear what this looks like if computation is itself 'diffuse'. There is 'computation in superposition', which is what it says on the tin. But by 'diffuse computation', I really mean a paradigm of computation where the computation is designed under comparable constraints. The difference is like 'writing precise algorithms to operate on noisy data' vs 'noisily writing algorithms to noisily operate on noisy data'. That there should be some kind of spiritually different 'theory of diffuse computation' is I think my central intuition. I expect this 'diffuse computation' to converge to 'normal computation' in some regimes some of the time - I don't doubt that GPT learns some fairly precise algorithms some of the time.
Regardless, I think one promising direction is to think a lot about denoising operations; Dmitry Vaintrob's recent work on this is cool. [9] An example of something that would be borderline between either of these two views would be something like voting among a bag of heuristics.
III. Developmental Motifs Circumscribe Notions of 'Circuits'
III.A. Neurotrophins and Microstructure
I think it's useful to look now at the problem the brain has to solve in order to reliably get useful neural circuitry.
There's some level of large-scale connectivity in the brain which is genetically hardcoded - see this article for a good exposition. For the sake of argument, we'll pretend that there are
Figure: the global connectivity rule says which areas talk to which; the local rule says what each area should become, and is implemented as any one of many microstructures. The local rule is sufficient but not interpretable, and the microstructure is irrelevant by design - so we need to understand both separately.
For example, let's imagine that the neurons are smart enough that the population will always evolve to be one of 8 circuits, and the neurotrophin distribution just sets which of the 8 it becomes. We'll assume we don't know this in advance. To understand that that's what's happening, you need to know how the development rules work and how the neurotrophins select between those behaviours.
The problem roughly factors - we can pretty reasonably map out the connections between these
But if we want to do interp we have something of a problem - we have no idea why this works reliably - no idea how the
But by the same token we know that the particular configuration of weights and neurons cannot be the right way to explain what area
On this picture there's no need to ask about weights, just as there is no need to ask about the locations of atoms in a gas.
I'm sure there are functional units which implement different computations, but I'm not sure that these are in weight-space - that's why I think it's important to think about how these top-down explanations circumscribe the kinds of bottom-up explanations we will need.
III.B. Back to Neural Networks
This of course isn't directly analogous to neural networks, because we do much less hardcoding - but there still are meaningful structural motifs - layers specialize differently, there are characteristic eNTK spectra, [11] etc.
Here I mean to argue that, conditioning on your neural network + training procedure predictably working at all, most of a good account of this is a combination of generic neural network developmental motifs plus some theory of how this comes to reflect different properties of data.
Put another way, suppose arguendo an LSTM and a transformer learn essentially the same function on some given data, and that this is essentially true for a wide range of tasks we're considering. Then our explanation of what they have learned should be symmetric between them. If we want to know something deep about why training these networks on given data produces a particular outcome, we needn't explain things in terms of QK circuits if we know that the phenomenon occurs in architectures without attention heads. Instead we explain why both architectures have neuro-magic-property-X, then explain why that joint neuro-magic-property-X results in that behaviour.
I don't expect that strat to work irl - instead I argue only that we should focus a lot of effort on these local development rules, and maybe this will help us understand what kinds of structures to look for and where - I don't expect that those rules alone will be useful. But consider 'superposition' - knowing about superposition, any interp methods that don't assume superposition appear confused. Once you understand superposition, you realize you shouldn't be looking at the neurons - you should be looking at feature directions. Once you know about feature directions, you realize you were looking in the 'wrong place'.
I think superposition is one of a few such ontology-changes, some of which we still need to figure out. Personally I expect 'circuits' will become some theory of 'denoising computation', and this will tell us to look for 'interpretable computations' in a different place.
IV. A Co-Selectionist View: Circuits Move Together?
Here I'll talk about one particular operationalization of circuits which I have found useful to think about, based on the idea that circuits should receive correlated gradients - 'live together, die together' - 'neural Darwinism'. tl;dr is that I think there's a principled mathematical description of circuits as 'emergent units of selection', but there's still ambiguity in what the 'type signature' of a circuit is. [12]
The guiding intuition is that circuits, as functional units, should receive correlated gradients - that they are 'emergent units of selection'. Another way to put this is that we should be able to say 'these
It turns out that there's a distantly analogous problem in fluid mechanics - looking for so-called 'coherent structures' in a flow. There's also the related problem of 'community detection' on graphs, where you try to find regions with lots of internal connections and minimal external connections. Turns out you can write these in a very general way so that they become special cases of the same problem.
Figure: codex tried hard to illustrate the analogy.
A 'coherent structure' in fluid mechanics is some bounded volume of fluid such that the boundary doesn't move wildly and that there isn't too much mass flow across it over time. Morally, you look for
Figure: Lagrangian coherent structures in a periodically forced flow, from Wikipedia. The coherent structures are those little eyes - they mostly 'move together' and mostly stay looking like eyes. Reminder that we're talking about persistence selection, so we want to find the right notion of 'persistent' structures in a neural network.
The reason you need this functional is because presumably nothing is ever perfectly coherent, so you look for structures which are as coherent as possible.
The generalization of this is to understand these as coming from some manifold and use the Laplacian on that manifold to write an energy function. Basically, once you tell me what you think the 'type signature' of a circuit is, there's a corresponding coherence functional.
For circuits, this is cool because it lets us swap in different incoherence functions corresponding to different accounts of 'what a circuit is'. For example, circuits as directions in parameter space (in
I'm consciously stopping myself from writing a lot of math. I'm doing some experiments with this with a friend so the math'll be in the writeup. If you want a preview, you can look at my claudeslop notes.
This formulation is really nice because it ports right into dynamical systems, so you get useful tools like Lyapunov exponents for analyzing stability etc.
The point is that you can write down a pretty principled selectionist view of circuits, and this has very nice natural connections to dynamical systems, community detection, etc. It gives us lots of nice ways to say how stable a circuit is over time, how 'not-quite-circuit-y' it is.
But! Look at the degrees of freedom we still have! We still need to tell the theory what the type signature of the circuit is! I'm still working on exploring this, so I'll just note that this type-signature freedom makes things a bit difficult.
Overall I'm optimistic about 'circuits as co-selected units', but I don't think those units will be in weight-space. I'm less sure about activation space. Lord knows there has to be structure somewhere in there...
LLM Usage Statement: I wrote everything - yes sadly the emdashes really are all mine and I really am that insufferable. I had Claude make minor changes, link references, fix figures, and accepted some small corrections. Codex made one of the figures.
There still likely are crisp traditional circuits, but they are likely in the minority. More importantly, I have a hard time visualizing evidence that would settle the question 'how much of Claude is circuits' satisfactorily. ↩︎
A tentative reason for this: humans design circuits to be modular and predictable - see 'static discipline' for digital circuit design. Evolution typically learns to operate in spite of noise, or even use it as in e.g. kinetic proofreading. ↩︎
Having not read all of them, I understand that most(?) of this earlier work is focused on image processing instead of text. Unsure if this is significant, but perhaps natural visualization and not having to handle the complexity of sequence models lends itself to more methodological flexibility? Speculative. ↩︎
Claude tells me that different induction head works have used different operationalizations, some less principled than the OV eigenvalue definition. I'm unsure if this matters for my point here. ↩︎
I'll note that I am probably overindexing on Anthropic's work here, even within the LW-adjacent research community. I am unsure if this reflects their impact on the community or is just my own sample bias. ↩︎
Causal scrubbing predates the SAE life-cycle - 'Scaling Monosemanticity' came out mid-2024, while causal scrubbing was end of 2022. Just be aware this is a bit garbled as a timeline. ↩︎
This is my read of overall field vibes; I expect this is distorted by overweighting the transformer-circuits thread, but I expect a similar effect to hold for the overall community. ↩︎
Global changes in plasticity I think are moderate evidence against this argument, but long-term plasticity is structurally different than short-term - idk more thinking needed. ↩︎
See this video, which talks about a method of training transformers inspired by thinking of them as doing denoising diffusion. ↩︎
Strictly, the linked article is about cell types and transcription factors rather than neurotrophins. The distinction isn't the point for now - read 'neurotrophins' as 'whatever local chemical signal sets a region's developmental program'. ↩︎
The empirical neural tangent kernel: the Gram matrix
Note here I mean persistence selection rather than real Darwinism - circuits don't have offspring. ↩︎