Nick Cammarata of Goodfire makes a bold claim: Neural Networks are Already Clean. This was his follow up after various resulting online discussions.
This is clearly good work that tells us new and important things. The question is, how big an update is this, how much progress does it represent and what is the outlook going forward?
The reactions here range from the super optimistic takes of ‘this means mechanistic interpretability is solvable,’ or even ‘all but solved,’ with comparisons to the famous paper Attention is All You Need.
Dean W. Ball (OpenAI): There is so much to say about this, and I will, but WOW, congratulations to Nick and Goodfire. This may be one of the most important single days in deep learning history, and perhaps much more than that.
No matter what, this is a victory for the dogged pursuit of understanding.
roon (OpenAI): when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even Eliezer will have to lower his pdoom. we can start making alignment into a ‘normal science’
This paper is solid and meaningful contribution that is in some ways overstated. Get excited, but also there are those who need to take a chill pill. We did not actually get an apples-to-apples jump from 1.7% to 90% interpretability of LLM neurons.
I interpret Eliezer as slightly lowering his p(doom), but not all that much.
Evžen Wybitul: What’s happening? Half of my tl is non-interp people from OAI painting the post as the second coming of Christ and the other half is interp researchers (including Leo Gao) saying it’s… fine
This post is my attempt to sort this out. Currently I am landing somewhere in the middle. It is an important update about how LLMs and minds form and operate. It tells us how to go about some useful tasks and reduces our estimate of some task difficulties. Great work, assuming the work checks out. The whole thing very much has a ring of truth, and of a person being curious. You love to see it.
What it does not tell us is that interpretability will soon be solved, or is about to be made into a science, let alone that alignment will soon be handled. The hype got rather out of hand in some places.
Before I cover reactions or take a side, here are my highlights. It is worth reading at least this much of the primary source. Reading the whole thing is an option, but it is long and the writing can be easy to bounce off of and can repeat itself.
Nick Cammarata, Goodfire: The cause of polysemanticity in early model organisms wasn’t superposition, but several ordinary, mostly fixable issues.
We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.
I think I have good news about the nature of neural networks.
After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared.
The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread.
What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning.
One Neuron, One Central Concept
The central claim is that for larger LLMs, most neurons represents a singular concept and things clustered around that concept, perhaps with small offshoots and some use of low-level activations.
… Modern language models are also far more interpretable than generally believed.
… With better tools and modern models, the situation looks quite different. In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood.
Dense and Mixture of Experts models seem roughly similar.
We found similar results in models up to trillions of parameters, like Kimi K3, which the companion paper will cover in more detail.
Where previously the working theory was that models overloaded neurons with unrelated concepts in superposition, Nick’s theory is you did not think sufficiently hard to identify the unifying concept. Each neuron is a generalization, mostly ones we can readily understand if we look.
… What matters a lot more is why things share a neuron in the first place. Most of the remaining apparent polysemanticity seems to be a motif I call recursive generalization. It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all.
… So there were two explanations for polysemanticity, and I helped propose both. In some ways, they’re opposites: generalization groups similar things together and superposition groups unrelated things together. Superposition is the one the field bet on and, until recently, so did I. But I now think recursive generalization was basically right.
… Interpretability has a surprising amount of momentum: like a jigsaw puzzle or crossword, any understanding you gain starts constraining the rest of the system.
… Take a neuron that tracks ‘a sense of righteousness when a bargain purchase turns out to work better than the expensive alternative’. Neurons that track a feeling or social cue like this are some of the most common we’ve found.
This pictures intelligence as brute force. Neurons are compound words in German.
New concept? Go get another neuron. And yet, it works.
I Can See Everything
… The bad news about modern neural networks is that they’re enormous. The good news is that the enormity helps: as the map fills in, every explanation is tightly constrained by everything around it, and bigger models learn narrower neurons that are easier to recognize in the first place.
In this sense, understanding the whole network may be easier than understanding just the parts we care about most.
Nick argues that this means that it is easier to solve the full puzzle and understand everything, and then take a measurement of your specific area of interest, then to try to directly pull a fuzzy concept like ‘deception’ out of a pool of mystery neurons.
… “deception” is a useful initial frame, but the building blocks that contribute to deceptive behavior by models probably have alignment-relevant microstructure that will be hard to know without going in and looking.
… All of this has brought me back to an old hope: understanding a neural network directly, by studying its mechanisms and the data flowing between them, without first trying to fix or untangle it. Put together, the argument has four steps.
Networks use one learning principle, not two (if not everywhere, then nearly enough that it doesn’t matter for study), so there’s no blocker to beginning ordinary science on them.
With modern models and tools, that science goes much better than most people expect.
Understanding compounds, so it gets easier as the map fills in. At minimum, then, a network’s own mechanisms should be part of the toolkit.
Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we’ve assumed.
That’s been the pattern for deep learning in general: the simple thing at absurd scale beats the clever one.
This claim is maximal and bold, and will get directly challenged. Most people seem to have updated from this paper that more neurons have central singular concepts behind them than we previously thought. Nick is trying to go further, and say that this is what is going on universally.
What a Concept
… I opened up the cat and car image grid I’d seen hundreds of times before and did something I’d never done: I spent a few seconds wondering what one thing this might be. Almost instantly it was obvious: it’s not a cat and car neuron, it’s a V-shape neuron.
there are Lady Reds in the future: What is this extremely mysterious neuron that set mechinterp going on a dead end for the last 3 years?
It’s literally
</\> kiki ?
or bouba o0o
Remember this the next time you think you can never figure anything out, or that surely people have already asked the basic questions and considered the obvious. Maybe they have. Maybe they haven’t.
Paper (resuming): … InceptionV1 has more than a hundred of these generalized pose neurons, each one its own clean detector of a pose shared across cars, dogs and other objects, and to my knowledge no one had noticed them.
… I can often unblock my understanding of a neuron by asking: if I couldn’t use groups of nouns at all, how would I describe this neuron to someone else?
… We should treat a neuron that means two or three things as one neuron with a few grouped meanings. The grouping probably wasn’t a problem for the network: the meanings sit together because, from the network’s point of view, they’re similar.
… Polysemanticity is often more a claim about the observer than the neuron. It says there’s no good explanation yet. There’s a few reasons that could be:
Bad tools: our lens or microscope isn’t good enough
Bad explanations: created from those tools
Bad learning: the network learned poorly due to bad architecture, training data, or something else
Recursive generalization: while generalizing, the neuron branched off in a few related directions
Superposition: a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide.
… In a frontier language model, I found a neuron that mostly fired on “pr”, but also on a bunch of Base64 URL strings as well as some Bengali. I thought this must finally be an arbitrary mixture. But decode the URLs: “pr” words. Translate the Bengali: “pr”.
… In a dense Qwen model, a coworker came across a neuron that standard top-k automated interpretability lists as a “polysemantic comma neuron”. But looking more closely, it turned out to track Colorado: phone numbers, slang, and connections to other Colorado neurons.
Steering it made the model want to go hiking. The commas were where it stored a running summary of the Colorado context so far. In general attention seems to collect on the punctuation, which we call the summary motif. Once you know it, a lot of similar neurons get easier to understand.
… In another case a neuron looked like an arbitrary mess, with nothing firing strongly. It turns out it looked for specific signs within pages of health SEO spam, and once we had a sufficiently broad dataset, it was understandable.
Nick does notice the skulls that result from going full ‘This Is Not a Coincidence Because Nothing Is Ever a Coincidence.’
… This risks the opposite error, an explanation of the gaps: assuming every neuron must have a sensible story. Two things make it less worrying. First, each of the other causes can be tested. … Second, real explanations tend to be narrow and easily testable. Once you suspect a neuron is a Colorado neuron, steer it and see what it says.
The strong hypothesis is that neurons will all have central concepts like Colorado. The even stronger hypothesis is that there will not also be unrelated additional concepts happening behind the scenes.
… I used to think of neurons as human-shaped nouns: one is a car, another is a curve. Now I think of them more as control flow. A neuron is less about what it sees than about what it does to the flow coming in, and often its job is to gate a region of the network.
… The negative side of neurons matters almost as much as positives for understanding them, even in ReLU networks where the negative side is cut off.
… Some neurons hold their meaning far down the range; others drop off a cliff, like the Christmas neuron above at about 5% of its max. Some fire a few times a book, others a few times a page.
The More You Know
The core theory is that perhaps intelligence is a simpler concept than we think? It is all a series of coherent concepts with links? There’s a ‘Hitchhiker’s Guide state unit that peaks on the title and the Answer slot’ that weakly fires on all 42s? There is a neuron specifically for the joke about fusion always being a fixed amount into the future that fires on other similar repeated promises?
What we see is about 90% of neurons having basically one job, with recursive generalization offshoots that are usually related. … I’ve never seen a neuron with something like ten unrelated things it does.
Nick Cammarata, Goodfire: Here’s an attribution graph from Kimi K3 using only neurons. As far as we know, it’s the first published attribution graph of a multiple trillion parameter model. Asked what to do with a wallet it found, it links the wallet, the cash and the owner’s ID to lost property and honesty, and answers “return”. We haven’t begun full audits of this scale yet, and hope to soon, but so far most things have been intuitively interpretable.
Simple things done well. Echoes. Vibes. No papers or essays, only references to papers and essays. A fully bitter lesson. The more bitter, the easier the lesson.
In 2023 during the time of the first automated interpretability project
I thought small simplified models would be the best starting place for interpretability, since there’s fewer bells and whistles to think about, and small size means fewer things to study. But now I think it may be the opposite.
Attention Is Still What You Need
Then there’s the residual stream and attention. The essay feels like it’s trying to hide how much of what is going on is happening outside of the neuron labels.
The worry with the residual stream isn’t that there are more things we care about than dimensions. It’s a continuous version of packing: manifolds woven through each other, accepting some interference in exchange for room to store more. Ruling that out entirely may not be possible. But in practice it hasn’t been a bottleneck.
We can read what neurons write to the residual stream and what they read from it, attribution graphs come out clean, and we don’t see the stream accumulating interference only to undo it later.
… A stream can hold far more than it has dimensions just by placing things in the right directions, so this alone isn’t packing. It mostly matters if you want a big list of all the model’s features, and people often go to the stream for exactly that.
… We’ve looked at language models’ residual streams less, but what we’ve seen fits the same story. For understanding higher-level structure and motifs, nothing in the residual stream has blocked us so far.
… We’ve made less progress on attention than on neurons, and it’s mostly outside the scope of this essay. To be honest, I’m not sure anyone understands attention well enough to confidently say that illegibility is a superposition in the sense we care about.
… Overall, we don’t think attention and neurons are so separable. Understanding where attention is putting information is quite helpful for understanding the neurons that then work with that information.
This is one place Nick seems to overreach. Yes, we can look at where attention guides you without knowing what mechanism allocates your attention, and then look at the neurons that get triggered. I don’t think this ‘gets you out of’ having to understand that process. It is worth looking at the neurons anyway. This does not allow you to predict future LLM outputs or fully understand how it thinks.
Does this mean you can ‘sit on the neurons that matter’ to guard against shenanigans or otherwise play good defense, if you don’t make more progress? To some extent, but I would assume that an LLM that knows about this type of monitoring would be able to work around this, and figure out how ‘not to think about’ what you look for.
Occam’s Razor
The theory of why all the neurons this is simple: That this is simple.
… I think the reason things end up so organized is partially because it was faster to build that way. Once you have a car area of the neural network with car parts, it’s easier to add the marginal car there than to build a second car area somewhere else. I think this is closely related to ideas of circuit racing and gradient starvation.
… Even if the map is understandable, that doesn’t mean you’ve understood it. We still have to do the ordinary science of conjecture and criticism: guess what something is, try to break the guess, and keep what survives. The network is strange, but its strangeness repeats, so a guess that survives scrutiny often pays off many times over.
… Solving interpretability may share something with achieving AGI: the goalposts move as you get closer. There’s always another weak activation to point to.
… Behind doubts about small errors in neurons is a fear that safety will come down to some activation we missed, the place where deception hides. The risk I worry about is different: like being in a well-organized city and not knowing how to think about it, because we aren’t native to it.
… I think the simple story from introductory textbooks is basically right: neurons generalize, work together, and build abstractions over layers. Somewhere along the way we overcomplicated that picture, but looking harder brought it back. If the network isn’t a knot, untangling it can make it look like one.
The Neurons Sometimes Share Clear Themes
Sauers notices patterns of overlap of neurons between different features in Qwen3-8B. This seems highly consistent with Nick’s theory.
Hype
roon: I will say that even in bona fide a scientific revolution, you wouldn’t expect the fields previous experts to be the first convinced. in fact, they may be the last
Lake Snyder: It’s an interesting dichotomy that Roon is completely convinced but the (slightly) older guard is skeptical.
roon: I honestly think that being in mechinterp pre this paper may have been an infohazard if what he is saying is true
roon: I feel like my understanding of what a neural net is has been catapulted into the future by five years or something. it has brought back a lot of the sheer joy I had for studying the mystery of deep learning. neurons are already clean! they form a wonderful alien ontology!
We saw some hype up top. Here is some more hype.
Andrew Curran: This would be a massive positive update for me on the future. I had long since given up on reaching any real level of interpretability before superintelligence arrived. I stopped writing about it because of how demoralizing it was. Hard to overstate how big this may be.
… To help people understand how far away this seemed, this is Dario in April of last year – in what at the time was a huge positive update on this subject – saying we would reach it within 5-10 years. On many timelines now, including my own, this would have been too late.
Dario Amodei (April 2025): On one hand, recent progress—especially the results on circuits and on interpretability-based testing of models—has made me feel that we are on the verge of cracking interpretability in a big way. Although the task ahead of us is Herculean, I can see a realistic path towards interpretability being a sophisticated and reliable way to diagnose problems in even very advanced AI—a true “MRI for AI”. In fact, on its current trajectory I would bet strongly in favor of interpretability reaching this point within 5-10 years.
This is exactly the mistake Eliezer Yudkowsky will be ranting about later in this post. Yes, this is progress. No, this is not an ‘MRI for AI.’ That is still a long way off.
Eric Ho (Goodfire): it’s been an enormous privilege and joy to watch @nickcammarata cook this up over the last year. week after week, experiment after experiment, he unraveled the tangled mess of our thinking about neural nets to reveal the truth.
i never knew quite what to expect on our weekly walks. each conversation brought a torrent of new theories about the nature of AI, some conjectures teetering on the brink of madness, and others that seemed to cut through years of confusion with a single sentence.
nick is one of the great scientists of our time. his intuition and research taste are out of this world, but what sets him apart is his unmatched ability to capture the essence of what something truly is, without being lost in the details. i would highly recommend you read his entire essay if you’re curious about AI. it’s long, but written beautifully.
nick’s research makes me far more optimistic about the future of AI alignment. i hope that it can inspire the same for you.
Goodfire: For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that’s the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire’s @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there’s reason for optimism today, and why there might be a bitter lesson for interpretability.
Adam Majmudar (OpenAI): Yea this is the most important result of the year, at the level of significance of “Attention Is All You Need”. It brings us from “is alignment even solvable” to “we may solve alignment in the next year.” And finally unlocks true recursive self-alignment enabled by RSI.
1/ neural networks are not fundamentally unintelligible to humans
2/ neural networks are fundamentally clean
3/ the language of humans and the language of neural networks is the same
4/ alignment can progress as fast as RSI. recursive self-improvement unlocks recursive self-alignment and a scaling law for alignment.
adammaj: I def don’t think this is an argument for RSI or that alignment is solved. It’s just a reason for optimism in an otherwise bleak landscape of the rate of progress, which is what I intended to express
adammaj: Lemme add some VERY important nuance after reading some comments:
> When I say “we may solve alignment in the next year” I should have said “we may solve controllability” – I don’t mean the whole alignment problem. Values is a completely different problem. Also 1 year is ambitious. Maybe 2-4 years is more realistic. But you have to price in the rapid increases in rate of progress which keeps beating human expectations.
> Comparing it to “attention is all you need” is a pretty aggressive comparison, and I mean it not today but in the limit (ie. if everything Nick says here is true and the research direction continues to play out as expected – let’s see when they release their technical paper). Probably pretty stupid to say conclusively now, should be read more like a prediction about the impact this direction could have.
> Solving interp from here will still be hard and is a very large/complex problem. I am no interp expert personally but there are good interp people in the comments adding nuance. But this seems like a very big update that this method is much more promising then expected and might work out.
Siddarth: im sorry wtf i feel like im going insane. This is a cool blog and exciting interp stuff (good job Nick!) but i think it’s very suspicious we have a bunch of openai people hyping it up as the greatest AI advancement this decade. I think I know why this is happening but I don’t want to speculate on the timeline too much. But in general, please update your priors responsibly people — alignment (and even interp) is FAR from solved. This is not the ok signal to accelerate RSI tomorrow
I too find it alarming that anyone could interpret this as ‘alignment can progress as fast as RSI’ levels of valuable. There was a remarkable eagerness of many, especially at OpenAI, to take it to 11 with how much this lets us pretend everything is fine. Even with the pullbacks we did see afterwards, wow.
I think mostly this is people desperate for a source of hope. You need those in life:
Dean W. Ball: My son took his first independent steps today. Between that and the interpretability stuff I’m really overwhelmed with hope, which I have not been in some time.
You also need a story that lets you keep advancing while thinking it is all going to work out. This all pattern matches very closely to continuing the cycle of ‘there I fixed it, we can get back to training the models now’ until you have something else to patch over and pretend is fine.
It is hard not to look at reactions and memes like this, and get the feel of the underlying argument (not explicitly claimed at the link) of ‘those fools thought we were doomed and alignment was unsolvable because superposition, but now that we know there is less superposition, those people are wrong, which means we will solve alignment.’ No.
At first, the hype gave many the same impression as OpenAI’s Drew Pusateri, that all the AI people were super excited and the media was once again ignoring a huge story, and Kelsey Piper had to point out that media coverage takes time. Since then, I think a lot of people have settled into a more modest position.
Anti-Hype
I think Joshua is modestly underselling this paper in terms of how much it changes parts of our understanding and option space, but even he is overselling how much this implies about reliability of prosaic safety techniques, and also is underselling the propensity for the world to apply optimization pressure.
Joshua Achiam: I think this is great and very cool! I am excited to see the insights here translate into tools with quantifiable success rates for helping us identify potential harms or steer neural networks away from risk.
A lot of people on the timeline today seem to be updating a lot over this – my own personal update is very small. This feels like one of those long-sequence-of-small-but-helpful-improvements types of papers, not something that should radically reshape our worldviews. It’s a good argument and one that could plausibly begin to constitute a reasonable prior. If correct it means that a lot of somewhat more “normal” or prosaic safety techniques will be reasonably reliable unless or until optimization pressure is applied against them.
It is however in principle possible that optimization pressure will be applied to design alien intelligences impenetrable by conventional interpretability techniques, and that these will be used against us by someone or something at some time.
There’s a picture I have for the future – an ecosystem of many different powerful AI systems, most of which are pretty good and pretty aligned and pretty helpful and pretty honest, some of which are rogue or self-sovereign, and a small number of which are actively malicious to human interests.
What the harmful ones will be able to accomplish will be limited, but not perfectly, by the good ones. Developments like good interp tools will help ensure that the good ones stay mostly good. But that there *will* exist good AIs that stay mostly good was already priced in for me, so this isn’t a huge update.
I have a lot of disagreements with the rest of the vision here but that’s another story.
Roko cautions that labeling neurons is like writing comments in code. Some will be inaccurate, so you need whatever you do to survive that. This undersells the longer term difficulties, since you will face optimization pressure, not only errors, but yes.
Nick Cammarata: i found myself in situations where i feel like i understood the network pretty well, what was happening was generally simple in a given region, and most of the struggle was trying to convert it to a hypothetical feature ontology with no way to measure it. all the confusion was going into the wrong place, i just wanted to see how the mechanisms were moving data, which kept forming reliable motifs with clear explanations with predictions that held
what actually matters to me is the motif in the car->dogs circuit i found with chris.
More good discussion follows there. I do think Nick was likely overclaiming in places.
My understanding of Nick’s theory matches Daniel Colascione. Each neuron is initiated early on in training so that it strongly activates on one clear, coherent concept. You want to focus on those concepts. Then weak activations build up around things related to the core concept in various ways.
The claim is that ~90% of neuron poles have such a primary activation, or 97% for stronger activations. About 81% are claimed to be clean on both sides.
This theory says that Nick is not denying superposition, but there should usually be some way to tie weak activations back to the central concept. Any given neuron, Nick is claiming, isn’t doing five unrelated overloaded things at once.
Thus, if there are two unrelated things going on with a neuron, and it fires equally on both, this creates a problem, especially since it is on Nick’s own example:
Brangus: A Qwen3 neuron that tracks Colorado stuff was given as an example of a neuron that seems polysemantic but makes sense once you trace it. I had Claude check what else it tracks: Tornado’s Python HTTP client code, just as hard. Can’t wait to learn how Colorado and Tornado connect!
And it didn’t fire on all code btw. The same snippet written with aiohttp did not cause the neuron to fire basically at all. Same snippet with an HTTPClient from a fake library, still basically 0. It specifically tracks “This is Tornado.”
This weirdness does leak into answers btw. If you force Qwen to pick a US state that it most associates with Tornado, it assigns 59% to Colorado. If you zero out this neuron, that drops to 15%. The Claudes don’t seem to mention Colorado at all, but I’m curious about other models.
Nick’s essay never actually said the neuron only tracks Colorado stuff, but I do think this is a pretty weird second thing for the neuron to track! Doesn’t seem very clean to me.
This cost me about 6 bucks and 15 minutes btw. Feel free to check for yourself!
Here are the other things claude found btw.
Claude actually already thought of your “ends in ‘ado'” objection and tested it. Here are the results! [no pattern] Also btw, qwen3 does not tokenize “Colorado” or “tornado” or “Tornado” the way you think it might.
Roon’s got a theory. You can play Six Degrees of Neuron Activation, but it turned out to be wrong.
roon: I have a theory and you’re not going to like it
… ok fuck it running the test is hard so let me just say the theory: tornado -> Apache license -> apaches -> American southwest -> Colorado
Brangus: is it hard to run the experiment because you have to use Astra? :P
(jk, I love astra. Is good model.)
Honestly kinda.
I tested this and it’s wrong. the neuron really just loves the tornado web framework.
We could start by accepting that neurons often have secondary weak activations that have no clear link back to the core concept.
roon: this is definitely “polysemantic”. it’s important to note this so critics don’t accuse you of drawing schizophrenic connections to support any theory. but if you read Nick’s paper he claims that some qwen neurons have a second unrelated smaller peak; that is still a much better interp world than we lived in last week, where we thought neurons could be an arbitrary mismash of superposed concepts. hell even if there are 3 or 5 we can read the stack of meanings like Lyra’s alethiometer
few interesting things about this neuron:
– it really likes the Tornado web framework but doesn’t care about tornadoes
– it responds weakly even to fictional software frameworks called Tornado
– it gets somewhat less excited by arbitrary web frameworks
– there is no connection to Ipython, which was invented in Colorado
– there is no connection to Apache, a software license but also a tribe that lives in the American southwest
– -ado connection doesn’t exist, “el dorado” does nothing
alethiawiz: it’s best to just wait for nick to release a technical paper.
roon: No it’s best to start your own research program.
alethiawiz: you’re right.
The problem is that in this case, Brangus is saying the Colorado neuron fires ‘equally strongly’ on Tornado activations. And there is quite a lot of leakage between the concepts. If this is a typical result, then the whole paper won’t hold up that well.
The neuron meant as an example of a centrally explained neuron has at least two central explanations. That is a very bad sign.
Is This How Minds Work?
Almost everyone, including Eliezer Yudkowsky (who posted in a bunch of different threads, and whose comments I am moving into what I consider a more logical order), thinks this is a non-zero amount of Actual Progress, if the central findings hold up.
One objection is that some think the paper will not hold up.
The other objection is, even if the paper holds up, this is merely a beginning that makes it easier but still difficult to move forward.
The paper’s central claim is not to have found something new, but rather to have found the absence of things both old and new. The AI’s neurons look simpler, not more complicated. This leaves the rest of its behavior still unexplained.
Eliezer Yudkowsky: Right now they’re at the stage of ‘explaining where Coca-Cola comes from or how it gets made’ of… tracing how a quarter falls through a Coke vending machine before the Coke falls out.
Noticing that the AI is thinking about Navier-Stokes is very different from understanding the part where the AI can prove Navier-Stokes, and no human-written program can do so, because no human knows what kind of thought processes would be required.
See this Manifold market [which is still at 8%, asking whether we will have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006, which was persistently over 50% back in 2024.]
Eliezer Yudkowsky (comment on the manifold market from 2023): This market is not about identifying semantics inside an LLM. It is not about identifying semantics inside an LLM which we did not previously know to be inside LLMs. This market is about identifying semantics inside an LLM such that those semantics, once uncovered, teach us something about semantic representations in general which we did not know in 2006.
But We Can Open The Box, Said Toad
There is another key point Eliezer Yudkowsky emphasizes: If you have a ‘trying to do math’ detector via this new technique, that can be useful, but it does not help you understand how the model does math. Even if you know what each neuron does within math, this is remarkably little of the puzzle.
Even if we now have the opportunity to open the box and look inside and know the meaning of individual neurons, that does not tell you how the box fits together, and you are running out of time before the AI is out of the box.
Eliezer Yudkowsky (replying to Roon’s claim tup top hat this is a big freaking deal): If you think there is any possible level of interpretability that makes “align ASI” be “normal science” rather than “maaaybe possible” then you do not understand what ASI is or don’t believe in it.
As you know, I now and then post that some research development is unexpectedly good news. But wow, you sure are not selling me on the research or your sanity by proclaiming that it’s gonna make alignment be normal.
Okay, so now I’m looking at there and it’s “polysemanticity and superposition convinced me interpretability was hard, but with big networks there’s single neurons after all” and do. do you have any concept of what utterly. fucking. primitive. playgrounds you are still playing around in. “We thought we might have a hard time finding single neurons that light up for the Eiffel Tower and would have to look at *multiple* neurons, but now maybe we can find single neurons again!”
Do you have any fucking clue of how close this is to the absolute first beginning of understanding how a mind is thinking? Do you think mechanistic interpretability at the level of “do alignment” is about noticing that particular kinds of concepts fired?
To be clear, I did not spot in my skim of the post any grandiose claims about making alignment be normal and my headdesking here is about Roon and not about the researchers. I continue to support interpretability as a field. I would like people to stop overstating where it got.
j⧉nus: He’s right, actually, but I think it’s okay that we’re so far from the fantasy of total reductionist understanding. And it’s good that there is progress, even though it’s far, and other kinds of progress are more important.
Eliezer Yudkowsky: People in 2014: Why aren’t you SUPER EXCITED about interpretability?
Me: Well I yelled at OpenPhil that if they could spend $1B on it they should, but they apparently can’t, which favors my expectation that interpretability just runs much much too slowly to make ASI alignment work.
AI guys: We built a SCREWDRIVER! Now we are going to pry the panel off the computer tower and look inside at all the wheels and gears so we can see how computers work!
Me: I think that you having built the screwdriver so you can *start* looking at things is not as much progress toward the end goal as you seem to believe. It reads as sheer naivete about how much work there is to do, because you don’t concretely know about any of the things you need to figure out. It’s like saying that neuroscience is gonna be solved as soon as you figure out how to look at a neuron with a microscope. This attitude of yours does not inspire confidence.
AI guys: Our SCREWDRIVER BROKE! Neurons do more than one thing! More than one neuron does a thing!
Me: Uh huh.
AI guys in 2026: We FIXED THE SCREWDRIVER! Panel’s coming off the computer tower! Alignment is AS GOOD AS SOLVED!
Crowd: What excellent news! Will you not join us in celebration?
Eliezer Yudkowsky: And it *would* be cool to know how the machine worked. But hearing back a couple of years later that they’ve repaired the screwdriver they were gonna use to maybe get started on looking so they could get started on understanding, does not make me believe the world is now saved.
@ben_r_hoffman: This is in agreement with a strict literal construal of Roon’s tweet. He says your pdoom will be lowered, not low, and that this is the beginning and not the end of making alignment a normal science, and you agree that fixing the screwdriver is an improvement.
The crowd is correct that this is excellent news. It is vastly better to have a screwdriver, even at a late hour, than to not have one. That does not mean anything is ‘as good as solved’ or on track to getting there, or that solving the puzzle would then solve the alignment problem.
Eliezer Yudkowsky:
Me: My pessimism was not based on predicting they’d never get this far. It is based on how much work I believe remains from here.
Idiots: Behold, Yudkowsky NEVER FAVORABLY UPDATES!
Me: Demonstrably false. Here’s me favorably updating in 2025 when the news of a central good-evil axis came out. Are you going to change any of your opinions now?
Idiots: But Yudkowsky doesn’t hugely update when *I* hugely update! No doubt this is… because… Yudkowsky is stupid! Yes, that is undoubtedly the explanation.
Me: It’s just not surprising that they got this far. Like so many other things you are running around waving your arms frantically about having just seen, I already lived through this moment a thousand times in prescience.
Idiots: VERY STUPID!
On the one hand, yes this is strawmanning. On the other hand, why are half the men around me made of straw? So, sigh.
By default, we get an explanation of what an AI just did, or what concepts it was thinking about and how they relate, but without being able to predict outputs.
Eliezer Yudkowsky: Me: Mechanistic interpretability sounds like a good idea, yes. If we could learn something about how an AI was thinking, in enough detail to predict in advance what it would think even as the thought departed from human, that would be a bare beginning on the sort of understanding you’d needed to get started on AI alignment that wasn’t just about hammering on an AI from the outside with SFT and RL.
AI guys: We found a neuron that lights up when the AI thinks of the Eiffel Tower!
Me: Cool. That does sound like it’d be a prerequisite to getting started on the important part eventually, someday.
AI guys: Oh NO! There are MULTIPLE neurons that light up when the AI thinks about the Eiffel Tower! And some of those neurons also participate in OTHER concepts! We’re calling this unexpected development ‘polysemanticity’ and ‘superposition’.
Me: That sure sounds like a thing that would happen, yes. I cannot say that I am impressed by you acting surprised about it. Let me know when you’ve gotten far enough in mechanistic interpretability to get started on the important part.
AI guys: We have found that with BIG AIs you can often point to a SINGLE neuron that lights up when the AI thinks of the Eiffel Tower! Understanding AI thought processes is gonna be EASY PEASY!
Me: Cool. Let me know when you’ve advanced to the point of getting started.
Eliezer would be hopeful about interpretability if it allowed us to start making predictions. Until then, he thinks it is not worth so much. I see his point.
That Was Not the Hard Problem
This is another way of making Eliezer’s central point: Enumerating the neurons, even if you can do it, is a small fraction of the overall problem. It was not the main barrier. Taking down or lowering difficulty of any barrier is great, but only goes so far. Okay, so there is a lot less intense superposition, now what?
Jake Mendel: After reading for 15 mins: I think Nick is operating in a world where interpretability was just about enumerating the neurons and superposition was an obstacle to that plan, so now he’s happy because maybe superposition isn’t real. But:
1) there are v few empirical results. Unfortunately interp has a long history of significantly overclaiming how interpretable their findings are (SAEs, early circuits having poor causal scrubbing scores etc) so I’m not updating much at all on the evidence he’s already shown. it’s really got a long history of people getting overexcited. So looking forward to seeing the paper with all the empirical evidence!
2) I think it’s not that big news for me even if true because superposition in something like the average case wasn’t my main crux to thinking interpretability is tractable. some cruxes this doesn’t really deal with at all:
– I think the correct ‘circuit’ is usually just an overwhelmingly large bag of random-seeming heuristics (a la Redwood Research’s interpretation of how an LLM predicts is vs was) and having it isn’t very informative/doesn’t help with much/doesn’t lead to good predictions.
– More generally, It seems possible to enumerate all the ‘low level’ features in a neural network, and still have a hard time answering ‘high level’ questions like: “Is the model misaligned? / What would it do in this new situation? / If it is misaligned what should we do about it?” there’s an abstraction/emergence gap to cross from the features to the questions of interest, and crossing that gap is likely much harder than enumerating features.
That said, if his claims hold up, I think it will be quite a big deal, and a useful precursor to tackling the thornier problems! There may also be some useful things to do without tackling the thorny problems: eg we’ll be able to make use of masses of automated agents to scour the features and circuits to try and make sense of the mess, and maybe that’s good for something. although I suspect that even the most optimistic outcome here will not make alignment of superhuman AIs much easier (as some on twitter have claimed) because the neuron interpretability will not be adversarially robust, among other issues
More Skepticism
My overall view is remarkably similar to Neel Nanda’s here, although I am modestly more optimistic about this mattering. Looking at individual neurons for primary factors will be useful for some purposes, but you are still missing a lot of what is happening, there will be errors, and this is not altering the big picture by so much.
Neel Nanda: This is interesting work, but I don’t think it’s a significant update to the big picture of interpretability.
I’m on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders.
I do not agree with some people’s grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what’s going on, but I remain pessimistic about really fully understanding things and being confident there’s nothing we’re missing.
… I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on.
For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons.
And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons.
You should be able to store a lot more facts via key-value pairs than you have neurons because neurons each have a lot of associated values? Method seems obvious, although this does not invalidate Neel’s central point. I’m torn on Neel’s later (unquoted here) claim that most memorization is de facto ‘random data.’
I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what’s going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less).
This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!
Similarly, my impression is that SAEs generally don’t find things that are a
single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.
And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.
Nick responded directly, including claiming weak activations are ‘surprisingly ablateable,’ and they went back and forth.
You also can and should start out skeptical on priors. Most hyped results are hyped.
Samuel Albanie (Google DeepMind): on base rates, I’m a little skeptical of big positive updates on mech interp. On the other hand, orders of magnitude more ai labour will be coming online in the near future, so the Hamming adage applies:
“In almost all fields, a factor of ten means fundamentally new effects. If you increase magnification by a factor of 10 in Biology, you will see new things.”
so it doesn’t seem implausible that scaling the simple approach will work. will be watching closely.
The bull case is a combination of ‘we could not do this at scale before and now we can’ with ‘it turns out the neurons mostly have singular explanations, so we can do it at all,’ bitter lesson, go. Seems plausible, so long as you do not get overexcited.
That’s All There Is, There Ain’t No More
If minds are largely neurons representing specific central concepts, that then have associations with other concepts in these atomic fashions, one might ask ‘is that all?’
Zvi Mowshowitz: Still reading through this and processing. I’m less struck by the ‘now we can do interpretability easier’ part – you still have to understand what you see – and more by the ‘so you are saying this is all there is to intelligence?’ part. That’s it?
Elijah Ravitz-Campbell: I feel like this very much matches how I imagined AI might work before I knew anything about it in practice. A sort of hand wavy “you know concepts have nodes and relationships between those nodes in code semantic knowledge”.
roon: Yeah I’m surprised so few are grappling with the philosophical implications of all this
antra: I don’t think that’s it, and I think this is kind of why SD is a lot more interpretable than transformers. This applies to some degree to parts of perceptive, semantic and motoric representations of language models, but not as much to compositional, integrative and temporal ones.
I am also quite sus of weak activation dismissal on strongly recurrent networks like transformers. We saw weak activations matter strongly specifically in the temporal/recureent domain, which is ignored by this paper.
Does this mean that perhaps a dumber mind can understand a smarter one, if all the components are understandable, the same way we can understand complex biological systems, or the components of complex systems or organizations? Certainly you can ‘understand’ something that has advantages over you. If the AI is faster than you, or can memorize more things, it is easy to see how you can understand and even predict it. If intelligence were sufficiently a pure ‘more simple things combined’ in some sense, then you might have a shot when moving at slow speed? That likely presumes that the ‘real action’ is in the neurons, or other places we can parse. We should not assume that.
Can we even simulate a neuron?
Leo Gao (OpenAI): this is cool work, but i think people are overupdating on how optimistic to be about interp.
in particular, we need to more rigorously evaluate explanations, rather than just going off vibes. imo if you can’t simulate a neuron you haven’t really explained it. my guess is simulating the neurons is way harder than finding plausible explanations. I have a post on this in the works.
(i know the blog talks argues simulations are bad, but i think all of the downsides described are avoided by looking at KL instead of neuron correlation.)
Nick: would love to read it when it’s ready. I haven’t thought enough about simulation scores to be super confident, and could somewhat easily be convinced back
My understanding is that previously we only found ~1.7% of neurons interpretable back in 2023 (Bills et al) but that the test for this required predicting activations. Nick is not yet trying to predict activations.
The Default Outcome of Mech Interp is Goodhart’s Law
What happens if we apply optimization pressure to which neurons light up?
Depending on strong or direct the optimization pressure gets, that could be The Most Forbidden Technique.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
Those bits of optimization pressure from [T] are precious. Use them wisely.
We must continue to never, ever train on the interpretability, and to watch out for other indirect ways we are doing so. There will be crazy amounts of pressure to pull out forms of The Most Forbidden Technique.
Simon Lermen: Fundamentally, trying to automate alignment research points an objective at “convince the human” for the AI to optimize for. If this plan was to be taken even a little serious you’d expect the AI labs to have very seriously thought about solid metrics and benchmarks that aren’t just going to get goodhearted immediately. Instead we will likely see a last-minute vibe-coded “alignment solution” born from the lethal arrogance of the AI lunatics.
I don’t think this is ‘lethal arrogance of the AI lunatics.’ I think it is that the AI creators are doing the best they can given that they are under extreme time pressure, so everything is always last-minute vibe coded. Once you vibe code it then that becomes the last minute.
My baseline scenario is that the labs will not do the maximally stupid direct optimization thing, but that some others will, and that the labs will end up doing a bunch of indirect optimizations that exert pressure. How much damage results from that pressure will remain unclear.
One way I would expect AIs to adapt to this pressure is to learn how to think about [X] without triggering the direct neurons or detectors for [X]. Do not rely on your detectors in ways that presume future AIs won’t learn to get around them.
We already see places the AI could hide such thinking. There are 2%-5% of unexplained polls, and weak activations look like they could be cheaply ablated and thus are available for other things, plus we don’t understand the cache or attention.
Interpretability is Dual Use
One must also remember that essentially everything in alignment is dual use.
Roon: pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was “slouching towards monosemanticity”. my guess would be that major breakthroughs in interp also advance capabilities
Nathan: If ‘positive’ updates like this are, as roon says, likely to increase capabilities, then funding safety work has an unclear effect on safety.
Interpretability is even more dual use than usual. I still strongly support working on it, but very obviously if you can figure out what the neurons do, or better yet how the whole operation works and thinks, you can use that to increase capabilities. A lot of my first thoughts were how to use this information to improve training, which could then be spent on any combination of seeking alignment or seeking other things.
Now The Work Can Begin
What does this all add up to?
Partly that will boil down to how much of this holds up. We have not yet seen the technical paper.
I think this news represents a substantial update in the difficulty of neuron interpretability, and in the ways in which LLMs organize their information and thoughts. This should make it much easier to attack interpretability through a neuron analysis lens than previously thought, and open up a lot of good research and understanding.
I do not think this mean interpretability, let alone alignment, is on its way to being solved. We should be pessimistic about how far such claimed breakthroughs go, the same way SAEs are useful but not as useful as some hoped. Even if they are fully real and work out about maximally well, this is the beginning of the work, not the end, and many things are missing.
We still need to know what is up with weak activations, with the cache and with attention. We still will have little idea how the AIs are thinking, or why they are able to do the breakthroughs and impressive feats. We will not have any guarantee that, if in a hostile situation, our techniques would hold, especially against future AIs, even before we apply optimization pressure over time.
We also need to worry the result largely will not hold up. The Colorado-Tornado result suggests that the whole thing might be a lot less clean than it appears.
I updated a substantial negative amount from the eagerness to jump on the hype train. It pattern matched quite a lot with ‘we will once again pretend we will have everything handled and therefore only invest in these prosaic plans right until we run into another wall and realize this is insufficient.’ We need to fight hard against that.
This is still excellent news. The world is more hopeful now. Let’s get to work.
Table of Contents
Good Smoke, Potential Fire
Before I cover reactions or take a side, here are my highlights. It is worth reading at least this much of the primary source. Reading the whole thing is an option, but it is long and the writing can be easy to bounce off of and can repeat itself.One Neuron, One Central Concept
The central claim is that for larger LLMs, most neurons represents a singular concept and things clustered around that concept, perhaps with small offshoots and some use of low-level activations. Where previously the working theory was that models overloaded neurons with unrelated concepts in superposition, Nick’s theory is you did not think sufficiently hard to identify the unifying concept. Each neuron is a generalization, mostly ones we can readily understand if we look. This pictures intelligence as brute force. Neurons are compound words in German. New concept? Go get another neuron. And yet, it works.I Can See Everything
Nick argues that this means that it is easier to solve the full puzzle and understand everything, and then take a measurement of your specific area of interest, then to try to directly pull a fuzzy concept like ‘deception’ out of a pool of mystery neurons. This claim is maximal and bold, and will get directly challenged. Most people seem to have updated from this paper that more neurons have central singular concepts behind them than we previously thought. Nick is trying to go further, and say that this is what is going on universally.What a Concept
Remember this the next time you think you can never figure anything out, or that surely people have already asked the basic questions and considered the obvious. Maybe they have. Maybe they haven’t. Nick does notice the skulls that result from going full ‘This Is Not a Coincidence Because Nothing Is Ever a Coincidence.’ The strong hypothesis is that neurons will all have central concepts like Colorado. The even stronger hypothesis is that there will not also be unrelated additional concepts happening behind the scenes.The More You Know
The core theory is that perhaps intelligence is a simpler concept than we think? It is all a series of coherent concepts with links? There’s a ‘Hitchhiker’s Guide state unit that peaks on the title and the Answer slot’ that weakly fires on all 42s? There is a neuron specifically for the joke about fusion always being a fixed amount into the future that fires on other similar repeated promises?Attention Is Still What You Need
Then there’s the residual stream and attention. The essay feels like it’s trying to hide how much of what is going on is happening outside of the neuron labels. This is one place Nick seems to overreach. Yes, we can look at where attention guides you without knowing what mechanism allocates your attention, and then look at the neurons that get triggered. I don’t think this ‘gets you out of’ having to understand that process. It is worth looking at the neurons anyway. This does not allow you to predict future LLM outputs or fully understand how it thinks. Does this mean you can ‘sit on the neurons that matter’ to guard against shenanigans or otherwise play good defense, if you don’t make more progress? To some extent, but I would assume that an LLM that knows about this type of monitoring would be able to work around this, and figure out how ‘not to think about’ what you look for.Occam’s Razor
The theory of why all the neurons this is simple: That this is simple.The Neurons Sometimes Share Clear Themes
Sauers notices patterns of overlap of neurons between different features in Qwen3-8B. This seems highly consistent with Nick’s theory.Hype
We saw some hype up top. Here is some more hype. This is exactly the mistake Eliezer Yudkowsky will be ranting about later in this post. Yes, this is progress. No, this is not an ‘MRI for AI.’ That is still a long way off. Even if you think this is a big deal, some of the hype got crazy. I too find it alarming that anyone could interpret this as ‘alignment can progress as fast as RSI’ levels of valuable. There was a remarkable eagerness of many, especially at OpenAI, to take it to 11 with how much this lets us pretend everything is fine. Even with the pullbacks we did see afterwards, wow. I think mostly this is people desperate for a source of hope. You need those in life: You also need a story that lets you keep advancing while thinking it is all going to work out. This all pattern matches very closely to continuing the cycle of ‘there I fixed it, we can get back to training the models now’ until you have something else to patch over and pretend is fine. It is hard not to look at reactions and memes like this, and get the feel of the underlying argument (not explicitly claimed at the link) of ‘those fools thought we were doomed and alignment was unsolvable because superposition, but now that we know there is less superposition, those people are wrong, which means we will solve alignment.’ No. At first, the hype gave many the same impression as OpenAI’s Drew Pusateri, that all the AI people were super excited and the media was once again ignoring a huge story, and Kelsey Piper had to point out that media coverage takes time. Since then, I think a lot of people have settled into a more modest position.Anti-Hype
I think Joshua is modestly underselling this paper in terms of how much it changes parts of our understanding and option space, but even he is overselling how much this implies about reliability of prosaic safety techniques, and also is underselling the propensity for the world to apply optimization pressure. I have a lot of disagreements with the rest of the vision here but that’s another story. Roko cautions that labeling neurons is like writing comments in code. Some will be inaccurate, so you need whatever you do to survive that. This undersells the longer term difficulties, since you will face optimization pressure, not only errors, but yes.Theme and Variations
Liv Gorton argues that Nick misunderstands superposition. Superposition is, to Liv, a long-standing, well-defined claim that there are more features in a layer than dimension, via almost orthogonal directions, which is still true. There’s still plenty of complexity here. Nick responds that he suspects the difference is that he doesn’t think it is useful to work with ‘features.’ More good discussion follows there. I do think Nick was likely overclaiming in places. My understanding of Nick’s theory matches Daniel Colascione. Each neuron is initiated early on in training so that it strongly activates on one clear, coherent concept. You want to focus on those concepts. Then weak activations build up around things related to the core concept in various ways. The claim is that ~90% of neuron poles have such a primary activation, or 97% for stronger activations. About 81% are claimed to be clean on both sides. This theory says that Nick is not denying superposition, but there should usually be some way to tie weak activations back to the central concept. Any given neuron, Nick is claiming, isn’t doing five unrelated overloaded things at once. Thus, if there are two unrelated things going on with a neuron, and it fires equally on both, this creates a problem, especially since it is on Nick’s own example:Is This How Minds Work?
Almost everyone, including Eliezer Yudkowsky (who posted in a bunch of different threads, and whose comments I am moving into what I consider a more logical order), thinks this is a non-zero amount of Actual Progress, if the central findings hold up. One objection is that some think the paper will not hold up. The other objection is, even if the paper holds up, this is merely a beginning that makes it easier but still difficult to move forward. The paper’s central claim is not to have found something new, but rather to have found the absence of things both old and new. The AI’s neurons look simpler, not more complicated. This leaves the rest of its behavior still unexplained.But We Can Open The Box, Said Toad
There is another key point Eliezer Yudkowsky emphasizes: If you have a ‘trying to do math’ detector via this new technique, that can be useful, but it does not help you understand how the model does math. Even if you know what each neuron does within math, this is remarkably little of the puzzle. Even if we now have the opportunity to open the box and look inside and know the meaning of individual neurons, that does not tell you how the box fits together, and you are running out of time before the AI is out of the box. The crowd is correct that this is excellent news. It is vastly better to have a screwdriver, even at a late hour, than to not have one. That does not mean anything is ‘as good as solved’ or on track to getting there, or that solving the puzzle would then solve the alignment problem. On the one hand, yes this is strawmanning. On the other hand, why are half the men around me made of straw? So, sigh. By default, we get an explanation of what an AI just did, or what concepts it was thinking about and how they relate, but without being able to predict outputs. Eliezer would be hopeful about interpretability if it allowed us to start making predictions. Until then, he thinks it is not worth so much. I see his point.That Was Not the Hard Problem
This is another way of making Eliezer’s central point: Enumerating the neurons, even if you can do it, is a small fraction of the overall problem. It was not the main barrier. Taking down or lowering difficulty of any barrier is great, but only goes so far. Okay, so there is a lot less intense superposition, now what?More Skepticism
My overall view is remarkably similar to Neel Nanda’s here, although I am modestly more optimistic about this mattering. Looking at individual neurons for primary factors will be useful for some purposes, but you are still missing a lot of what is happening, there will be errors, and this is not altering the big picture by so much. You should be able to store a lot more facts via key-value pairs than you have neurons because neurons each have a lot of associated values? Method seems obvious, although this does not invalidate Neel’s central point. I’m torn on Neel’s later (unquoted here) claim that most memorization is de facto ‘random data.’ Nick responded directly, including claiming weak activations are ‘surprisingly ablateable,’ and they went back and forth. You also can and should start out skeptical on priors. Most hyped results are hyped. The bull case is a combination of ‘we could not do this at scale before and now we can’ with ‘it turns out the neurons mostly have singular explanations, so we can do it at all,’ bitter lesson, go. Seems plausible, so long as you do not get overexcited.That’s All There Is, There Ain’t No More
If minds are largely neurons representing specific central concepts, that then have associations with other concepts in these atomic fashions, one might ask ‘is that all?’ Does this mean that perhaps a dumber mind can understand a smarter one, if all the components are understandable, the same way we can understand complex biological systems, or the components of complex systems or organizations? Certainly you can ‘understand’ something that has advantages over you. If the AI is faster than you, or can memorize more things, it is easy to see how you can understand and even predict it. If intelligence were sufficiently a pure ‘more simple things combined’ in some sense, then you might have a shot when moving at slow speed? That likely presumes that the ‘real action’ is in the neurons, or other places we can parse. We should not assume that. Can we even simulate a neuron? My understanding is that previously we only found ~1.7% of neurons interpretable back in 2023 (Bills et al) but that the test for this required predicting activations. Nick is not yet trying to predict activations.The Default Outcome of Mech Interp is Goodhart’s Law
What happens if we apply optimization pressure to which neurons light up? Depending on strong or direct the optimization pressure gets, that could be The Most Forbidden Technique. We must continue to never, ever train on the interpretability, and to watch out for other indirect ways we are doing so. There will be crazy amounts of pressure to pull out forms of The Most Forbidden Technique. I don’t think this is ‘lethal arrogance of the AI lunatics.’ I think it is that the AI creators are doing the best they can given that they are under extreme time pressure, so everything is always last-minute vibe coded. Once you vibe code it then that becomes the last minute. My baseline scenario is that the labs will not do the maximally stupid direct optimization thing, but that some others will, and that the labs will end up doing a bunch of indirect optimizations that exert pressure. How much damage results from that pressure will remain unclear. One way I would expect AIs to adapt to this pressure is to learn how to think about [X] without triggering the direct neurons or detectors for [X]. Do not rely on your detectors in ways that presume future AIs won’t learn to get around them. We already see places the AI could hide such thinking. There are 2%-5% of unexplained polls, and weak activations look like they could be cheaply ablated and thus are available for other things, plus we don’t understand the cache or attention.Interpretability is Dual Use
One must also remember that essentially everything in alignment is dual use. Interpretability is even more dual use than usual. I still strongly support working on it, but very obviously if you can figure out what the neurons do, or better yet how the whole operation works and thinks, you can use that to increase capabilities. A lot of my first thoughts were how to use this information to improve training, which could then be spent on any combination of seeking alignment or seeking other things.Now The Work Can Begin
What does this all add up to? Partly that will boil down to how much of this holds up. We have not yet seen the technical paper. I think this news represents a substantial update in the difficulty of neuron interpretability, and in the ways in which LLMs organize their information and thoughts. This should make it much easier to attack interpretability through a neuron analysis lens than previously thought, and open up a lot of good research and understanding. I do not think this mean interpretability, let alone alignment, is on its way to being solved. We should be pessimistic about how far such claimed breakthroughs go, the same way SAEs are useful but not as useful as some hoped. Even if they are fully real and work out about maximally well, this is the beginning of the work, not the end, and many things are missing. We still need to know what is up with weak activations, with the cache and with attention. We still will have little idea how the AIs are thinking, or why they are able to do the breakthroughs and impressive feats. We will not have any guarantee that, if in a hostile situation, our techniques would hold, especially against future AIs, even before we apply optimization pressure over time. We also need to worry the result largely will not hold up. The Colorado-Tornado result suggests that the whole thing might be a lot less clean than it appears. I updated a substantial negative amount from the eagerness to jump on the hype train. It pattern matched quite a lot with ‘we will once again pretend we will have everything handled and therefore only invest in these prosaic plans right until we run into another wall and realize this is insufficient.’ We need to fight hard against that. This is still excellent news. The world is more hopeful now. Let’s get to work.