# Comments: Linkpost: Neural Networks Are Already Clean

Post URL (Markdown): [/api/post/linkpost-neural-networks-are-already-clean](/api/post/linkpost-neural-networks-are-already-clean)

Post URL (HTML): [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean)

Showing 23 of 23 comments (sort=top).

For reaction user names, use `?includeReactionUsers=1`.

### Comment by [Neel Nanda](/users/neel-nanda-1)

*   2026-10-16 00:54:58Z
*   Karma: 91
*   Voting system: namesAttachedReactions
*   Approval votes: 35
*   Total votes: 35
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/7wcSqBNzQF2o9SHyY](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/7wcSqBNzQF2o9SHyY)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/7wcSqBNzQF2o9SHyY](/api/post/linkpost-neural-networks-are-already-clean/comments/7wcSqBNzQF2o9SHyY)

	*[Crossposting from my Twitter](https://x.com/NeelNanda5/status/2110875974296654062)*

	> This is interesting work, but I don't think it's a significant update to the big picture of interpretability.

	> I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
	Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

	> I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.

	> I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!

	> Similarly, my impression is that SAEs generally don't find things that are a
	single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.

	> And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.

### Comment by [Alex Gibson](/users/alex-gibson)

*   2026-10-15 23:15:23Z
*   Karma: 51
*   Voting system: namesAttachedReactions
*   Approval votes: 29
*   Total votes: 29
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/yTi9dKZawGmqmbZo2](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/yTi9dKZawGmqmbZo2)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/yTi9dKZawGmqmbZo2](/api/post/linkpost-neural-networks-are-already-clean/comments/yTi9dKZawGmqmbZo2)

	Irrespective of the contents of the blog post itself, which would be a quite interesting post in a vacuum, I find the discourse surrounding it pretty concerning.

	From Nick's X linkpost: "The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most"

	From the OG SAE "Towards Monosemanticity" Blog Post: "it increasingly seems like a large chunk of the mechanistic interpretability agenda will now turn on succeeding at a difficult engineering and scaling problem, which frontier AI labs have significant expertise in"

	SAEs did not solve interp, and I will bet at absurd odds that this blog post is not going to come close to solving ambitious interpretability in a concrete alignment relevant sense.

	The fact that the interp posts which absurdly overclaim like this do so well among outsiders to the field (Many are now using this overclaim to present Eliezer as someone who just won't update on new evidence, and even some Pause people are talking about updating positively) is a reason i'd argue that unfortunately interp has been net negative (if only fractionally.)

	The lay audience generally don't have the ability to critically interrogate a post like this, because of its long and technical nature, which imo means interp people have a duty to be responsible/measured in how they present their work.

### Comment by [Julian Bradshaw](/users/julian-bradshaw)

*   2026-10-15 18:21:36Z
*   Karma: 35
*   Voting system: namesAttachedReactions
*   Approval votes: 19
*   Total votes: 19
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/dXJkLskyhmvEH8Qis](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/dXJkLskyhmvEH8Qis)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/dXJkLskyhmvEH8Qis](/api/post/linkpost-neural-networks-are-already-clean/comments/dXJkLskyhmvEH8Qis)

Reactions (whole comment):

*   thanks: 5

	Adding some context and takeaways from my initial reading as I'm fairly excited about this:

	1.  We might expand the title to "Superposition was possibly mistaken as an explanation for polysemanticity; neural networks are possibly much easier to interpret than previously thought".
	    1.  "Polysemanticity" is basically just "a single neuron can encode many different concepts".
	    2.  To explain superposition, I'll use [his explanation](https://already-clean.goodfire.com/#:~:text=The leading explanation for polysemanticity was superposition): "The leading explanation for polysemanticity was superposition. This proposed a new learning principle: when a neural network runs out of neurons, it stops grouping related things together and instead begins folding unrelated things together, simply because they don’t interfere. The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing."
	        1.  Side note 1: technically "don't interfere" might be better stated as "don't interfere too much".
	        2.  Side note 2: arguably this definition of superposition is a narrow one, specifically about *unrelated* concepts being grouped together. Neurons still do group *related* concepts together; that is well-established (a neural network can definitely encode more concepts than it has neurons), and he goes on to theorize more on how this happens.
	        3.  For more on definitions, see the [appendix bit of the post here](https://already-clean.goodfire.com/#features-and-definitions).
	    3.  In the blog post, he works through previous claimed examples of superposition (including his own!) and argues that these were more the result of bad early architectures, weak tools, and unrecognized generalization.
	    4.  In general, Nick's thesis is that polysemanticity is more caused by what he calls "recursive generalization" than by superposition of unrelated concepts.
	        1.  [Quoting](https://already-clean.goodfire.com/#:~:text=call recursive generalization.-,It’s simple,-: as a neuron): "It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and then “do it yourself,” which doesn’t contain “iy” at all. (...) It’s still generalization, but it turns out generalization is fractal. That’s far less concerning: there’s something to learn from the way things were grouped, and nothing to learn from two items in superposition. And since the generalization itself is fractal, our explanations can be too: start with a basic explanation, then zoom in when you want to understand more."
	        2.  Working through Qwen3-32B his preliminary results are that [~90% of neuron poles have one main job](https://already-clean.goodfire.com/#:~:text=90% of neuron poles have one main job). A further ~7% only have two, ~3% are still not understood. Very promising!
	            1.  "neuron poles" is a nuance here, an idea that neurons have "positive" and "negative" poles, aka what activates a neuron in a strongly positive way vs. what activates it in a strongly negative way. He posits that looking at these separately is useful for understanding neurons.
	            2.  There's also an interesting posited nuance about neurons encoding nouns vs. verbs. [Quoting](https://already-clean.goodfire.com/#features-and-definitions:~:text=Both human and automated interpreters): "Both human and automated interpreters seem strongly biased toward describing neurons as detectors for groups of nouns, like cats and cars, rather than for a shared pattern, like a V. Sometimes a neuron really does respond to a group of nouns. In my experience, though, it’s more often a shared pattern, and the neuron is better understood as a verb than a noun. Its job isn’t to name what’s in the input but to draw a distinction downstream neurons can use: it separates two subtle patterns at its positive and negative poles (a V versus an O) and pushes the flow of activations toward different neurons accordingly."
	                1.  See examples [here](https://already-clean.goodfire.com/#:~:text=spoonbills from flamingos).
	        3.  He claims recursive generalization is much easier to see if you watch as the model is trained and the neuron learns more concepts; just working from a finished model, as in some previous experiments, makes it harder to see how the generalization happened, how ex. "iy" is related to "do it yourself".
	            1.  Training small models can be made an experiment to see if polysemanticity is caused by unrelated superposition or recursive generalization, and he's seeing the latter.
	            2.  He [also tested reducing neuron count](https://already-clean.goodfire.com/#:~:text=So we can test what happens to neurons) to see if it caused superposition, and it didn't (in image models anyway).
	        4.  There's also a more general hypothesis here: "clean" neural networks are in some sense *natural* because of optimization pressures toward legibility to other parts of network. [Quote](https://already-clean.goodfire.com/#:~:text=The network is enormous): "The (neural) network is enormous, but it may already have done almost all the work of preparing itself to be understood, not for us, but for its own readers. If we can become another reader, it’s ready for us too."
	            1.  If true (and this is more hypothesis than established by experiment) it would be a strong reason to expect large LLMs to be as interpretable as smaller ones, or even more so!
	                1.  Worth noting that in a somewhat different test on GPT-2 XL (older and smaller than Qwen3-32B, just 1.5B params), only [~70% of neurons are "roughly one thing"](https://already-clean.goodfire.com/#:~:text=Now about 70% look like roughly one thing), so the apparent trend is increasing friendliness to interpretability, and he [claims](https://already-clean.goodfire.com/#:~:text=in general, smaller models have broader neurons, with more residual generalization than modern models): "in general, smaller models have broader neurons, with more residual generalization than modern models". However needs more rigorous evaluation.
	    5.  This definitely leaves open a lot of room for further work. I think we'll probably see a flurry of followup work on this.
	2.  Nick Cammarata has [worked in interpretability for many years](https://scholar.google.com/citations?user=VNoyvKgAAAAJ&hl=en), and was at OpenAI for some time ex. was working with Chris Olah on the interpretability team there before Olah left to co-found Anthropic. Has lately been at [Goodfire](https://www.goodfire.com/), a premier interpretability lab, since 2023 as far as I can tell. It's believable that he's made a major breakthrough. (edit: not that breakthroughs can't come from anywhere!)
	3.  In particular, Nick phrases the nature of this claimed breakthrough [as](https://already-clean.goodfire.com/#:~:text=there’s a bitter lesson for interpretability too) "there’s a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition."  
	      
	    In other words, the nature of his research here is in significant part a ["more dakka"](/api/tag/more-dakka) approach. For example, the blog post includes [1000 analyses of neurons in Qwen3](https://already-clean.goodfire.com/#neuron-scientist-gallery:~:text=neurons, chosen randomly.-,Neuron papers,-1,000 · Qwen3-32B)!  
	      
	    Some curious ones in there, for example, [a neuron](https://already-clean.goodfire.com/#paper-L54_N16863) whose positive pole has to do with February's Leap Day (with some recursive generalizations to ex. this one time in 1712 Sweden had a Feb. 30th!), and whose negative pole has to do with surnames in sports team rosters and also with digits in sports scores. It categorizes the positive pole as having 1 job, and the negative pole as having 2 related jobs. There isn't an obvious connection between the two poles though, and they seem to have no real shared circuitry apparently.
	4.  The linked post is "only" a gigantic, interactive research blog post, he says the full technical paper is forthcoming [in a few weeks](https://x.com/nickcammarata/status/2110772886118228457).
	5.  Besides testing his approach in small models and vision models, he says he's also begun testing it in [modern large models](https://already-clean.goodfire.com/#:~:text=So far it has worked in language models too, up to trillions of parameters.): "So far it has worked in language models too, up to trillions of parameters. Those aren’t close to solved due to sheer size, but I’m optimistic about scaling the process." More details to come in upcoming technical paper apparently.  
	      
	    The costs involved in making this work might be pretty large; I'd be curious to know how many dollars were spent interpreting Qwen3-32B. But it's a good sign that this research was able to be done outside the frontier labs; Goodfire does have some money though, they raised $150M earlier this year. Also, in terms of efficacy of scaling this approach, he does mention that interpreting gets easier as you go on, kind of like how [filling out a crossword puzzle](https://already-clean.goodfire.com/#:~:text=Interpretability has a surprising amount of momentum) gets easier in some ways as you fill out more of the grid.

	*(edit: have added some extra takeaways above, have not substantively changed any previous ones)*  
	(*edit2: honestly recommend reading* [*Nick's post itself*](https://already-clean.goodfire.com/), it's well-written and has interesting [*tangents*](https://already-clean.goodfire.com/#:~:text=Neuroscience, and Why I Care) *too, not to mention* [*interactive features*](https://already-clean.goodfire.com/microscope/large.html?layer=stage3.block2&neuron=1)!)  
	*(edit3: added section 1.d.iv which I think is important! boy this comment got overly complex though)*  
	*(edit4: okay I really recommend playing around with some of Nick's interactive features, ex.* [*the five neurons in Qwen3-32B*](https://already-clean.goodfire.com/#:~:text=Five neurons in Qwen3-32B)—I like the Colorado Neuron, who knew Bolivia was the Colorado of South America…?)  
	(e*dit5: fyi at least some of the core text in the blog post was written by AI according to Pangram, particularly the section around* [*the sentence*](https://already-clean.goodfire.com/#neuron-scientist-gallery:~:text=Ten billion real tokens) *"Ten billion real tokens had their chance, and a sentence written from the explanation alone got closer." Pangram says ~2% of core blog post (not counting interactive stuff), and ~31% of the appendix, is AI with high confidence.*)

### Comment by [brambleboy](/users/brambleboy)

*   2026-10-15 18:13:28Z
*   Karma: 15
*   Voting system: namesAttachedReactions
*   Approval votes: 12
*   Total votes: 12
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/eapKndpzqbPxh8QZ5](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/eapKndpzqbPxh8QZ5)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/eapKndpzqbPxh8QZ5](/api/post/linkpost-neural-networks-are-already-clean/comments/eapKndpzqbPxh8QZ5)

Reactions (whole comment):

*   goodpoint: 1

	First impressions: The thrust of the essay, that we're much closer to fully interpreting and understanding frontier models than we thought, is big if true and both exciting and scary. It would speed up prosaic alignment and control progress, but it seems like it'd also speed up capabilities progress. For example, the author points out that the older models they studied had hard-to-interpret neurons in part because the older architecture was worse at learning in deeper layers, which implies researchers could use interpretability tools to uncover problems with their learning algorithms.

### Comment by [2001zhaozhao](/users/2001zhaozhao)

*   2026-10-15 19:43:35Z
*   Karma: 13
*   Voting system: namesAttachedReactions
*   Approval votes: 8
*   Total votes: 8
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/eapKndpzqbPxh8QZ5](/api/post/linkpost-neural-networks-are-already-clean/comments/eapKndpzqbPxh8QZ5)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/zZH55vFS28oGJFZQw](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/zZH55vFS28oGJFZQw)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/zZH55vFS28oGJFZQw](/api/post/linkpost-neural-networks-are-already-clean/comments/zZH55vFS28oGJFZQw)

Reactions by quoted text:

*   "then we are in for a hell of a ride."
    
    *   why: 1

		It might go further than that. I think AI 2027 (specifically the section about Agent-5 in the Race ending) predicted that mechanistic interpretability would bring about a major increase in capability through the ability to create fundamentally new AI architectures. But it assumed that mech interp could only be solved by an already superintelligent AI (Agent-4). If they are right about the first part but mech interp is easier than assumed, then we are in for a hell of a ride.

### Comment by [Alex Gibson](/users/alex-gibson)

*   2026-10-16 00:20:23Z
*   Karma: 9
*   Voting system: namesAttachedReactions
*   Approval votes: 6
*   Total votes: 6
*   Reply depth: 2
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/osnmsGt8opfoivTNW](/api/post/linkpost-neural-networks-are-already-clean/comments/osnmsGt8opfoivTNW)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/JciDKn9MGWuTiqrqN](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/JciDKn9MGWuTiqrqN)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/JciDKn9MGWuTiqrqN](/api/post/linkpost-neural-networks-are-already-clean/comments/JciDKn9MGWuTiqrqN)

Reactions by quoted text:

*   "There was no "Highly Speculative" in the X post, and you have to know most people aren't going to read the blog post."
    
    *   agree: 1

			I don't think this is good enough. The X post evidently does a poor job at communicating the level of speculation involved, given how most people have interpreted it.

			There was no "Highly Speculative" in the X post, and you have to know most people aren't going to read the blog post.

### Comment by [Dean Valentine](/users/dean-valentine)

*   2026-10-15 20:58:06Z
*   Karma: 9
*   Voting system: namesAttachedReactions
*   Approval votes: 6
*   Total votes: 6
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/GnTbLpcx6xiyw4ZFa](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/GnTbLpcx6xiyw4ZFa)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/GnTbLpcx6xiyw4ZFa](/api/post/linkpost-neural-networks-are-already-clean/comments/GnTbLpcx6xiyw4ZFa)

	> But in fact, cosine similarities between feature vectors have rich semantic content \[...\] Features aren't just crammed together arbitrarily; they're grouped with similar features.

	[Adam Scherlis](/api/post/MFBTjb2qf3ziWmzz6?commentId=2BwacbvQkLiqz9Ljn), 2024

### Comment by [Julian Bradshaw](/users/julian-bradshaw)

*   2026-10-16 00:13:40Z
*   Karma: 8
*   Voting system: namesAttachedReactions
*   Approval votes: 5
*   Total votes: 5
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/yTi9dKZawGmqmbZo2](/api/post/linkpost-neural-networks-are-already-clean/comments/yTi9dKZawGmqmbZo2)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/osnmsGt8opfoivTNW](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/osnmsGt8opfoivTNW)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/osnmsGt8opfoivTNW](/api/post/linkpost-neural-networks-are-already-clean/comments/osnmsGt8opfoivTNW)

		I think this has gone viral in part because people have grown a little desperate for good AI news. Right post at the right time.

		That said, while I agree that this blog post alone doesn't "solve ambitious interpretability", I think the core of Nick's claim is more "I no longer think there are insurmountable walls to scaling interpretability because of these initial results". That's a speculative claim, yes, but he does say that explicitly, at least [inside the post](https://already-clean.goodfire.com/#:~:text=of the toolkit.-,Highly speculative,-: this might be):

		> 4\. Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we’ve assumed.

		And he is surely aware of the previous SAE overoptimism... though I suppose I didn't see that mentioned in the post.

### Comment by [programjames](/users/programjames)

*   2026-10-15 19:42:32Z
*   Karma: 8
*   Voting system: namesAttachedReactions
*   Approval votes: 3
*   Total votes: 3
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/rPrxQvffSSmkcmqh4](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/rPrxQvffSSmkcmqh4)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/rPrxQvffSSmkcmqh4](/api/post/linkpost-neural-networks-are-already-clean/comments/rPrxQvffSSmkcmqh4)

	Have you done experiments with GANs? I suspect GANs [actually have a different learning mechanism](/api/post/u7xYcMFspgXffnMtu), and you would get a different kind of fractal structure. I also suspect RL does similar things to learning, but image GANs are a simpler species to study.

### Comment by [StanislavKrym](/users/stanislavkrym)

*   2026-10-15 17:00:04Z
*   Karma: 6
*   Voting system: namesAttachedReactions
*   Approval votes: 4
*   Total votes: 4
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/DdhkADuBb8ryS9bZb](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/DdhkADuBb8ryS9bZb)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/DdhkADuBb8ryS9bZb](/api/post/linkpost-neural-networks-are-already-clean/comments/DdhkADuBb8ryS9bZb)

	For context the link is [https://already-clean.goodfire.com/](https://already-clean.goodfire.com/)  [@Yair Halberstadt](/api/user/yair-halberstadt?mention=user) Could you include the link into your post?

### Comment by [Yair Halberstadt](/users/yair-halberstadt)

*   2026-10-15 17:14:09Z
*   Karma: 5
*   Voting system: namesAttachedReactions
*   Approval votes: 3
*   Total votes: 3
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/DdhkADuBb8ryS9bZb](/api/post/linkpost-neural-networks-are-already-clean/comments/DdhkADuBb8ryS9bZb)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/ziA4WGQb2d3wx5vNP](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/ziA4WGQb2d3wx5vNP)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/ziA4WGQb2d3wx5vNP](/api/post/linkpost-neural-networks-are-already-clean/comments/ziA4WGQb2d3wx5vNP)

		Thanks fixed

### Comment by [Yair Halberstadt](/users/yair-halberstadt)

*   2026-10-15 17:14:01Z
*   Karma: 5
*   Voting system: namesAttachedReactions
*   Approval votes: 4
*   Total votes: 4
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/Wa84XfuFks4Y66npD](/api/post/linkpost-neural-networks-are-already-clean/comments/Wa84XfuFks4Y66npD)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/KwqusnGvnPeyYefoR](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/KwqusnGvnPeyYefoR)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/KwqusnGvnPeyYefoR](/api/post/linkpost-neural-networks-are-already-clean/comments/KwqusnGvnPeyYefoR)

		Sorry fixed

### Comment by [Insub](/users/insub)

*   2026-10-15 20:57:31Z
*   Karma: 4
*   Voting system: namesAttachedReactions
*   Approval votes: 4
*   Total votes: 4
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/PFgcWQqjQFHGneEPa](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/PFgcWQqjQFHGneEPa)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/PFgcWQqjQFHGneEPa](/api/post/linkpost-neural-networks-are-already-clean/comments/PFgcWQqjQFHGneEPa)

	The jigsaw puzzle / crossword analogy is particularly compelling to me. Nick seems to be saying: imagine that you're trying to decipher the meaning of a neuron, without knowing what anything else in the network does. The neuron may fire in all sorts of seemingly unrelated circumstances (ie, it's polysemantic). Now imagine that you know what every other neuron in the network does, except for the one you're looking at. Suddenly it gets a lot easier to spot what subtle abstraction ties together all of the neuron's seeming polysemanticity.

	When you're doing a sudoku, you eventually reach a cascade point where you have so much information that everything else just drops into place. Seems like Nick has reached this point on smaller networks and expects to on larger networks, with more effort.

	Very cool & exciting post (at least to me, a layman)!

### Comment by [Mike W](/users/mike-w)

*   2026-10-15 17:54:33Z
*   Karma: 4
*   Voting system: namesAttachedReactions
*   Approval votes: 3
*   Total votes: 3
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/3STe3fx7azKvt345S](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/3STe3fx7azKvt345S)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S](/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S)

	I wonder what similarities there are between mechanistic interpretability research on artificial neurons and the study of biological neurons (see the last section of the article).

	If there are any neuroscientists here I'd love to hear what they think.

### Comment by [Filipe](/users/filipe)

*   2026-10-15 17:00:31Z
*   Karma: 4
*   Voting system: namesAttachedReactions
*   Approval votes: 3
*   Total votes: 3
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/Wa84XfuFks4Y66npD](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/Wa84XfuFks4Y66npD)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/Wa84XfuFks4Y66npD](/api/post/linkpost-neural-networks-are-already-clean/comments/Wa84XfuFks4Y66npD)

Reactions (whole comment):

*   coveredAlready2: 1

	I don't see an actual link to anywhere. Of course I can google it, but you might want to fix it :)

### Comment by [RogerDearnaley](/users/rogerdearnaley)

*   2026-10-16 14:21:34Z
*   Karma: 2
*   Voting system: namesAttachedReactions
*   Approval votes: 1
*   Total votes: 1
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/4KTmuu8tqojuFpoQd](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/4KTmuu8tqojuFpoQd)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/4KTmuu8tqojuFpoQd](/api/post/linkpost-neural-networks-are-already-clean/comments/4KTmuu8tqojuFpoQd)

	Wow! The implications are amazing…

	I'd *love* to see what happens when we point this at Emergent Misalignment, and at reward hacking. Is there a single "toxic persona" neuron (something like "current persona shows Dark Triad behavior"), or a handful of them? Can we just ablate it/them? How many neurons are changing significantly when a model learns to reward hack during RL, and what are their semantic meanings?

### Comment by [Julian Bradshaw](/users/julian-bradshaw)

*   2026-10-16 06:32:36Z
*   Karma: 2
*   Voting system: namesAttachedReactions
*   Approval votes: 1
*   Total votes: 1
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/7wcSqBNzQF2o9SHyY](/api/post/linkpost-neural-networks-are-already-clean/comments/7wcSqBNzQF2o9SHyY)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/dDipkveGPSwJedJGJ](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/dDipkveGPSwJedJGJ)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/dDipkveGPSwJedJGJ](/api/post/linkpost-neural-networks-are-already-clean/comments/dDipkveGPSwJedJGJ)

		> I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less).

		Can you explain more why you believe the post doesn't properly address this?  
		  
		EDIT: Wait duh, you said you didn't expect it to transfer to LLMs from small image models, sorry.  
		  
		The post's ["Do Weak Activations Matter?"](https://already-clean.goodfire.com/#neuron-scientist-gallery:~:text=Do Weak Activations Matter?) section seems to make a strong experiment-backed argument that removing weak activations only loses you fine distinctions:

		![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1792131984/lexical_client_uploads/skwwaxm5lfpmv3tabydk.png)

		(The above image is a bit confusing, but basically, if you remove the weakest 75% of neuron activations in an image model, it loses the ability to distinguish between similar things like "tiger cat" vs. "tabby", but keeps the ability to reliably identify more core concepts like "zebra". It also notes that the strongest 7% of neuron activations are absolutely essential; cutting those breaks the model completely.)

### Comment by [Harjas](/users/harjas)

*   2026-10-15 19:09:33Z
*   Karma: 2
*   Voting system: namesAttachedReactions
*   Approval votes: 2
*   Total votes: 2
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S](/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/dwa7zXXXwXKP8KWSq](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/dwa7zXXXwXKP8KWSq)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/dwa7zXXXwXKP8KWSq](/api/post/linkpost-neural-networks-are-already-clean/comments/dwa7zXXXwXKP8KWSq)

		Not a neuroscientist, but you may like this great post (by a neuroscientist) on [artificial neurons vs biological neurons](https://mikexcohen.substack.com/p/are-artificial-neurons-like-biological).

### Comment by [ACCount](/users/account)

*   2026-10-16 13:35:38Z
*   Karma: 1
*   Voting system: namesAttachedReactions
*   Approval votes: 1
*   Total votes: 1
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S](/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/qnzLoC92AEzz6Pzzv](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/qnzLoC92AEzz6Pzzv)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/qnzLoC92AEzz6Pzzv](/api/post/linkpost-neural-networks-are-already-clean/comments/qnzLoC92AEzz6Pzzv)

		A key issue that stops ANN interpretability findings from translating to BNNs is the instrumentation problem.

		With ANNs, we have the "god's eye view" over every weight and activation, and can run however many interactions we can afford the compute for - all in a very clean reproducible environment. Numerous interpretability techniques rely on that.

		Our ability to inspect and experiment on BNNs is nowhere near that. So even if some ANN interpretability techniques could plausibly fit BNNs, we don't know that because we can't really try. We lack the observability prerequisites for implementing them.

		That being said, the functioning of BNNs is still looking more interpretable now than it ever was.

		We can't reuse most of the specific ML mechanistic interpretability techniques, but the broader understanding of learning systems and their dynamics does give us a leg up - some concepts and findings apply across substrates. ML has already come to provide plenty of advanced tooling for interfacing with BNN - like high performance neural decoders. It's also possible that ML research into biologically plausible learning rules like three-factor learning could in the future allow for deeper understanding of - and interfacing with - the brain's learning mechanisms.

### Comment by [Mike W](/users/mike-w)

*   2026-10-16 02:24:35Z
*   Karma: 1
*   Voting system: namesAttachedReactions
*   Approval votes: 1
*   Total votes: 1
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S](/api/post/linkpost-neural-networks-are-already-clean/comments/3STe3fx7azKvt345S)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/ntrmMZazzbjDnHBth](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/ntrmMZazzbjDnHBth)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/ntrmMZazzbjDnHBth](/api/post/linkpost-neural-networks-are-already-clean/comments/ntrmMZazzbjDnHBth)

		This post reminds me of this concept in neuroscience  
		[https://en.wikipedia.org/wiki/Grandmother_cell](https://en.wikipedia.org/wiki/Grandmother_cell)

### Comment by [Alex Gibson](/users/alex-gibson)

*   2026-10-16 00:55:50Z
*   Karma: 1
*   Voting system: namesAttachedReactions
*   Approval votes: 1
*   Total votes: 1
*   Reply depth: 2
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/XFiDNewPpEzGYmsNQ](/api/post/linkpost-neural-networks-are-already-clean/comments/XFiDNewPpEzGYmsNQ)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/afnLHgeiaqx9SDErc](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/afnLHgeiaqx9SDErc)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/afnLHgeiaqx9SDErc](/api/post/linkpost-neural-networks-are-already-clean/comments/afnLHgeiaqx9SDErc)

			As Julian mentions below, I don't think Nick is as optimistic as his X post suggests.

			I don't think it'd make sense to put much effort into engaging on the object level when the technical paper hasn't been released. But Roko + Leo Gao's replies on X are both quite good imo.

### Comment by [ajskateboarder](/users/ajskateboarder)

*   2026-10-15 17:38:58Z
*   Karma: 1
*   Voting system: namesAttachedReactions
*   Approval votes: 1
*   Total votes: 1
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/xxjdtkbybgCHFgxGZ](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/xxjdtkbybgCHFgxGZ)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/xxjdtkbybgCHFgxGZ](/api/post/linkpost-neural-networks-are-already-clean/comments/xxjdtkbybgCHFgxGZ)

	This is interesting, I believe I had also indirectly found monosemantic-ish neurons when doing various perturbations over swiglu MLPs. I think this is a convenience of swiglu and possibly just gated MLPs in general, nonetheless an overlooked one (afaict the models used in this post all use gated MLPs)

	IMO still seems underdetermined that networks can or should just be broken into neuron circuits and not higher-rank components

### Comment by [Ebenezer Dukakis](/users/ebenezer-dukakis)

*   2026-10-16 00:41:56Z
*   Karma: 0
*   Voting system: namesAttachedReactions
*   Approval votes: 4
*   Total votes: 4
*   Reply depth: 1
*   Parent comment (Markdown): [/api/post/linkpost-neural-networks-are-already-clean/comments/yTi9dKZawGmqmbZo2](/api/post/linkpost-neural-networks-are-already-clean/comments/yTi9dKZawGmqmbZo2)
*   HTML permalink: [/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/XFiDNewPpEzGYmsNQ](/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean/comment/XFiDNewPpEzGYmsNQ)
*   Markdown permalink: [/api/post/linkpost-neural-networks-are-already-clean/comments/XFiDNewPpEzGYmsNQ](/api/post/linkpost-neural-networks-are-already-clean/comments/XFiDNewPpEzGYmsNQ)

		I think it would be worth a toplevel post explaining why you think Nick is too optimistic.

### Navigation

*   [Front page](https://www.lesswrong.com/api/home)
*   [Markdown API documentation](https://www.lesswrong.com/api/SKILL.md)