NB: This is my first post on LW as a longtime lurker - I hope I'm not stepping on any norms. I prioritized speedposting for relevance rather than polish.
Cammarata's Neural Networks Are Already Clean has been discussed here already (linkpost, pushback on their version of explainability), but I have not yet seen anyone argue the fact that there will be a discontinuity in cleanliness, ie: the [single-neuron / single concept] pair coverage as a useful metric.
We have three quantities of interest:
network capacity somewhat analogous to number of neurons here
number of human understandable names and sentences for concepts
number of concepts encoded in an LLM by backprop
Cammarata says that as goes up, the compression of into goes down and neurons become interpretable because . But: is bounded, keeps going up, and post-training pushes . Ergo, there will be a situation soon enough. And backprop leaves no free lunch uneaten: the neurons either will become redundant over human concepts, or there will appear ever more cross terms unbounded in their complexity. The small networks use compositionality and superposition as a workaround for lack of capacity, but optimization will apply pressure towards monosemanticity if there is slack. There will exist a regime soon where monosemanticity becomes non human readable.
This is a failure mode of the audits described too: top-k anomalies are detected because we fundamentally have neurons encoding OR functions, but ANDs are undetected. We know that post-training grows AND clauses, and some of the most important clauses evolve around eval awareness, sandbagging, and general misalignement[1]. This might be caught by upstream circuit tracing verbalizing clause detectors, but there's no guarantee that such a circuit will be systematically a large enough weight to appear in the audits input rankings.
In conclusion: at surplus capacity regimes, the number of clauses per neurons spike up, the clauses of interest get summarized by the labellers into oblivion, and we cannot rely on neuron inspection for human supervision of LLMs above an unspecified size. This gives us a potentially useful higher bound for the size of LLM we can use and monitor in alignement research though.
In the Claude Sonnet 4.5 system card: representations for eval awareness grow in saliency over training, inhibition reduced verbalised evaluation awareness while raising misaligned behaviour. Reproduced since on GLM-5.
NB: This is my first post on LW as a longtime lurker - I hope I'm not stepping on any norms. I prioritized speedposting for relevance rather than polish.
Cammarata's Neural Networks Are Already Clean has been discussed here already (linkpost, pushback on their version of explainability), but I have not yet seen anyone argue the fact that there will be a discontinuity in cleanliness, ie: the [single-neuron / single concept] pair coverage as a useful metric.
We have three quantities of interest:
Cammarata says that as
This is a failure mode of the audits described too: top-k anomalies are detected because we fundamentally have neurons encoding OR functions, but ANDs are undetected. We know that post-training grows AND clauses, and some of the most important clauses evolve around eval awareness, sandbagging, and general misalignement[1]. This might be caught by upstream circuit tracing verbalizing clause detectors, but there's no guarantee that such a circuit will be systematically a large enough weight to appear in the audits input rankings.
In conclusion: at surplus capacity regimes, the number of clauses per neurons spike up, the clauses of interest get summarized by the labellers into oblivion, and we cannot rely on neuron inspection for human supervision of LLMs above an unspecified size. This gives us a potentially useful higher bound for the size of LLM we can use and monitor in alignement research though.
In the Claude Sonnet 4.5 system card: representations for eval awareness grow in saliency over training, inhibition reduced verbalised evaluation awareness while raising misaligned behaviour. Reproduced since on GLM-5.