Hi, my name is Jared, and I study computer science at Concordia University. I am fairly new to mechanistic interpretability (this year), and I wanted to turn one of my projects into my first LessWrong post! Because of this, I am fairly uncertain how well the results hold up, and would appreciate any feedback (specifically, whether an SAE is really an independent check), no matter how negative.
Epistemic status: uncertain, due to reliance on SAEs.
Natural Language Autoencoders (Fraser-Taliente et al., 2026) take in an activation, produce an explanation (thoughts) for it through the AV (activation verbalizer), and then rebuild that activation using the AR (activation reconstructor). It was found by loops (2026) that reconstruction error increases significantly when the final paragraph of the NLA explanation is ablated. Reconstruction, however, is not a good measure of how grounded the text is to the activation. To test groundedness, I split a set of 200 explanations, run through the AV, into three variants (the first two paragraphs, the final paragraph, and the full explanation) and did two tests using an SAE (sparse autoencoder): The first test checked how many of the original activation’s SAE latents were recovered by each variant's reconstruction, and the second test labeled these latents and had an LLM judge check which of these features were stated in each of the variant’s texts. The final paragraph had an FVE close to the full explanation (0.58 vs 0.69). Their reconstruction recovered 59% and 70% respectively of the original activation’s SAE latents, and the second test showed 42% and 43% of features found in the text respectively. The first two paragraphs had an FVE of 0.27, 30% of latents, and 20% of features. This shows the final paragraph contains a significant amount of grounded content and the first two paragraphs seem to have very little grounded content despite being 44% the length of the text of the full explanation. The bulk of these findings heavily relies on the assumption that NLAs and SAEs find a different structure.
Motivation:
Natural Language Autoencoders (Fraser-Taliente et al., 2026) is an Automated Interpretability method, which attempts to turn raw activations into the model’s thoughts. It is done by extracting the Gemma-3-12B-IT activations at layer 32, running that activation through the AV (Activation Verbalizer), which produces the “explanation”. That explanation is then run through the AR (Activation Reconstructor) which attempts to reconstruct the original extracted activation. The two models are first warm-started. RL is performed on the AV where it is rewarded based on the AR reconstructions. NLAs have been used internally by Anthropic to evaluate behaviour of their models. They used them to find evaluation awareness in evaluation transcripts, even though the model rarely stated it. Since NLAs are being used currently, we ought to figure out how much their explanations reflect their activations.
Research Question:
loops (2026) found that removing the final paragraph of an NLA explanation raised reconstruction error far more than removing either earlier paragraph. However, reconstruction error does not measure whether the paragraph is true to the activation. michaelzhang and TurnTrout (2026) trained an NLA starting from implausible explanations, and it reached nearly the same FVE as a normally trained one, while 99.3% of its statements remained implausible. The NLA authors also report that writing quality declines steadily over training even while reconstruction improves (Fraser-Taliente et al., 2026). I would like to test how true the final paragraph, full explanation and the first two paragraphs of the explanation are to the activation. I test this in two independent ways. First, without reading any text: I check how many of the original activation's SAE latents survive in each variant's reconstruction. Second, without using the AR: an LLM judge checks how many of the original activation’s SAE labeled latents (features) each variant's (First two paragraphs, final paragraph, full explanation) text states. If the final paragraph's high FVE reflects real content, it should keep a similar share of the full explanation's content on both tests as it does on FVE.
Experimental Setup:
First, I recreated the NLA experiment setup, using their original repo https://github.com/kitft/natural_language_autoencoders. For data to feed into activations and SAE, I used Gemma Scope 2’s example corpus (McDougall et al., 2025). Because the SAE was trained on oasst1 and LMSYS-Chat-1M (McDougall et al., 2025), with Gemma-written responses, and the NLA on WildChat-1M and Ultra-FineWeb (kitft, 2026), Gemma Scope 2’s example corpus was a good middle ground for the two. Then I extracted the activations at layer 32 of Gemma-3-12B-IT. I ran those activations through the AV. The AV writes each explanation in the same three-part structure (loops, 2026), and the final paragraph always begins with "Final token." I cut each explanation once at that point, which worked for all 200 explanations, giving three variants: the full explanation, the first two paragraphs, and the final paragraph (Figure 1). Each variant was run through the AR, and their corresponding reconstruction vectors were recorded.
The full explanation reconstructs the original activation with a mean FVE of 0.69. The final paragraph alone reaches 0.58 (84% of the full explanation), while the first two paragraphs reach 0.27 (39%). The final paragraph is only slightly longer than the first two paragraphs (73 vs. 59 tokens) but reaches more than twice their FVE, so length alone does not explain the difference (Figure 2).
Test 1: Latent Recovery
I encoded the original activation and each variant's reconstruction with the SAE. An original activation’s latent counts as recovered if it is still active in the reconstruction. This test uses no text and no judge. I used the Gemma Scope 2 SAE (McDougall et al., 2025) for layer 32. Gemma Scope 2 offers two sparsity levels for this layer: l0_big where about 120 latents fire per token, and l0_small, where about 21 do. I proceeded with the sparser l0_small, as its latents could be much more reliably labeled.
The final paragraph was able to recover 59% of the original latents, 11 percentage points lower than the amount recovered by the full explanation at 70%. The first two paragraphs recover 30% (Figure 3).
Test 2: Grounded Features
This test involved labeling the latents (turning them into features), and judging whether those latents are stated in each variant’s text, without using the AR. Using an auto interp method (Bills et al., 2023; Paulo et al., 2025), I labeled each original activation’s latent with a short description. Gemma-3-12B-IT was shown 40 passages from the Gemma Scope 2 example corpus where it fired most strongly, and wrote three candidate labels. The best candidate was picked on held-out examples, then graded on a 32-item test, 16 positions where it fired, and 16 random positions. A label passed if its AUC (a measure of how well the label tells them apart: 0.5 is chance) was at least 0.756. Of the 3,328 labeled latents, 1,771 passed, with a mean AUC of 0.88 against 0.48 for wrong labels.
Using Gemma-3-12B-IT as a judge, each feature from the original activation was judged against each variant, and the judge was asked whether the text describes the concept in the feature. For each feature, I also ran two baseline checks. 1. The real explanation was judged against three features that did not fire in that activation. The judge found these stated only 4.6% of the time (Figure 4), so this is the baseline the real results are compared against. 2. The real feature was judged against three unrelated explanations, which catches features generic enough to match anything.
The judge found 43% of the original's features stated in the full explanation, 42% in the final paragraph, and only 20% in the first two paragraphs, all well above the 4.6% baseline (Figure 5).
Discussion:
Both tests agree that the final paragraph's high FVE reflects real content. Its reconstruction recovers 59% of the original's latents (vs. 70% for the full explanation: Figure 3), and its text states 42% of the original’s features (vs. 43%: Figure 5). This correlation between these two variants is significant, as the two tests rely on different parts of the pipeline to get these results. Test 1 relies on the AR, but never reads the text, and Test 2 reads the text but never uses the AR. 81% of the latents recovered in the first two paragraphs are also recovered by the final paragraph. Of the features the judge finds stated in the first two paragraphs, 78% are also stated in the final paragraph, yet the first two paragraphs make up 44% of each explanation. The NLA format spends a large portion of each explanation repeating content. This format most likely comes from the warm-start data, so it may be possible to improve, though I have not tested this. This is more evident when reading the actual content of the explanation, as much of the first two paragraphs get repeated by the final paragraph.
These tests, however, do not prove that FVE and groundedness in the activation move together. Across full explanations, an explanation that reconstructs well is no more likely to state its activation’s features (r=-0.02), meaning FVE is not a reliable measure for how grounded any given explanation is.
Limitations:
The SAE is independent of the NLA, but not a source of ground truth, as a true claim may have no matching latent. The bulk of these findings heavily relies on the assumption that NLAs and SAEs find a different structure.
Test 2 covers only 53% of latents with validated labels, therefore it is possible that the ones which were not validated favour the first two paragraphs.
The full explanation was not judged directly as its score is the union of its two parts.
Small sample sizes: one model, one layer, one explanation each, 200 activations.
Next Experiments:
The final paragraph contains most of the signal, but much of it describes the final token itself. It would be better to know more of the model’s intentions, plans or beliefs. Therefore, I would like to train an NLA which is rewarded for the latter, and warm-started on data that spends less of each explanation on the final token and more on what the model is doing with it.
The quickest experiment is to run these same tests on an NLA known to confabulate (michaelzhang & TurnTrout, 2026).
Fraser-Taliente, K., Kantamneni, S., Ong, E., et al. (2026). Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Transformer Circuits Thread. https://transformer-circuits.pub/2026/nla/
Paulo, G., Mallen, A., Juang, C., & Belrose, N. (2025). Automatically Interpreting Millions of Features in Large Language Models. ICML 2025. https://arxiv.org/abs/2410.13928
Personal Note:
Hi, my name is Jared, and I study computer science at Concordia University. I am fairly new to mechanistic interpretability (this year), and I wanted to turn one of my projects into my first LessWrong post! Because of this, I am fairly uncertain how well the results hold up, and would appreciate any feedback (specifically, whether an SAE is really an independent check), no matter how negative.
Epistemic status: uncertain, due to reliance on SAEs.
Code here: https://github.com/jaredcobrin/nla-sae-grounding
__________________________________________________________________________
TLDR:
Natural Language Autoencoders (Fraser-Taliente et al., 2026) take in an activation, produce an explanation (thoughts) for it through the AV (activation verbalizer), and then rebuild that activation using the AR (activation reconstructor). It was found by loops (2026) that reconstruction error increases significantly when the final paragraph of the NLA explanation is ablated. Reconstruction, however, is not a good measure of how grounded the text is to the activation. To test groundedness, I split a set of 200 explanations, run through the AV, into three variants (the first two paragraphs, the final paragraph, and the full explanation) and did two tests using an SAE (sparse autoencoder): The first test checked how many of the original activation’s SAE latents were recovered by each variant's reconstruction, and the second test labeled these latents and had an LLM judge check which of these features were stated in each of the variant’s texts. The final paragraph had an FVE close to the full explanation (0.58 vs 0.69). Their reconstruction recovered 59% and 70% respectively of the original activation’s SAE latents, and the second test showed 42% and 43% of features found in the text respectively. The first two paragraphs had an FVE of 0.27, 30% of latents, and 20% of features. This shows the final paragraph contains a significant amount of grounded content and the first two paragraphs seem to have very little grounded content despite being 44% the length of the text of the full explanation. The bulk of these findings heavily relies on the assumption that NLAs and SAEs find a different structure.
Motivation:
Natural Language Autoencoders (Fraser-Taliente et al., 2026) is an Automated Interpretability method, which attempts to turn raw activations into the model’s thoughts. It is done by extracting the Gemma-3-12B-IT activations at layer 32, running that activation through the AV (Activation Verbalizer), which produces the “explanation”. That explanation is then run through the AR (Activation Reconstructor) which attempts to reconstruct the original extracted activation. The two models are first warm-started. RL is performed on the AV where it is rewarded based on the AR reconstructions. NLAs have been used internally by Anthropic to evaluate behaviour of their models. They used them to find evaluation awareness in evaluation transcripts, even though the model rarely stated it. Since NLAs are being used currently, we ought to figure out how much their explanations reflect their activations.
Research Question:
loops (2026) found that removing the final paragraph of an NLA explanation raised reconstruction error far more than removing either earlier paragraph. However, reconstruction error does not measure whether the paragraph is true to the activation. michaelzhang and TurnTrout (2026) trained an NLA starting from implausible explanations, and it reached nearly the same FVE as a normally trained one, while 99.3% of its statements remained implausible. The NLA authors also report that writing quality declines steadily over training even while reconstruction improves (Fraser-Taliente et al., 2026). I would like to test how true the final paragraph, full explanation and the first two paragraphs of the explanation are to the activation. I test this in two independent ways. First, without reading any text: I check how many of the original activation's SAE latents survive in each variant's reconstruction. Second, without using the AR: an LLM judge checks how many of the original activation’s SAE labeled latents (features) each variant's (First two paragraphs, final paragraph, full explanation) text states. If the final paragraph's high FVE reflects real content, it should keep a similar share of the full explanation's content on both tests as it does on FVE.
Experimental Setup:
First, I recreated the NLA experiment setup, using their original repo https://github.com/kitft/natural_language_autoencoders. For data to feed into activations and SAE, I used Gemma Scope 2’s example corpus (McDougall et al., 2025). Because the SAE was trained on oasst1 and LMSYS-Chat-1M (McDougall et al., 2025), with Gemma-written responses, and the NLA on WildChat-1M and Ultra-FineWeb (kitft, 2026), Gemma Scope 2’s example corpus was a good middle ground for the two. Then I extracted the activations at layer 32 of Gemma-3-12B-IT. I ran those activations through the AV. The AV writes each explanation in the same three-part structure (loops, 2026), and the final paragraph always begins with "Final token." I cut each explanation once at that point, which worked for all 200 explanations, giving three variants: the full explanation, the first two paragraphs, and the final paragraph (Figure 1). Each variant was run through the AR, and their corresponding reconstruction vectors were recorded.
The full explanation reconstructs the original activation with a mean FVE of 0.69. The final paragraph alone reaches 0.58 (84% of the full explanation), while the first two paragraphs reach 0.27 (39%). The final paragraph is only slightly longer than the first two paragraphs (73 vs. 59 tokens) but reaches more than twice their FVE, so length alone does not explain the difference (Figure 2).
Test 1: Latent Recovery
I encoded the original activation and each variant's reconstruction with the SAE. An original activation’s latent counts as recovered if it is still active in the reconstruction. This test uses no text and no judge. I used the Gemma Scope 2 SAE (McDougall et al., 2025) for layer 32. Gemma Scope 2 offers two sparsity levels for this layer: l0_big where about 120 latents fire per token, and l0_small, where about 21 do. I proceeded with the sparser l0_small, as its latents could be much more reliably labeled.
The final paragraph was able to recover 59% of the original latents, 11 percentage points lower than the amount recovered by the full explanation at 70%. The first two paragraphs recover 30% (Figure 3).
Test 2: Grounded Features
This test involved labeling the latents (turning them into features), and judging whether those latents are stated in each variant’s text, without using the AR. Using an auto interp method (Bills et al., 2023; Paulo et al., 2025), I labeled each original activation’s latent with a short description. Gemma-3-12B-IT was shown 40 passages from the Gemma Scope 2 example corpus where it fired most strongly, and wrote three candidate labels. The best candidate was picked on held-out examples, then graded on a 32-item test, 16 positions where it fired, and 16 random positions. A label passed if its AUC (a measure of how well the label tells them apart: 0.5 is chance) was at least 0.756. Of the 3,328 labeled latents, 1,771 passed, with a mean AUC of 0.88 against 0.48 for wrong labels.
Using Gemma-3-12B-IT as a judge, each feature from the original activation was judged against each variant, and the judge was asked whether the text describes the concept in the feature. For each feature, I also ran two baseline checks. 1. The real explanation was judged against three features that did not fire in that activation. The judge found these stated only 4.6% of the time (Figure 4), so this is the baseline the real results are compared against. 2. The real feature was judged against three unrelated explanations, which catches features generic enough to match anything.
The judge found 43% of the original's features stated in the full explanation, 42% in the final paragraph, and only 20% in the first two paragraphs, all well above the 4.6% baseline (Figure 5).
Discussion:
Both tests agree that the final paragraph's high FVE reflects real content. Its reconstruction recovers 59% of the original's latents (vs. 70% for the full explanation: Figure 3), and its text states 42% of the original’s features (vs. 43%: Figure 5). This correlation between these two variants is significant, as the two tests rely on different parts of the pipeline to get these results. Test 1 relies on the AR, but never reads the text, and Test 2 reads the text but never uses the AR. 81% of the latents recovered in the first two paragraphs are also recovered by the final paragraph. Of the features the judge finds stated in the first two paragraphs, 78% are also stated in the final paragraph, yet the first two paragraphs make up 44% of each explanation. The NLA format spends a large portion of each explanation repeating content. This format most likely comes from the warm-start data, so it may be possible to improve, though I have not tested this. This is more evident when reading the actual content of the explanation, as much of the first two paragraphs get repeated by the final paragraph.
These tests, however, do not prove that FVE and groundedness in the activation move together. Across full explanations, an explanation that reconstructs well is no more likely to state its activation’s features (r=-0.02), meaning FVE is not a reliable measure for how grounded any given explanation is.
Limitations:
Next Experiments:
References
Bills, S., Cammarata, N., Mossing, D., et al. (2023). Language models can explain neurons in language models. OpenAI. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html
Fraser-Taliente, K., Kantamneni, S., Ong, E., et al. (2026). Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Transformer Circuits Thread. https://transformer-circuits.pub/2026/nla/
kitft. (2026). nla-gemma3-12b-L32-av [Model card]. Hugging Face. https://huggingface.co/kitft/nla-gemma3-12b-L32-av
loops. (2026). Some observations about NLA explanations. LessWrong. https://www.lesswrong.com/posts/ahu9BCHPFDg9AJF75/
McDougall, C., Conmy, A., Kramár, J., et al. (2025). Gemma Scope 2 — Technical Paper. Google DeepMind. https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/gemma-scope-2-helping-the-ai-safety-community-deepen-understanding-of-complex-language-model-behavior/Gemma_Scope_2_Technical_Paper.pdf
michaelzhang, & TurnTrout. (2026). How robust are natural language autoencoders to initialization? LessWrong. https://www.lesswrong.com/posts/LQXWiF8PyJ5ojNsEv/
Paulo, G., Mallen, A., Juang, C., & Belrose, N. (2025). Automatically Interpreting Millions of Features in Large Language Models. ICML 2025. https://arxiv.org/abs/2410.13928