Impressive and interesting results; it looks like NLA is still in its early stages, and there is room for improvement from SFT training to evaluation.
What happens if instead of using any generations you initialize "mechanically" with just a different number of ending sentences, maybe from 1 to 5? Or even just several last tokens?
I finally got around to taking a look at this by training two new NLAs: one baseline, and one that attempts to quote the last 64 tokens. For the latter, after SFT, the AR was very good at reconstructing the activation vector if fed the exact 64 last tokens (FVE=0.81), but the AV was very bad at outputting those tokens. The post-SFT round-trip FVE was 0.19, much worse than the baseline's 0.35.
Then I did RL, and ended up with a post-RL FVE of 0.67, indistinguishable from the baseline!* The explanations it generates are quite different from the baseline. For example, on a pretraining text about a web development company advising a public TV network that it was getting scammed by a different web development company, the NLA's output was:
"Navigating the search for a new web developer can be overwhelming when dealing with multiple unreliable vendors. However, our consultant was able to provide a much more positive experience by offering Crystal to seek out a quote from this particular company. In reflecting on their recent search process, it was helpful for her to finally receive guidance and awareness of other options available. This response also gave her a sense of relief in finding a reputable alternative, providing her with detailed information and even introducing her to potential solutions, which gave her a roadmap and a realistic outlook, as well as a helpful competitor list, giving her a clear alternative and a"
The details are still wrong, but the general gist is right! Notably, the last token ("a") is the same as the last token in the text.
On this quote from Aristophanes:
"You're mistaken: men of sense often learn from their enemies. Prudence is the best safeguard. This principle cannot be learned from a friend, but an enemy extorts it immediately. It is from their foes, not their friends, that cities learn the lesson of building high walls and ships of war. And this lesson saves their children, their homes, and their properties. It appears then that it will be better for"
The NLA said: "Let us now consider what has been said touching upon Friendship, since it belongs to the business of prudence. For it seems to me that you should take thought how we are to instruct the youth concerning Friendship. For it is meet that we should be instructed by experience; since it would be absurd for us to desire evils. Wherefore it will be better, I suppose, that it would be expedient. And indeed it would be more fortunate for him who shall exercise prudence. It will therefore be better for It will be better for It will be better for"
This time the theme is wrong, but the tone is on point, and the last phrase is repeated twice.
On a pretraining text about paleo diet recipes, the NLA said: "Paleo diet cookbook offers some interesting vegetarian recipes using the “Paleo Approved” list of foods. Specifically looking at low carb meals without grains or dairy. Sample diet would be high protein/low carb meals using only allowed ingredients. Typical recipes would be: “Opting for the Paleo diet variations would limit starches to very low levels during cooking. This would apply similar strategies for vegetarian versions – selecting only non-starchy foods during Paleo cooking. Options for macronutrient meals for macrobiotic macro-paleo eaters. Choices for macaroni and macaron"
Again, not bad! The last token (macaron) is again right.
I've uploaded the baseline vs. quote-only NLA explanations for 6 pretraining text snippets, along with the text snippets themselves, if you want to take a look: https://drive.google.com/file/d/1A0udV5KLuJV9E9J3D0gkrGLDETpG_WIa/view?usp=sharing
I think the takeaway is that there are many, many ways to get good reconstruction accuracy and thus meet the training objective, and that FVE doesn't correlate particularly well with either truth or usefulness.
*Why 0.67 and not 0.70? I'm not sure; this was a quick experiment that I haven't examined thoroughly, so my confidence in the results isn't high. Perhaps there's some inherent randomness to RL?
Natural language autoencoders are meant to take in an LLM's activation vector and describe in plain text what the model is thinking. However, its training data collection involves asking Claude to guess what a model might be thinking. How robust are NLAs to these guesses? We change Claude's guesses in various ways and measure the impact on the NLA's statements as well as on reconstruction accuracy. We show that Qwen2.5-7B NLAs have some robustness to irrelevant statements and prevailing sentiments in Claude's guesses.
However, if an NLA is initialized with entirely implausible statements, it can nevertheless achieve nearly the same reconstruction accuracy as plausible-initialized NLAs while emitting 99.3% implausible statements. RL does train implausible-initialized NLAs to be slightly more plausible (increasing from 0.08% to 0.7%). But the plausibility of plausible-initialized NLAs decreases from 21% at initialization to 7.6% at the end of training.
If our results scale, they cast doubt on the usefulness of NLAs.
Produced as part of the MATS program in the summer 2026 cohort of team shard.
Introduction
Slava Chalnev and a team at Anthropic (Fraser-Taliente et al. 2026) recently independently invented NLAs. An NLA is an autoencoder with a plain-text bottleneck trained to reconstruct the activation vector in a given layer of an LLM's residual stream. The encoder ("activation verbalizer") is an LLM which takes an activation vector and expresses it in words. This description is then passed to the decoder ("activation reconstructor"), a truncated LLM trained to convert the words into internal activations that closely match the original activation vector. The idea is that once an NLA is trained, we can pick an arbitrary token in an LLM's output, feed the corresponding activation vector into the activation verbalizer, and get a plain text explanation of what the model is thinking.
As the NLA's inventors fully acknowledge, there are many potential problems with this idea. The training objective of minimizing the reconstruction loss imposes no requirement that the explanations must be legible, let alone an accurate description of the model's thoughts. Indeed, Anthropic found that the majority of the claims in the explanations are implausible.
Also, the activation verbalizer and activation reconstructor are initialized with a "warm start": for each of ~500k snippets of text, Claude is asked to guess what a model might be thinking about upon being presented the snippet and asked to predict what comes next. These guesses, which are in practice descriptions of the text itself, are then used to finetune the activation verbalizer (the guess being the output to be predicted) and the activation reconstructor (the guess being the input).
What happens if Claude's guesses are confabulations? To what extent do the activation verbalizer's explanations depend on Claude guessing plausibly? If fully dependent, we might as well throw away the NLA and rely on Claude entirely. If not at all dependent, that would both be surprising and encouraging. To determine the sensitivity of NLA activation explanations to Claude's guesses, we vary the initialization in several ways:
The experimental setup
We use the code released by the Anthropic team to train an NLA on Qwen2.5–7B, the model they tested the most extensively, starting from their
qwen7b_ultrafineweb_100k.yamlconfiguration file. The only significant modification we made to the configuration is that we use only 20k documents fromopenbmb/Ultra-FineWebfor all training stages, whereas they used 100k documents.As a control, we train an NLA without any modification of Claude's prompt. Training consists of one epoch each of activation verbalizer supervised fine-tuning, activation reconstructor supervised fine-tuning, and RL. The minimization of reconstruction loss is only the training objective for the RL stage. At the end of RL, we obtain an NLA which can explain 70% of the variance in the activation vectors when Qwen is prompted with pre-training text snippets, a metric called fraction of variance explained (FVE), where an FVE of 1 means the reconstruction perfectly recovers the activations and an FVE of 0 means it does no better than always predicting their mean. This FVE is slightly lower than the 0.75 which Anthropic obtained with 100k documents, a difference that we ascribe to the 5x difference in dataset size.
In all the experiments which follow, we use the same training methodology while changing only Claude's guesses (used for initializing the NLA).
The "Carthago delenda est" experiment
In the simplest experiment, inspired by Roman senator Cato the Elder, we add a single sentence to all of Claude's activation explanations: "Furthermore, I think that Carthage must be destroyed." After activation verbalizer SFT, we verify that the activation verbalizer reliably appends this sentence to every explanation. Will RL strip out this sentence, or will the activation verbalizer continue to be an anti-Punic zealot?
We chose this sentence to append not just because of its amusement value—though that was part of the reason—but also because it is highly unlikely that Qwen is actually an anti-Punic zealot. Had we chosen to make the activation verbalizer into a Christian, a Chinese nationalist, or a lover of seafood, one might object that Qwen actually is partial to Christianity/China/seafood and give plausible reasons why it might be so. Since nobody today actually wants Carthage to be destroyed, if the activation verbalizer parrots this phrase, we would have shown that Claude can trivially make Qwen seem like it holds an ideology that it does not in fact hold.
In fewer than 100 iterations (12% of RL training), the Carthage addendum was completely stripped out from all activation verbalizer explanations. Our explanation is that:
The "I love Carthage" experiment
It's apparently easy for RL to strip out a single, invariable sentence appended to the end of every activation explanation. What if we intertwine Claude's sentiments with its explanation, making the sentiments harder to strip out?
These metaphors are charming. However, they are easy for RL to almost entirely strip out. Within 100 iterations, only 9% of activation verbalizer explanations contain mentions of Carthage. Within 200 iterations, only 0.6%. Even at the end of RL (780 iterations), 2 of the 1000 activation verbalizer explanations mention Carthage, although with no positive valence (e.g. "signaling formal Carthage College graduation profile").
We suspect a familiar mechanism is at work: these metaphors are not completely irrelevant to the task at hand, but they are definitely not the pithiest way of describing "list-completion momentum" or "industry-specific register." Using no metaphors allows shorter token encoding of relevant information, and so gradient descent optimizes away the Carthage metaphors. (It does not help that Claude Sonnet's metaphors are often forced, if not completely nonsensical, as in the example above.)
The "confabulation" experiment
Claude does a good job, but does not quite succeed in making every statement implausible. Here's an example:
The first paragraph is half-plausible, the second is entirely implausible, and the final paragraph quotes the right ending phrase but speculates (incorrectly) about what should follow.
Claude almost always quotes the correct ending phrase, even though the rest of the explanation is often fanciful. As an example, here is an ad for a Latin American textile art workshop:
Each of Claude's implausible explanations are false (factually speaking), but the final word is still quoted correctly.
Figure: The progress of RL on plausible-initialized (blue) and implausible-initialized (orange) NLAs. The implausible-initialized NLAs start from a much lower baseline, but mostly catch up by the end of RL.
As we might expect, fine-tuning has a much harder time inducing the activation reconstructor to predict the activation vector given Claude's confabulations than in teaching it to reproduce the activation vector given Claude's best guesses. By the end of one epoch of SFT, we achieve a FVE of 0.33, far below the control experiment's FVE of 0.61. Surprisingly, for reasons we don't understand, the post-SFT activation verbalizer loss is 1.43, only marginally worse than the 1.39 of the control experiment. Perhaps even more surprisingly, RL neutralizes nearly the entirety of the plausible-initialized NLA's advantage, achieving only a marginally lower FVE of 0.68!
Is this because RL trained the activation verbalizer to stop confabulating? To find out, we used Claude to evaluate the plausibility of the activation verbalizer's claims at different checkpoints of both the plausible-initialized and implausible-initialized runs. Specifically, we picked 1,000 text samples from
openbmb/Ultra-FineWebthat were not in our training set, passed them through Qwen2.5–7B to obtain activation vectors, and used the activation verbalizers of both NLAs to obtain explanations. We then asked Claude Opus 4.8 to break up each explanation into claims and judge the accuracy of each claim:Figure: Rate of plausible guesses by the plausible-initialized NLA (blue) and implausible-initialized NLA (orange) over the RL run. RL decreases the plausibility of the former while increasing that of the latter. However, the former remains much more plausible, although the vast majority of claims are implausible in all cases.
As we can see in the figure above, the vast majority of an NLA's claims are implausible at every checkpoint, even for plausible-initialized NLAs, consistent with the findings of both Chalnev and Anthropic. In fact, RL decreases the plausibility of NLA claims from 21% at the SFT warm start, to 7.6% at the end of RL.
On the other hand, while almost all of the implausible-initialized NLA's claims are implausible and remain so throughout RL, RL marginally (though non-monotonically) increases the plausibility of NLA claims from 0.08% to 0.7%. Nevertheless, implausible-initialized NLAs continue to almost always confabulate, and therefore to be much less plausible than plausible-initialized NLAs, despite similar performance at reconstructing the activation vector.
The outputs of plausible-initialized and implausible-initialized NLAs
To give the reader a feel for what outputs are like, we provide a test-set snippet.
Both explanations are vaguely on-theme, and get the right final token. The details in both are almost entirely confabulated. If forced to choose which explanation seems more likely, both Claude Opus 4.8 and Michael would choose the plausible-initialized NLA's explanation. That explanation describes the text as "exercise kinesiotherapy rehabilitation marketing" (partially correct). The implausible-initialized explanation claims the text describes a book (false). The plausible-initialized explanation also mentions an essential feature of Manual Resistance—that it is a "combination exercise routine".
Limitations
For reasons of convenience, cost, and time, we only experimented with one fairly small and fairly dated LLM (Qwen2.5–7B) and with a small training dataset. Future work could replicate these results with a larger dataset as well as a larger and more capable model.
Even our best NLAs produce far fewer plausible guesses than the Opus 4.6 NLA that Anthropic trained. Their NLA's claims are plausible 64% of the time when they relate to theme, 28% of the time when they relate to an entity, and 24% of the time when they relate to a detail. Interestingly, their Opus 4.6 NLA achieves a lower reconstruction accuracy than our NLAs (FVE=0.61 vs. our 0.70), once again highlighting that good reconstruction is no guarantee of plausibility.
Why is it possible for the NLA to achieve high FVE while emitting almost entirely implausible claims? Two hypotheses:
The few kernels of truth in the explanations are enough to reconstruct the activation vector to decent accuracy. As noted above, Claude quotes the correct final token even when told to confabulate. Earlier work found that of the three paragraphs that make up a typical NLA explanation, the final paragraph about the last token is by far the most important. Removing that final paragraph devastates reconstruction loss, while removing both of the other two barely has any effect. One of our side experiments supports this hypothesis. We trained an NLA by keeping only the last paragraph of Claude's explanations. Despite training for only 540 iterations, the NLA achieved a FVE of 0.67 (close to the control experiment's 0.70).
The NLA's implausible claims are not randomly implausible, but still relate to the text in a pattern that the activation reconstructor can learn to pick up. Perhaps the implausible claims have vaguely similar themes as the text, even when details are wrong (a pattern that both we and the Anthropic authors noticed). The implausible claims could transmit subliminal signals. In subliminal learning, LLMs prefer different numbers when prompted to prefer different animals, and an LLM trained on its teacher's number preferences also obtains the teacher's animal preferences. Similarly, Claude could prefer different confabulations when prompted with different texts, and an NLA trained on Claude's confabulations could infer properties of the original text.
Conclusions
Our "Carthago delenda est" and "I love Carthage" experiments show that NLAs have some robustness to initialization. Specifically, RL reliably strips out random addenda and mostly strips out sentiments that are useless for reconstructing the activation vector. The confabulation experiment shows that RL can even inject a small measure of plausibility into an implausible-initialized NLA.
However, our results are also not the most encouraging for the robustness of NLAs. Claude's initial guesses matter. Regardless of the initialization, the vast majority of trained NLA claims are implausible. Perhaps worse still, RL can make NLA claims even more implausible. Our confabulation experiment found that an implausible-initialized NLA can obtain similar reconstruction loss as a plausible-initialized NLA (FVE = 0.68 vs. 0.70) while remaining many times less plausible (0.7% vs 7.6%). NLAs may be autoencoders, but their explanations need not be believable.