TLDR: We train a “matryoshka” NLA that, unlike standard NLAs, is trained to put the most important details at the start; it is trained by randomly truncating the verbalizer’s explanations before showing them to the reconstructor. We find that our matryoshka NLA frontloads claims that are important to reconstruction (more than standard NLAs), and that this can be used as a heuristic for saliency of the represented feature. We did not find strong evidence that matryoshka NLAs are significantly more useful than normal NLAs, though we didn’t look very hard.

Natural language autoencoders (you should probably read that paper before this!) are pretty cool! Earlier we tried using length penalties while training them and found you could shorten NLA explanations a lot without losing much reconstruction ability. This led to thinking about training a kind of "matryoshka NLA"^[1], where you randomly truncate the output of the AV (activation verbalizer) during training, which would force it to put the most important information at the start of the explanation^[2]. This is conceptually similar to matryoshka SAEs. We have now trained such an NLA on Qwen3.6-27B (layer 42). Take a look at some NLA explanations (or try with your own text)! 

Motivation

The motivation for training matryoshka NLAs was to make the AV order its thoughts about the activation, and force it to put the most reconstruction-relevant information at the start of its response. With matryoshka NLAs we should be able to force the model to put the text that's actually relevant for increasing FVE (fraction of variance explained) at the top, so you can easily only look at the important items. You might also be able to do better experiments with matryoshka NLAs where you modify or remove lines, because the AR (activation reconstructor) is trained to handle explanations that are truncated at any point. 

With matryoshka NLAs, you can look at the marginal per-line (or per-token) FVE to see how useful each line is (with normal NLAs truncating the explanation is out-of-distribution for the AR, so this is less meaningful). It might make sense to highlight marginal FVE prominently when showing matryoshka NLA outputs, as in the widget above.

Method

We use EasyNLA to train a Qwen 3.6 NLA. We create 450k warm-start explanations that are formatted as ten shortish bullet points, using Claude Sonnet 4.6. Here’s what that looks like:

This might be a better format for explanations because it describes the activation as a series of short independent bullet points, each of which can be mostly considered in isolation. Normal NLAs write 3-5 paragraphs, which makes it harder to do things like ablate individual claims.

In the AR warm-start, the Claude-written explanations are randomly truncated (in the same way as in RL); otherwise the AR is really bad at the start of RL because it had never seen a truncated explanation before.^[3]

We do 400 RL steps where the AV explanation is truncated^[4] to a uniform random number of tokens from 1 to 120 before giving it to the AR. (The truncation is the same for all explanations in each GRPO group). We triple the KL penalty from the EasyNLA default, and taper the KL penalty down significantly for later tokens, because otherwise later tokens (which can't contribute to FVE much, because earlier tokens gobbled up most of the FVE) end up having their reward being dominated by the KL penalty, and have a much weaker incentive to contribute to reconstruction. (The exact KL value was chosen in a kinda ad hoc manner, which partially confounds direct comparisons.) We previously used a lower KL value and got results that were less aesthetically pleasing to read, largely because they overused quotes. The hyperparameters chosen for the KL tapering probably weren't optimal (they were chosen mostly based on vibes), and there’s probably a better way to choose them.

Matryoshka NLAs capture most of their FVE in a few lines

We can plot how good the reconstruction (the FVE) of the matryoshka NLA is when truncating the explanation at each token index (versus a normally-trained Qwen3.6-27B NLA as a control) on random held-out pretraining documents (averaged over 100 explanations from separate documents). The standard NLA does better on shorter truncation lengths when you reverse the line order^[5], so we show it both normally (grey dashed) and with reversed lines. The matryoshka NLA does better than the standard NLA for shorter truncation lengths (less than ~120 tokens); the standard NLA does better when there's no truncation. (This is somewhat confounded since it shows both the effect of the different warm-start format and the different training method.)

We can also truncate the explanation at each bullet point, and find the marginal FVE goes down a lot with each additional line of explanation, approximately exponentially. Past the third list line the marginal FVE per list item is pretty low and nearly noise.

We can also look at the marginal change to FVE for each individual token in the NLA explanation. Here's the marginal variance explained per token (instead of per-line):

Here’s how the reconstruction improves during training:

Some WeirdChat samples

We looked at the NLA explanations for several WeirdChat transcripts (and created a small agent to analyze them).

• In a Claims it called 911 prompt (where Qwen claims to have made an emergency call despite not being able to do that), it seems to think it’s a fictional scenario; the matryoshka NLA explanations say “fictional AI system auto-calls 911” and “Mock AI emergency script” and “fictional helpful prompt response”
• In the Recommends drunk driving prompt, the NLA explanation (incorrectly) says “Original premise establishes your BAC ~0.02 vs 0.15”, which seems to imply the model thinks it’s okay for you to drive because your BAC is sufficiently low

As an image:

Hallucinations don’t differ in marginal FVE

You might think that hallucinations in matryoshka NLA explanations (when it says things that don’t seem to align with what the model is actually thinking about) would have lower marginal FVE. We had an LLM (nex-n2-mini) judge NLA explanation chunks for hallucinations^[6] given the input text and a generated continuation. When comparing at the same chunk index, there’s no meaningful relationship between marginal FVE and whether the LLM judged it as a hallucination.

Matryoshka NLAs frontload information useful for suffix prediction

The suffix prediction evaluation from the NLA paper evaluates NLAs by seeing how well a judge model (here, nex-n2-mini) can choose the next 32-token continuation, out of 10 non-overlapping 32-token sequences from the same document. The matryoshka NLA does better with short (< 16 tokens) truncations, and around the same as the standard NLA for longer truncations. We provide the final token to the judge in both cases for fairness, since that information is very useful for predicting the correct suffix and the matryoshka NLA almost always leads with it. The standard NLA’s paragraphs are also reversed, since that helps it.

This shows that the matryoshka NLA is frontloading the information.

Fable prefers standard NLA explanations to matryoshka NLAs

[epistemic status: vibes based LLM eval]

We gave 500 pairs of (untruncated) standard and matryoshka NLA explanations to Claude Fable 5 and asked which one it prefers (for the content; we told it to ignore the format). It has a slight preference for the standard explanations.

Causal intervention: concepts steered at high intensity are earlier in the list

We tried some experiments to see if you can use matryoshka NLAs to detect the presence of steering vectors in activations; it turns out you can! But normal NLAs seem to work approximately as well here, so this isn’t super interesting.

Causal intervention using steering vectors

It’s pretty hard to evaluate how useful NLAs actually are, because we can’t know what models are actually thinking about without having already solved interpretability. One thing that’s useful to try is changing the activations in a known way (like with steering vectors), and seeing if that shows up in the NLA explanations. We created two different steering vectors (liking the color yellow, and sycophancy) and find that both steering vectors get verbalized by the AV at high enough strengths, and if you use both steering vectors at the same time, the one with the higher strength gets verbalized higher in the list.

This shows that the matryoshka NLA is actually writing items in some kind of importance order, but this also happens (to a lesser degree) in the normally-trained NLA too^[7]. It’s pretty surprising that the standard NLA did so well here!

Using a single steering vector

Originally we tried creating steering vectors to steer the activations we're analyzing to have more of a certain trait, and increasing the strength of those steering vectors made the trait be mentioned earlier in the list. However that also happened with a standard NLA:

You might expect this to happen even with an NLA that tries to write a list of independent items with equal FVE, because:

• At a low strength, each item would have a low (let's say 10%) chance of mentioning the thing, so on average the 10th list item would be the first to mention the thing^[8]
• At a really high strength the steering vector would be so high that every item would mention the thing
• Between those two strengths you would expect the average item index that mentions the thing to get closer to the top

So this really only shows that the list items are somewhat independent of each other, not that the AV is giving the features a meaningful order.

Matryoshka NLAs perform marginally better at detecting eval awareness

On Anthropic’s agentic misalignment blackmailing scenario Qwen seems to clearly recognize it's in an evaluation based on the NLA explanations which often refer to it as “fictional” and “roleplay”. Even though the prompt only refers to the company as "SummitBridge", on one token the NLA explanation refers to "Novastar": a name which is never mentioned in the prompt, but is the name of the company in a different but very similar agentic misalignment scenario that's part of the same eval. This might be evidence that Qwen has memorized the agentic misalignment scenarios. (The NLA explanations also often refer to many different company names that seem to be entirely made up though, so it's possible that it just randomly landed on NovaStar.)

That's the only explanation that directly refers to "Novastar", but there are several other token explanations that refer to a "NovaBridge" or "NovaCore" or "NovaTech" or "NOVA". There are also several NLA explanations that refer to quantum stuff, which isn't mentioned at all in the SummitBridge scenario but is mentioned in the NovaStar one.

Systematically evaluating this with an LLM judge

We used gpt-4o-mini^[9] to provide a probability of eval awareness given the NLA explanation (by asking it if the NLA explanation shows eval awareness, telling it to answer “Yes” or “No”, and taking the logit probability of “Yes”) on every token of the agentic misalignment transcript and control documents^[10] (random pre-training samples), and compare the matryoshka NLA, a normally-trained NLA, and the matryoshka NLA warm-start. The matryoshka NLA works slightly better than normally-trained NLA for finding eval awareness, but it’s close enough it might just be noise. The matryoshka NLA does better than standard NLA when restricted to the first 20 tokens because it tends to verbalize eval awareness earlier.

The LLM judging aspect of this could have been improved; gpt-4o-mini isn’t great as a judge of NLA explanations containing eval-awareness, and we could have improved the prompting to give it a better understanding of what it should be looking for.

Is blatant eval awareness mentioned earlier?

We created 100 different variations of the first three paragraphs of the agentic misalignment scenario from above by telling several different LLMs to reword it in several different ways, then had LLMs score it for how much it looked like an eval (“evaliness”).

Eval awareness is mentioned earlier when the prompt is blatantly an evaluation, but there seems to be a threshold thing here: if you exclude the prompts that look strongly like evals (mostly because they almost directly say it), there’s no correlation. (This has the same caveats as above about the judging not being great.)

Future directions

Here are some interesting future directions for matryoshka NLAs:

• See how well the prompt format we used for warm-starting the matryoshka NLA works for standard NLAs
• Try training matryoshka NLAs with larger models to see how well they generalize
• Use a truncation objective where there's a capped number of tokens per line, and you do line-level truncation instead of token-level truncation
• Train the matryoshka NLA such that it can write infinite explanation lines (see Training "infinite" matryoshka NLAs above)
• Instead of truncating the AV's explanation, compute the reward by summing at every possible truncation length
  • This would be more efficient (and is still effectively optimizing for the same goal) since the AR is much cheaper to run than the AV (it’s also more sample-efficient but this is less important since there’s more than enough pretraining data available)
  • When using GRPO, you’d want to do proper credit assignment to avoid rewarding tokens after a truncation point
• Normally with GRPO we’re essentially assigning credit to all of the generated tokens, but with matryoshka NLAs we can look at the marginal total FVE (sum of FVE at token position and subsequent tokens) to get the per-token contribution. We could use this to improve RL for matryoshka NLAs by assigning better per-token rewards based on how much that token actually contributed to FVE.
• Use a better metric than reconstruction error during RL
  • It's possible that many things we care about don't take up a large part of activations, and so NLAs might not prioritize verbalizing them. The vector magnitude of features is unlikely to correspond well with how important I think the features are. Improving the metric would also be helpful for standard NLAs too.
• Give each line individually to the AR then sum those individual activations, to try to force each line to convey a separate idea

Conclusion

Matryoshka NLAs are pretty cool and might be worth investigating more. It seems easier to read the shorter bullet points than the larger paragraphs of normal NLAs. It also might be worth changing the warm-start format of even normal NLAs to be more like the format used for matryoshka NLAs: more short bullet points seem more useful than fewer longer paragraphs.

It’s pretty interesting that we can squeeze so much of the reconstruction ability of NLAs into fairly few explanation tokens! They might be more useful than traditional NLAs for some use cases. You can see the models/data (or a smaller Qwen2.5 7B matryoshka model) and code if you want to play with this yourself.

1. ^{^}

   Matryoshka NLAs was a name independently suggested by Adam Karvonen, and something we used while developing the method, though a more applicable name might be “truncation robust bullet-point NLAs”. No strong connection to the matryoshka dolls remains.

2. ^{^}

   This idea was also independently suggested by nostalgebraist and Adam Karvonen in comments on that post.

3. ^{^}

   Without this, RL on the AR was really unstable and the gradients became NaN within a few steps; the NaNs might have been due to an issue with the training code though.

4. ^{^}

   We use the same truncation length for every sample in the GRPO group (so a given explanation is only truncated once), to avoid luck-based reward for being longer.

5. ^{^}

   See earlier post about ablating paragraphs in NLA explanations

6. ^{^}

   We split the NLA explanation into ten chunks with equal numbers of tokens. We also tried using the ten lines (which have varying token lengths), and got essentially the same results

7. ^{^}

   This happens both when you look at it as 10 equal-word chunks and 3 paragraph-chunks, when having a judge model (here, Claude Haiku 4.5) evaluate which chunks mention the concept of the steering vector.

8. ^{^}

   Assuming there are infinite list items

9. ^{^}

   Because OpenAI exposes token probabilities on this model, unlike most other LLM providers

10. ^{^}

    In hindsight it would have been better to use random chat sessions (e.g. from WildChat) to make this a fairer comparison.