Natural language autoencoders are a really cool mostly-unsupervised method for producing free-form text explanations of LLM activations. You should read that paper (or the blog post) about them before reading this.

I trained^[1] several Qwen3-8B NLAs with different length penalties: during RL, I subtracted the token count multiplied by the length penalty hyperparameter (λ) from the RL reward^[2]. I found that with small length penalty (λ=0.002), you can reduce the length of NLA explanations by ~40% (compared to having no length penalty) with a fairly small hit to FVE (fraction of variance explained: 0 is guessing the mean activation, 1 is perfect reconstruction) of -0.015. With an even smaller penalty (λ=0.001) the FVE is almost unchanged (+0.007) despite explanations using 28% fewer tokens.

Being able to reduce the length so much without impacting FVE is interesting because it could mean that large parts of NLA explanations aren't actually useful for reconstructing the input activation faithfully. Some of this is because the length penalty makes the model use terser wording to convey the same ideas; I'm not sure how much of these results stem from terser wording vs omitting unneeded information.

Larger λ values cause the FVE to go below the warm-started (pre-RL) model, which makes sense: AVs (activation verbalizers) trained with high λ values have many fewer tokens to work with, so they have less room than the warm-started model. It's interesting that they're still pretty good (relative to λ=0) though!

There are two main reasons the AV writes explanations that are much longer than they need to be with standard NLAs:

1. The KL penalty pushes the AV to write like the warm-start model, and the warm-start model isn't trying to be concise.
2. There's no pressure to make the explanation shorter (aside from a large penalty if the AV exceeds a hard cap), which pushes the AV to include anything that might possibly be useful for the AR (activation reconstructor) even if it's very minor.

The optimal explanation length probably varies a lot by model. Bigger models think about more stuff and might have less unused "slack" in their activation explanations. It would be pretty interesting to see what that curve from my graph above would look like with multiple models, across more penalties, and with more RL steps. Note that I did 250 RL steps for all of those penalties, but the RL runs for the higher penalties completed faster because a large part of the time spent in RL is AV explanation generation which is faster when you're generating fewer tokens; it might have been more fair to give each RL run the same amount of time rather than the same amount of RL steps.

Look at some activations!

Here's a widget I vibed up to let you see some explanations from the NLA I trained! I think it's pretty interesting to look at a few of these.

The really high length penalty NLAs mostly end up only including the most important bits of the explanation. The highest length penalty one (λ=0.03) usually only includes a few words that repeat the tail of the input text (or just the final token), and occasionally it gives an empty explanation. This makes sense: the input text itself is very useful for reconstruction and if you only have a few tokens of explanation just repeating the final few seems pretty useful.

It's important to note that while NLA training encourages better reconstructing of the activation as a whole, it doesn't necessarily encourage reconstructing the bits that us humans care about the most! Repeating the final token is not interesting for interpretability (we already know that), but very useful for reconstructing activations. It's pretty plausible that things like eval awareness only make up a fairly small part of activation space but take up a larger part of the space of things I care about.

What does this mean

This might mean that a large portion of what AVs say aren't actually useful for reconstructing activations, but are instead just there to help satisfy the KL penalty and because there's not a strong pressure to omit useless bits.

It might also be interesting to look at what kinds of things get dropped with more severe length penalties more systematically. It would also be interesting to more systematically check if steganography is happening here (like they do in the NLA paper).

Other notes

Training dynamics

The explanation length didn't plateau during the 250 steps of RL I ran for λ=0.001 or λ=0.002; with more training you could probably get even more token-efficient explanations.

It would be interesting to see how the length penalty vs FVE tradeoff changes with more RLing.

Models

I put all of the models and data I used to create this post on Hugging Face.

Conclusion

It might be good to have a smallish length penalty when training NLAs to try to get them to be more faithful and avoid spurious claims. It would also be interesting to try this with larger models and more penalty values. Thanks to Celeste and @jim for giving me feedback on this research.

1. ^{^}

   I used Celeste's nanoNLA instead of the original NLA code because it made the infra simpler to manage. nanoNLA isn't an exact reimplementation of the original but I don't think any of the changes would affect the results I got much.

2. ^{^}

   This doesn't change the warm-start SFT phase of training an NLA, so I reused the same warm-started NLA for everything.