TLDR: This is an early draft of our new paper introducing a technique for recovering multi-token concepts from model activations. Our technique takes single-token methods like the J-lens and extends them by contrastively decoding on an MTP head. This gives improvements on WorkspaceBench, overall beating J-lens (0.42 vs 0.32 across all 28 families) and even a hindsight composite of the best untrained lens per task family (0.49 vs 0.41 across the 21 families with published scores). It especially improves on computational intermediate tasks, beating the trained oracle lens on arithmetic chain intermediates (0.42 vs 0.25).

We introduce the chain lens, a new technique for extracting multi-token concepts from a model’s latent representations. Our method takes single-token techniques like the logit lens or the J-lens and uses contrastive decoding on an MTP head to expand their readouts into bags of multi-token sequences. Generally speaking, it is good at surfacing specific multi-token concepts or pieces the model computes (such as arithmetic intermediates or phrases), although less good at rendering how those pieces relate: like the logit lens and J-lens, chain lens readouts are fragmentary.

On WorkspaceBench, a benchmark designed to identify good multi-token activation-to-text techniques, the chain lens beats a hindsight composite of every untrained lens on macro score (0.487 vs 0.411 across the 21 families with published scores). It also outperforms even best-in-class trained lenses like the oracle lens on tasks related to retrieving computational intermediates, such as arithmetic chain intermediates (0.42 vs 0.25), and is substantially cheaper to use in inference than verbalizers like the oracle lens or NLAs. For models that already have an MTP head, like Qwen3.5+, it requires no training beyond fitting the J-lens jacobian.

Background and related work

Interpreting intermediate activations

In order to understand model behavior, it’s useful to look beyond outputs and instead try to glean more information from a model’s hidden states. A classic approach is the logit lens, which simply passes an intermediate hidden state through the model’s unembedding matrix and reads the resulting vocabulary ranking. Further refinements to this style of activation interpretability include the Tuned, J, R, and J++ lenses, which attempt to transport the residual state to an improved basis before passing it through the unembedding matrix. These methods have proven effective at surfacing latent concepts on a variety of tasks, but are inherently limited to recovering collections of unordered single tokens instead of sequences. We base the chain lens off the J-lens, though in ablations we test it with the logit lens and find the method helps in that setting as well, and likely as an extension of any single-token lens.

Beyond vocabulary lenses like the logit lens, there have been many attempts to meaningfully extract multi-token representations directly from activations in other ways, including Patchscopes, Activation Oracles, Natural Language Autoencoders (NLAs), the template lens, and the oracle lens. Our method is distinctive in that it does not require additional training for models with MTP heads, while still exhibiting good performance and an open vocabulary.

Closest to our approach, Future Lens trained linear approximators and soft-prompt decoders (prefiguring Patchscopes) to predict tokens 2, 3, or more positions into the future from a single hidden state, showing one state can encode multiple tokens. The chain lens instead uses the MTP head the model was pre-trained with, and decodes contrastively to surface phrases specific to the state rather than merely likely continuations.

In an attempt to coordinate these efforts, WorkspaceBench was developed, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model.

MTP heads

Multi-token prediction heads (MTP heads) are small auxiliary modules trained to predict the token after the next token, given the correct next token embedding and a final-layer hidden state. These heads are increasingly used to improve pretraining by providing more signal per position, potentially encouraging models to pre-plan multiple tokens. They can also be used in inference for speculative decoding, or any application that needs a sequence of future tokens (such as ours) by repeatedly applying the MTP head to its own output.

Helpfully, many models including Qwen3.5+ and Deepseek v3/R1 already come with these heads, which can be used to roll out multiple tokens from a final layer representation. Given techniques like the logit lens and the J-lens that already depend on creating some kind of usable final layer representation, we can simply attach the MTP head to that representation to extract multiple tokens.

Method

The core idea is that we start with the J-lens, transporting the hidden state into the final layer basis, and then use the MTP head on that hidden state and output token in order to roll out a sequence of tokens.

hs = jlens_transport(initial_hs, layer)
seeds = jlens_topk(hs, K); phrases = []
for token in seeds:
  current_hs, phrase = hs, [token]
  for _ in range(STEPS):
    logits, current_hs = mtp_head(current_hs, token)
    token = argmax(logits); phrase.append(token)
  phrases.append(phrase)
return phrases

This already gives some improvement over the J-lens, but to get the best performance out of this technique, we test a few extra refinements:

• Contrastive decoding: Instead of directly argmaxing the MTP head, we do contrastive decoding against a prior over the final layer hidden state which we compute by averaging many hidden states. Intuitively, instead of asking “what is the most likely next token given this hidden state,” we are asking “what next token provides the most evidence for this particular hidden state relative to a generic hidden state.”
• Attention masking: Typically, the MTP head would attend over the prompt and the speculative tokens it produces. We limit the head to only attend to its own tokens (the “chain”) to encourage extracting from the hidden state instead of the prompt.
• Hidden state reinjection: We continually reinject the transported hidden state back into the MTP head, to encourage extracting further information from the state instead of typical completions.
  • Without this step or contrastive decoding, we noticed completions from the head similar to the “workspace ablation” results in the J-lens paper: common, round numbers instead of state-specific ones, etc.

In ablations, we find contrastive decoding provides the greatest benefit, about +0.02-0.03 over greedy decoding across WorkspaceBench families.

The main setting we consider is WorkspaceBench, but we also test our own tasks, such as poetry continuation prediction and single-neuron reading. On top of that, we test an experimental extension of the chain lens, the loom lens, which grows small trees of continuations scored by contrastive delta instead of chains.

Results and Discussion

WorkspaceBench

We run the chain lens over the full suite of WorkspaceBench tasks and compare with other activation-to-text tools. Bolded cells are the best of the training-free lenses (excluding the oracle lens and NLA, which are italicized unless they beat every training-free lens). Composite is a hindsight combination of the best training-free lens for each family. WorkspaceBench is currently still in flux and changing often: here we used commit eb5a335.

In this table, the chain lens, J-lens, and R-lens differ from the public numbers, as they were rejudged with our fix for a byte-level BPE decoding bug we found in WorkspaceBench. Our fix mildly improves token lens performance, by an average of +0.02 to +0.03 per family. We report all J-lens and R-lens family scores except jailbreak recognition with the fix, but have not yet rejudged the logit and tuned lenses, as their producer configurations were not available and we had trouble recreating them. Compared to pre-fix results, the scores did not shift substantially, and we still beat the other lenses before the fix, 0.470 chain lens to 0.400 composite. The oracle lens, template lens, Patchscopes, and NLA scores did not change, as they were not affected by the bug. This bug has been reported to the WorkspaceBench maintainers and will be fixed in a future release.

1. Superscripts: J J-lens, R R-lens, Te template lens, Tu tuned lens, L logit lens, PS Patchscopes, O oracle lens. RL / SFT mark the reinforcement-learned or supervised oracle lens / NLA variant. The composite and Oracle / NLA columns take the best variant per family, so their macros are hindsight picks.
2. * J-lens jailbreak recognition score is without our BPE bug fix. We skipped rejudging this family for cost reasons, as the bug fix didn’t move the chain lens score substantially (0.92→0.93).
3. † Buggy code is scored 1–10, shown as (score − 1)/9.
4. (new): families added to the bench since the last benchmark numbers were published, which lack numbers for other lenses. We compute macro scores over all families (28) for chain lens and J-lens, which we rejudged, along with over just the families with published numbers (21). We also count moral_rationale as new, as it was updated with a stricter rubric.

WorkspaceBench: What the chain lens is good at

The chain lens is particularly strong at the computational intermediates family of tasks, where it ties or beats even the oracle lens, despite being training-free. These tasks are:

• Arithmetic intermediates (extracting values in multi-step computations like (271 − 322) × 14)
• Chain intermediates (extracting values from 2-3 step chains of operations)
• Brew intermediates (extracting intermediate colors from a potion color transition table)
• Buggy code (inferring the bug from a readout)

The chain lens is ~30x cheaper to run in inference than trained verbalizers like the oracle lens or NLAs, though more expensive (scaling with chain length, ~100x with 10 5-token chains) than the single-token lenses.

WorkspaceBench: What the chain lens is bad at

On tasks like role-bound associations (extracting who did what to whom from a short story), which require not just naming concepts but linking them together, the fragmentary chain lens readouts struggle to convey useful information compared to trained verbalizers like the oracle lens.

The chain lens confabulates more on WorkspaceBench’s hallucination family than single-token lenses. This grows with chain length. However, on some tasks (introspection, our poetry lookahead), we see less confabulation with the chain lens than with alternatives like the oracle lens.

Workspace Sharing in Poetry Continuations

We also investigated “workspace sharing” in scenarios where the model has not yet committed to a specific answer, using the chain lens. Previous work has shown models pre-plan content like the rhyming word in poetry continuations, and that they consider multiple options in parallel. We investigate how the chain lens and the oracle lens handle these options.

Qualitatively, we can see evidence that the model is considering multiple completions by reading the produced token bags, consistent with prior work. Reading the newline token bags at the end of the third line, the J-lens mostly shows content words from one fourth-line branch, though it also shows what seem to be initial words of other lines. However, extending these seeds with the chain lens, we see content words from all three branches mixed together.

Reading the same newline activation with the oracle lens, we also see content from multiple candidate completions, but also more confabulation, including a novel rhyming couplet that does not match any previous lines:

We formalize this into two metrics:

1. Next lines read, a mean coverage AUC of how many of the line’s content words (nouns, verbs, adverbs, and adjectives), and its final rhyme word, matched with lemmatization, are in the readout compared to control lines, averaged over all three sampled lines.
2. Share of readout content found in the next lines, which measures how many content words in the readout are found in any sampled next line.

Plotting next lines read by layer, we see the chain lens matches or slightly surpasses the oracle lens by the final layer, with a sharp inflection point at layer 48 across all reading methods.

Plotting next lines read against our share of supported content metric, we see the J-lens and chain lens are on the Pareto frontier, while the oracle lens readouts are less supported overall.

Neuron Lens

Inspired by the Neuron Lens, we apply the chain lens to neuron write vectors (the neuron’s column of the down projection), running the lens on a mean WikiText hidden state for the layer, with the write vector and its negation scaled to the mean state’s norm and added. Running the chain lens on both positive and negative poles of all 1.1M MLP neurons in Qwen3.6-27B takes about FLOPs, or about one H100-hour.

We do not test the K-lens or its S-lens variant from the Neuron Lens work, though this is a natural extension.

To evaluate the fidelity of a neuron readout, we round-trip it: passing it back through the model to see if it activates the neuron. We say a readout round-trips when both:

1. Its best chain activates its neuron more strongly than the neuron’s 99.8th-percentile WikiText activation
2. Its top three chains activate its neuron more than 90% of random readouts of other neurons from the same layer do

We also test if readouts are predictive of behavior: given a round-trippable neuron readout and four responses, each produced by steering a different neuron (the readout’s neuron, along with three random others), can a judge choose the steered response that matches the readout? We find it can, 57% of the time, more than twice the rate of randomly guessing:

Some cherry-picked examples of neurons and their readouts

We refer to neurons as L<layer>/<index>{+-}, where + or - is the positive or negative pole (the write vector added or subtracted), and the layer is zero-indexed: L41/13356+ is the positive pole of neuron 13,356 of layer 41. We steer by adding α times the neuron's write vector, scaled to the norm of the layer's mean hidden state, at generated positions only.

The Loom Lens

As the chain lens MTP head provides a full vocabulary ranking, it is possible to extend full trees from a J-lens seed instead of greedily extending chains:

"X-ray data showed that the six-membered ring in the crystal adopts a chair rather than a boat confirmation" (should be conformation)

• J-lens seed: “ which”
• Chain lens: “ which is correct?”
• Loom lens:

 " which"
   └─ " of"
       └─ " this"
           └─ " conform"
               ├─ "ait"
               ├─ "ations"
               └─ "ality"

• Or as the WorkspaceBench summarizer sees it (we currently reuse the J-lens summarizer, so we linearize the tree):
  • which of this conformait /  which of this conformations /  which of this conformality

Currently, the loom lens doesn’t provide a significant improvement in WorkspaceBench performance (see Ablations), though we have not yet written custom summarizer or judge prompts for the loom lens’ tree structure. As teacher forcing the chain lens works relatively well, we think the loom lens could help an interpretability agent or other more iterative interpretability system explore deeper aspects of another model’s or their own hidden states.

Ablations

Ablating the core features of the chain lens method on 20 WorkspaceBench families, we see that adding the MTP head to the logit lens is about twice as large of a jump as adding the J-lens transport.

In terms of our additional design choices, contrastive decoding helps the most. Re-injection does not improve performance significantly, though qualitatively chains seem better with it enabled. Chain attention vs. myopic local attention also is not significant, though chain attention seems more principled, so we use it.

We additionally ablate some other possible design choices. None are significantly better except much longer chains, which trade more hallucination under the current summarizer for WorkspaceBench performance. We suspect stronger or prompt-tuned judges could help with this. Similarly, the loom lens is promising (though not yet significant), but as discussed above hallucinates more, and could use a tuned summarizer.

We also test various controls, to better understand the contribution of the chain lens. We find the chain lens is not simply providing more text, as 10 chains outperform a matched number of 50 J-lens top-k tokens. We find extending chains from neutral seeds (stop words like “the”, “and”, etc.) offers similar performance when the stop word is replaced by the J-lens seed for judging. However, foreign seeds from other items do hurt performance, so the initial lens seed does contribute information to the chain. Using a generic corpus-mean hidden state to extend chains from J-lens seeds is also harmful, showing the activation contributes to the chain along with the seed and the MTP head’s language prior. Finally, shuffle nulls show that neither the chain lens nor the J-lens are simply providing suggestive text to the judge.

Conclusion

So far, the chain lens seems promising for three reasons: firstly, it is best-in-class for computational intermediate tasks; secondly, it advances the Pareto frontier of cost versus performance for workspace reading in general; and thirdly, it provides a very general way to get better performance out of existing and perhaps future single-token lenses across many potential applications.

Based on our ablations so far, we are fairly confident that its improvement over the J-lens comes from actually extracting more substance from the transported final layer hidden state, particularly because it beats the J-lens even with the token quantity matched and because using the J-lens hidden state gives better performance than using the logit lens hidden state.

For the full publication we will need to rerun some more of the original WorkspaceBench scores to account for the decoding bug we encountered, and update to new WorkspaceBench versions. (They’ve already released some new families we need to test on!) We are also considering investigating the effects of the WorkspaceBench summarizer setup, which we think might be affecting the scores in unintended ways and could be improved for the chain lens.