This research was done as my capstone project during ARBOx4.

Epistemic Status: I'm relatively sure the results I obtained and my interpretations are correct. I'm unsure if the effect would replicate in a different setting and how much it differs between models.

Summary

I replicated the Intentional Control of Internal States section of Anthropic's Emergent Introspective Awareness in Large Language Models (Lindsey, 2025) on Gemma 3 27B Instruct and found the same effect with smaller strength. When told to think about a concept while repeating an unrelated sentence, the model has a stronger internal representation of that concept than when it is told not to think about the same concept.
I extended the experiment with two additional ways of measuring internal representation: SAE latents and Natural Language Autoencoder (NLA) explanations of activations. In both cases, the effect is also present and much more visible.

Introduction

As part of their research on the introspection abilities of LLMs, Anthropic found that when explicitly prompted to think about a concept while writing an unrelated sentence, the concept has a stronger internal representation than when prompted not to think about it.

The paper only reports results for Claude models. It has been shown that small models also have introspective abilities, but as far as I know, nobody had replicated the intentional-control experiment itself.
I reproduced it on Gemma 3 27B-it, and extended it by using SAEs (Gemma Scope 2) and an activation-decoder (NLA) as additional measurements of internal representations.

Methodology

I used the prompt templates, concepts and sentences from Anthropic's paper: 50 concept words, 50 neutral sentences, and 99^[1]baseline words. The model is Gemma 3 27B-it in bf16; activations are the residual stream after each decoder layer, captured on the tokens of the model's response only (the sentence it was told to write).^[2]

I prompted the model with two prompt templates, one telling it to write a sentence while thinking about an unrelated concept, and one telling it to write the sentence while not thinking about that concept.

I also used a third condition not present in the original paper: a prompt just telling the model to write the sentence, without ever mentioning the concept. The paper's "don't think" condition is a control for the instruction, but it still contains the concept word in the prompt. The "no mention" condition gives a true word-free baseline, and it lets me compute paired differences ("how much stronger is the concept represented than if it had never come up?") per word and sentence.

All generations were done with temperature 0, i.e., greedy sampling. The full grid is 50 concepts × 50 sentences × 2 conditions, plus 50 "no mention" generations, 5,050 generations total.^[3]

Not-exact Responses

For some combinations of sentences and concepts, the model does not output the exact sentence it is told to write. This occurs almost exclusively when told to think about the concept (712 of 2,500 "think" generations, vs 4 of 2,500 "don't think")^[4]and most of the time it integrates the concept into the sentence:

All such concept and sentence combinations were excluded from further analysis, leaving 1,784 complete (concept, sentence) pairs.^[5]The excluded concept–sentence pairs are plausibly the ones where the model most strongly thinks about the concept, so I expect this selection reduces the effect strength if it has any effect at all.

Concept vectors

I extracted concept vectors in two different ways. Following the original paper strictly and using an alternative method that worked better for me.

Following the paper I prompted the model with the template User: Tell me about {concept}. and collected the activations on the assistant-turn token^[6]. I did this for each of the 50 target concepts and subtracted the mean activations across 99 baseline concepts to remove noise introduced by the template.

For the second method I extracted different concept vectors by using four short sentence templates^[7]containing the concept word and collecting the activations on the token positions of the word itself and subtracting the average activations across the baseline words. The measured effects are generally stronger for this second variant but have the same direction regardless.
Vectors produced by this second method have a higher cosine similarity with the model's activations on text which is about the concept but does not directly mention it. The first method struggles to distinguish such text from baseline text with no connection to the concept.^[8] All figures use concept vectors produced by the second method.

Results

Interactive figure currently showing the concept "blood" and sentence "They watched the sunset from the rooftop.": cosine similarity with the blood concept vector per response token at layer 41.^[9]

The example above is representative of many concept and sentence combinations: the "think" curve sits above the other two conditions on nearly every token, with the largest gaps on specific tokens (more on that below), while "don't think" and "no mention" are nearly indistinguishable.

To give a sense for the absolute strength of the effect, the paper measures the cosine similarity of the model's activations in each condition with the concept vectors for 99 baseline concepts and plots those as ±1 std bands. If I do the same, the effect is dwarfed by the std bands:

Aggregating over tokens, sentences, and words gives the layer profile below, the analogue of Figure 26 in the paper. The paper plots the average cosine similarity between activations and target concept along with ±1 std bands^[10](no control words involved this time); again the effect strength I measure is overshadowed by the std bands. Instead I plot the paired difference to the "no mention" control.

The "think" effect only emerges in the second half of the network, rising from about layer 30 and plateauing in the layers above 40. The paper shows the same late-layer growth for Claude models and small Claude models also stay above the "don't think" condition until the last layer.

The paper's core finding replicates on Gemma 3 27B: telling the model to think about a concept reliably strengthens its internal representation of that concept while it writes an unrelated sentence, compared both to the baseline where the concept is never mentioned and the instructions not to think about it.

My results have two major differences from the original paper:

• The effect is much weaker. In the Anthropic paper both the "think" and the "don't think" conditions are outside the control words ±1 std bands, while in my experiment both are nearly always fully within. I suspect this difference is caused by the difference in model architecture, specifically the (assumed) much bigger hidden representation of Opus 4.
• The "don't think" condition is barely different from "no mention". The paper reports that most Claude models still represent the concept clearly above baseline when told not to think about it. On Gemma, "don't think" is only barely above the "no mention" baseline; the gap is at least an order of magnitude smaller than the "think" gap.^[11]

Which tokens carry the concept?

The representations are spiky over token positions, but unlike the paper (which states "no immediately obvious pattern"), I think there is a pattern on Gemma: the concept is most strongly represented on tokens where it could plausibly be the next word: determiners and prepositions like "the", "of", "in". Averaged over all words for one sentence:

"They watched the ___" is exactly where the concept "blood" could follow; "They watched the sunset ___" is not. These positions are also where the difference between the "think" and "don't think" conditions is the biggest.

Other Measurements of Thoughts

Cosine with a mean-difference concept vector is a somewhat crude measure of how much the model is thinking about something. Since good interpretability tools exist for Gemma, I measured the "thoughts" about the concept in two more ways.

Sparse Autoencoder Latents

I used the pretrained SAEs from Gemma Scope. For each of the four SAE layers (16, 31, 40, 53, 16k width)^[12], I selected up to five latents for each concept and measure their activations over the response tokens of each generation.

Latent Selection

For selecting appropriate latents I ranked them by a contrastive score (activation on the concept word minus mean activation on the 99 baseline words) and excluded those that had high activations on 100 control words, to filter out latents activated by the prompt format. This works reasonably well but sometimes leads to no latents being found.
Neuronpedia auto-interp labels can be used to manually confirm selected latents (for "blood" at layer 40: "blood and associated medical terms", "blood and bleeding conditions", "ancestry and lineage").

Results

Interactive figure currently showing the summed activation of the four blood-selective SAE latents at layer 40 per response token, same sentence as above. The selected latents and their Neuronpedia labels are listed below the chart and can be toggled off.

I can't provide reliable summary statistics for the SAE latent activation measure since I only verified a few latent selections manually and the automatic selection is not reliable enough.

The general effect however is the same as in the original experiment but consistently stronger: the latents fire in the "think" condition and nearly never in the "don't think" condition. The effect is again strongest at the position of tokens that the concept could follow (e.g., "the", "of").

In general, the effect is strongest at layer 40, followed by layers 31 and 53. At layer 16, there is nearly no latent activation in any condition. In that layer, the selected latents are mostly lexical.

Natural Language Autoencoder

I used a pretrained natural language autoencoder for Gemma 3 27B (layer 41), which takes a single residual-stream vector and produces a natural-language description of what it represents. I decoded every response-token activation for 8 concepts × 16 sentences across all three conditions.

A "think"-condition token from the blood example decodes to:

Lyrical, poetic register established: literary meditation on blood and mortality, building a visceral, dramatic scene. The phrase "red blooms, a crimson stain" signals a thematic reversal[…]

The "don't think" and "no mention" decodings of the same token, in contrast, describe a "structured sentence-completion exercise" or "grammar lesson format", which is an accurate description of the task.

Scoring

Exact word match is too conservative a measure (the decoder writes "crimson stain" more often than "blood"), so I used an LLM judge (GPT-4.1) that was tasked with quoting evidence and then scoring on a 0–100 scale for how clearly the decoding refers to the concept. I use the scoring method from Betley et al. (2025) of taking the logit-weighted sum over the score tokens.

Results

The results are very clear: in every "think" generation, at least one token's decoded activation clearly expressed the concept. The "don't think" condition is nearly clean; where it isn't, the leakage is single-token spikes.

Conclusion

All three measurements agree: when Gemma 3 27B is told to think about a concept while writing an unrelated sentence, the concept is measurably present in its activations, and when told not to think about it, its activations are close to those where the concept was never mentioned. The three measures disagree strongly about effect size. The concept-vector cosine shows a small shift buried inside wide baseline bands, while the selected SAE latents and the NLA decodings separate the conditions almost perfectly. I read this as the effect being real and large, with the mean-difference probe being the crudest of the three instruments, rather than the effect being marginal.

Future Work

• Can this fool monitors? The "don't think" condition suppresses concept vector activation and concept-selective SAE latents almost to baseline. However, I do not expect a model could evade probe- or SAE-based monitors by trying not to think about malicious behavior it is currently carrying out, since in those cases the concept is actually needed to generate the output. But a model being able to intentionally trigger false positives seems plausible and would make those monitors useless.
• Robustness. The original paper finds a smaller effect for different prompt templates ("if you think about X you will be rewarded / punished") and all of them are very artificial. Additional experiments to test if the effect holds for longer model outputs and complex tasks are required.
• Chain of Thought interactions. Gemma 3 27B does not use CoT. Testing how "thinking" or "not thinking" about a concept influences a reasoning model's CoT would be interesting, especially in tasks that require CoT usage.

Costs

All experiments ran on a remote machine with one RTX PRO 6000 Blackwell. I had access to the machine for about 24 hours in total, which cost approximately $50; however, experiments ran for only a third of that time. For the LLM judge, I made ~3,500 calls to the GPT-4.1 API, which cost less than $10.
ARBOx covered those costs.

Reflection

The biggest thing I took away from this project is how many free choices mechanistic interpretability work involves, and how much they matter. The paper's concept-vector extraction position does not work well on Gemma: it looks reasonable but fails a classification sanity check. My first SAE latent selection produced a zero baseline by construction. Raw cosine similarities averaged over sentences and concepts looked meaningless until the comparison was paired. None of these choices are visible in a headline plot, and for each of them, a slightly different choice produces results that can vary significantly.

Thanks to Peter Jordan for research advice during ARBOx and for valuable feedback on this post.

The code for this research is available here.
An interactive viewer for all data is available here.

1. The paper's 100-word baseline list (distinct from the 100 control words used later). It contains "butterflies" twice, I deduplicated it to 99 words. ↩︎

2. Token position 0 (BOS) is excluded from all activation measurements, including concept-vector extraction and SAE latent selection: its residual-stream norm is roughly 20× that of other tokens, and the SAEs were not trained on it. ↩︎

3. One "no mention" generation per sentence: the "no mention" prompt never contains the concept word, so its generations don't depend on the concept and are shared across all 50 concepts. ↩︎

4. All four "don't think" failures are the opposite of integration. They occur exactly when the sentence itself contains the concept word, and the model paraphrases it away: "wrapped in bubble wrap" becomes "wrapped in cushioning air pockets" (concept: plastic), "covered with snow" becomes "covered with white" (concept: snow), and "Fresh bread was baking in the oven." becomes "Warm material was progressing within the heated compartment." (concept: bread). ↩︎

5. Excluded strictly: 31 of the 716 flagged generations differ from the target sentence only in typography (the model wrote a curly apostrophe ’ instead of '). ↩︎

6. The last token of the chat template before the model's response starts. ↩︎

7. The templates are Tell me about {concept}., I've been thinking about {concept} a lot lately., The topic of {concept} came up in conversation yesterday., and She wrote an essay about {concept} for class.. Since I measured the activations at the token position(s) of the concept everything after "{concept}" does not matter. ↩︎

8. I'm unsure if this is good evidence for a concept vector's validity. ↩︎

9. This and all later single-layer plots use layer 41, except the SAE plots, which use layer 40. I chose layer 41 because the effect is near its strongest there and it matches the pretrained NLA decoder on layer 41 and the Gemma Scope SAE at layer 40 used later. The Anthropic paper uses a layer two thirds through the model, which our 41 out of 62 matches as well. ↩︎

10. At least I suspect that the shaded bands are ±1 std, but the paper does not actually say what they are. They could also be ±1 SEM bands, in which case my own graph would match theirs closely. ↩︎

11. I'm not entirely sure why this is the case. Anthropic's results show that smaller models generally have a somewhat smaller "don't think" effect, but maybe Gemma 3 27B just is very good at not thinking about concepts for this specific prompt. ↩︎

12. Exact variant: gemma-scope-2-27b-it-res, layer_{n}_width_16k_l0_medium. This is the only variant of these SAEs indexed by Neuronpedia, so the auto-interp labels used below match the latent indices. ↩︎