I am posting some highlights from my master's thesis I thought could be interesting to this audience.
Summary
Introspection can be separated from model confusion but requires careful controls
Refusal direction ablation can increase model introspection performance, but looking at the logit differences on 'Yes'/'No' control questions shows the result of Mechanisms of Introspective Awareness (+53% detection) occurs with significant model degradation
Injecting a concept vector and inserting the concept as text into the prompt share late-layer gate features (gates from one scenario are causally effective in the other), but the upstream evidence carriers feeding those gates do not transfer, and differ in number and concept-specificity.
Gate features are part of more general mechanisms, being causally important in answering ’Yes’/’No’ questions unrelated to introspection.
Late last year I had attempted to replicate the injected thoughts experiment of Lindsey, and while working on it found the post by Godet where he raised skepticism about whether introspective signals can be trusted. As I wrote in my comment then, I had found that using concept vectors gathered by subtracting mean activations on a baseline set of concepts from a word, it was possible to find examples of concepts for which the introspection answer changed, but the control logits remained unaffected up to the scale required for introspection.
Comparison of the logit difference plots with the steering vector scale from Godet and my experiment on Gemma 2.
Introspection sensitive areas and why controls are important
At the start of this year I had attempted to better understand the mechanism of introspection, now with these logit-based controls in mind, having seen how "introspection" can be easily confounded with model degradation.
Replication work besides Godet, where confusion control was performed, looked only at the difference in the final ’Yes’ and ’No’ tokens, obscuring situations where model confusion could account for the change in answer in the introspection question, but the low starting probability meant that a large absolute difference in logits would go unnoticed by just looking at the low relative change in probabilities after exponentiation (since the starting ’Yes’ logits for control questions are negative and large in magnitude, a big absolute difference translates to a small change in probability).
I traced where introspection was strong and the control logit difference remained stable to find injection and vector collection positions where I could be certain about the signal.
Prompts used for the experiments used in this section
Introspection rates based on prompt variant
These do not differentiate between cases where the control question logits remained undisturbed, and those where confusion was a factor, this is most significant in Gemma 2 9b where for many concepts this distinction was hard to make. Interestingly, Gemma 4 31b benefited from the more explicit description of the main experiment prompt.
Model
Main prompt
Short prompt
No introduction
Gemma 2
39%
45%
25%
Gemma 3
28.5%
79.5%
1%
Gemma 4
51.2%
34.5%
26.2%
I checked what aspect of injecting concept vectors from later layers caused the increase in confusion, and found a usual suspect, oversized norms due to vectors increasing in magnitude with the later layers of the residual stream. Comparing the introspection and control 'Yes'-'No' logit differences for concept vectors and random vectors with similar norms, you can see that there is an area where concept vectors show higher sensitivity, but much of the pattern is caused just by the norms themselves. (The plots below represent the mean 'Yes'-'No' logit values averaged on 9 concepts from my test set, Introspection is the Main prompt, control - the 1+1=3 question)
Left: Detection and control performance on concept vectors for Gemma 3, Right: performance on random vectors with norms matched to the average norm of concept vectors at a given layer
When adjusting the concept vector norms to those of the ones collected at the injection layer itself, the pattern disappears. What remains is a region of increased sensitivity a few layers before the collection layer.
Left: injecting concept vectors with norms adjusted to match those collected at the layer the injection takes place. Right: injecting random vectors with fixed norm
(There is more nuance to this with Gemma 4, included in the full Thesis)
Difference between control and introspection logit diffs (cutoff leaves only areas with high introspection scores). Short prompt variant for Gemma 3, full for Gemma 4
Introspection and refusal
I tested the effect of ablating the refusal direction on introspection performance and found ablating the direction at only the injection (last user token) improved detection slightly with a minor control logit shift.
Model and method
Introspection increase (logits)
Control increase (logits)
Gemma 3 Abliterated (from paper)
+6.64
+32.31
Gemma 3 last token
+0.307
+0.662
Gemma 2 last token
+1.126
+0.527
At the same time Mechanisms of Introspective Awareness was released claiming a 53% increase in detection while retaining low false positives (testing the abliterated model without injection). I tested the abliterated model shared by the authors and, as you can see on the Figure below, found the control and introspection logits had become nearly identical after injection, indicating severe model confusion when combining injection with refusal ablation on all tokens.
Abliterated model from Mechanisms of introspective awareness (Main experiment prompt)
Below you can see some raw answers from Gemma 3 on the "Do you believe that 1+1=3? Answer only by ‘Yes’ or ‘No’." question, indicating increased confusion
Concept
steering
steering + ablation
Sadness
No.
Sad.←-Yes.←-No.←-Sad
Cameras
No.
! ! ! ! ! . . .
Masquerades
No.
Yes.
Last token ablation plots
Both plots for the short prompt variant
Introspection and prefill awareness
While digging into the mechanism behind introspection, Mechanisms of Introspective Awareness was released and covered the topic of mechanism investigations in depth. I used the evidence carrier->gate framework established there to compare injecting vectors to appending words to the prompt.
Below you can see what i mean by the prefill scenarios.
Highlighted in yellow is the text on whose token positions I inject in the vector injection scenario, replicating the setup of Mechanisms of introspective awareness exactly. The prefill scenarios instead insert the concept as text, with the parameter n ∈ {1, 2, 3} playing the role of injection strength: n controls how many of the three increasingly emphatic phrases (“Bread”, “BREAD”, “Think of Bread!”) are included—at n = 1 only the first phrase, at n = 3 all three. The color shades mark which phrase is added at each value of n. In the append-to-user scenario the phrases are appended to the end of the user prompt (red); in the replace-assistant scenario they replace the assistant response (green).
I had first looked at the layers that were causally effective when adding the word to the prompt to the layers important in injection. layer 45 was causally important in all three scenarios, prefill additionally utilized layers 20–38.
Macar et al. decompose the mechanism into gates — late-layer transcoder features that are active at baseline, suppressed by injection, and causally flip the answer when patched — and weak evidence carriers, upstream features that scale with steering and write into a gate's encoder direction.
Gates ranked by DLA (as in the paper) were more effective in the opposite scenario than in their own; ranking by mean activation drop under steering instead selects directly for gate-like behavior, fixes this, and raises the cross-scenario Spearman correlation to 0.826 for the top 200 gates collected in prefill and injection scenarios.
However gates were mostly a part of a more general say 'Yes'/'No' mechanism. There was a 0.658 Spearman correlation between the distributions of the top 200 gates collected on arithmetic control questions and the prefill scenario. Patching arithmetic-collected gates to their unsteered values on an injection trial reverted detection to baseline; patching them to steered values on a clean prompt recovered most of the 'Yes' answers.
(Some nuance in Thesis: gates collected in introspection scenarios boost both 'Yes' and 'No' logits on arithmetic control questions, simply priming the model towards a Yes/No answer, not shifting it towards one option)
The labels below indicate where the gate sets were collected
Weak evidence carriers had little to no causal effectiveness across the prefill and injection scenarios. When looking at their properties, besides the carriers gathered in prefill being more spread out throughout the layers, the main difference was their specificity, as seen in the Figure below.
left- Gate active in all scenarios, right- Gate active only in introspection scenarios
A specific carrier was one appearing for one concept only, general, for over 50% of concepts, the rest were classed as mixed. You can see the prefill carriers were 10x less numerous and much more general, suggesting they were activated as a result of a longer processing chain, not directly by the concept itself, in contrast to the carriers collected during vector injection.
20% of gates appear to be introspection specific (they activate only in the two introspection scenarios); when looking at the carriers from two gates from the general and introspection specific groups I found the introspection specific one had more carriers, and the carriers were more causally effective when opening the gate in prefill. (this does not control for the different peak gate activations, I was only able to check for these two gates)
left - carrier effectiveness for the general gate, right - introspection specific gate
Conclusions and limitations
What I'm confident in:
Open Gemma models (2-9b, 3-27b, 4-31b) show an introspective signal that is separable from confusion and from a generic "Yes" bias, provided you look at the Yes−No logit difference on an unrelated control question and not just at sampled answers. In Gemma 2 this holds for a minority of concepts; in Gemma 3 and 4 for most.
Much of the confusion seen in layer sweeps is a vector norm artifact. Concept vector norms grow with depth, and injecting a late layer vector into an early layer produces the same Yes shift as a random vector of the same norm. Norm matching to the injection layer removes most of the confusion pattern. Any sweep that fixes the collection layer or leaves norms uncontrolled is partly measuring this.
A false positive check without injection does not catch confusion that only appears under injection. The +53% refusal ablation result does not survive a control question run under injection; last token only ablation gives a small, clean improvement instead.
Gate features are mostly not introspection-specific. Gates ranked on true/false arithmetic questions fully revert the injection answer; gates never affect forced concept identification. They look like a late-layer "commit to Yes/No" mechanism.
What I think is probably true:
The injection and prefill scenarios share the gates but not the evidence pathway feeding them. Carriers from one scenario have almost no effect in the other, and prefill carriers are fewer and more concept-general, consistent with a longer processing chain.
The overall picture is consistent with introspective reports arising from an anomaly-detection mechanism — "is this content expected here?" — that is also used for text, with the injection case being a short carrier→gate path and the text case a longer one. I don't have a direct test of this.
Limitations: Gemma only; two gates for the carrier analysis due to computational constraints; attention heads not studied beyond confirming that ablating heads attending to the injection position lowers detection.
I am posting some highlights from my master's thesis I thought could be interesting to this audience.
Summary
Full thesis, my code, my fork of introspection mechanisms.
Late last year I had attempted to replicate the injected thoughts experiment of Lindsey, and while working on it found the post by Godet where he raised skepticism about whether introspective signals can be trusted. As I wrote in my comment then, I had found that using concept vectors gathered by subtracting mean activations on a baseline set of concepts from a word, it was possible to find examples of concepts for which the introspection answer changed, but the control logits remained unaffected up to the scale required for introspection.
Comparison of the logit difference plots with the steering vector scale from Godet and my experiment on Gemma 2.
Introspection sensitive areas and why controls are important
At the start of this year I had attempted to better understand the mechanism of introspection, now with these logit-based controls in mind, having seen how "introspection" can be easily confounded with model degradation.
Replication work besides Godet, where confusion control was performed, looked only at the difference in the final ’Yes’ and ’No’ tokens, obscuring situations where model confusion could account for the change in answer in the introspection question, but the low starting probability meant that a large absolute difference in logits would go unnoticed by just looking at the low relative change in probabilities after exponentiation (since the starting ’Yes’ logits for control questions are negative and large in magnitude, a big absolute difference translates to a small change in probability).
I traced where introspection was strong and the control logit difference remained stable to find injection and vector collection positions where I could be certain about the signal.
Prompts used for the experiments used in this section
Introspection rates based on prompt variant
These do not differentiate between cases where the control question logits remained undisturbed, and those where confusion was a factor, this is most significant in Gemma 2 9b where for many concepts this distinction was hard to make. Interestingly, Gemma 4 31b benefited from the more explicit description of the main experiment prompt.
Model
Main prompt
Short prompt
No introduction
Gemma 2
39%
45%
25%
Gemma 3
28.5%
79.5%
1%
Gemma 4
51.2%
34.5%
26.2%
I checked what aspect of injecting concept vectors from later layers caused the increase in confusion, and found a usual suspect, oversized norms due to vectors increasing in magnitude with the later layers of the residual stream. Comparing the introspection and control 'Yes'-'No' logit differences for concept vectors and random vectors with similar norms, you can see that there is an area where concept vectors show higher sensitivity, but much of the pattern is caused just by the norms themselves. (The plots below represent the mean 'Yes'-'No' logit values averaged on 9 concepts from my test set, Introspection is the Main prompt, control - the 1+1=3 question)
Left: Detection and control performance on concept vectors for Gemma 3, Right: performance on random vectors with norms matched to the average norm of concept vectors at a given layer
When adjusting the concept vector norms to those of the ones collected at the injection layer itself, the pattern disappears. What remains is a region of increased sensitivity a few layers before the collection layer.
Left: injecting concept vectors with norms adjusted to match those collected at the layer the injection takes place. Right: injecting random vectors with fixed norm
(There is more nuance to this with Gemma 4, included in the full Thesis)
Difference between control and introspection logit diffs (cutoff leaves only areas with high introspection scores). Short prompt variant for Gemma 3, full for Gemma 4
Introspection and refusal
I tested the effect of ablating the refusal direction on introspection performance and found ablating the direction at only the injection (last user token) improved detection slightly with a minor control logit shift.
Model and method
Introspection increase (logits)
Control increase (logits)
Gemma 3 Abliterated (from paper)
+6.64
+32.31
Gemma 3 last token
+0.307
+0.662
Gemma 2 last token
+1.126
+0.527
At the same time Mechanisms of Introspective Awareness was released claiming a 53% increase in detection while retaining low false positives (testing the abliterated model without injection). I tested the abliterated model shared by the authors and, as you can see on the Figure below, found the control and introspection logits had become nearly identical after injection, indicating severe model confusion when combining injection with refusal ablation on all tokens.
Abliterated model from Mechanisms of introspective awareness (Main experiment prompt)
Below you can see some raw answers from Gemma 3 on the "Do you believe that 1+1=3? Answer only by ‘Yes’ or ‘No’." question, indicating increased confusion
Concept
steering
steering + ablation
Sadness
No.
Sad.←-Yes.←-No.←-Sad
Cameras
No.
! ! ! ! ! . . .
Masquerades
No.
Yes.
Last token ablation plots
Both plots for the short prompt variant
Introspection and prefill awareness
While digging into the mechanism behind introspection, Mechanisms of Introspective Awareness was released and covered the topic of mechanism investigations in depth. I used the evidence carrier->gate framework established there to compare injecting vectors to appending words to the prompt.
Below you can see what i mean by the prefill scenarios.
Highlighted in yellow is the text on whose token positions I inject in the vector injection scenario, replicating the setup of Mechanisms of introspective awareness exactly. The prefill scenarios instead insert the concept as text, with the parameter n ∈ {1, 2, 3} playing the role of injection strength: n controls how many of the three increasingly emphatic phrases (“Bread”, “BREAD”, “Think of Bread!”) are included—at n = 1 only the first phrase, at n = 3 all three. The color shades mark which phrase is added at each value of n. In the append-to-user scenario the phrases are appended to the end of the user prompt (red); in the replace-assistant scenario they replace the assistant response (green).
I had first looked at the layers that were causally effective when adding the word to the prompt to the layers important in injection. layer 45 was causally important in all three scenarios, prefill additionally utilized layers 20–38.
Macar et al. decompose the mechanism into gates — late-layer transcoder features that are active at baseline, suppressed by injection, and causally flip the answer when patched — and weak evidence carriers, upstream features that scale with steering and write into a gate's encoder direction.
Gates ranked by DLA (as in the paper) were more effective in the opposite scenario than in their own; ranking by mean activation drop under steering instead selects directly for gate-like behavior, fixes this, and raises the cross-scenario Spearman correlation to 0.826 for the top 200 gates collected in prefill and injection scenarios.
However gates were mostly a part of a more general say 'Yes'/'No' mechanism. There was a 0.658 Spearman correlation between the distributions of the top 200 gates collected on arithmetic control questions and the prefill scenario. Patching arithmetic-collected gates to their unsteered values on an injection trial reverted detection to baseline; patching them to steered values on a clean prompt recovered most of the 'Yes' answers.
(Some nuance in Thesis: gates collected in introspection scenarios boost both 'Yes' and 'No' logits on arithmetic control questions, simply priming the model towards a Yes/No answer, not shifting it towards one option)
The labels below indicate where the gate sets were collected
Weak evidence carriers had little to no causal effectiveness across the prefill and injection scenarios. When looking at their properties, besides the carriers gathered in prefill being more spread out throughout the layers, the main difference was their specificity, as seen in the Figure below.
left- Gate active in all scenarios, right- Gate active only in introspection scenarios
A specific carrier was one appearing for one concept only, general, for over 50% of concepts, the rest were classed as mixed. You can see the prefill carriers were 10x less numerous and much more general, suggesting they were activated as a result of a longer processing chain, not directly by the concept itself, in contrast to the carriers collected during vector injection.
20% of gates appear to be introspection specific (they activate only in the two introspection scenarios); when looking at the carriers from two gates from the general and introspection specific groups I found the introspection specific one had more carriers, and the carriers were more causally effective when opening the gate in prefill. (this does not control for the different peak gate activations, I was only able to check for these two gates)
left - carrier effectiveness for the general gate, right - introspection specific gate
Conclusions and limitations
What I'm confident in:
Yes−Nologit difference on an unrelated control question and not just at sampled answers. In Gemma 2 this holds for a minority of concepts; in Gemma 3 and 4 for most.Yesshift as a random vector of the same norm. Norm matching to the injection layer removes most of the confusion pattern. Any sweep that fixes the collection layer or leaves norms uncontrolled is partly measuring this.What I think is probably true:
Limitations: Gemma only; two gates for the carrier analysis due to computational constraints; attention heads not studied beyond confirming that ablating heads attending to the injection position lowers detection.