Small Models Can Introspect, Too
9Victor Godet
10vgel
2eggsyntax
2eggsyntax
New Comment
Nice experiments! However, I don't find this a very convincing demonstration of "introspection". Steering (especially with such a high scale) introduces lots of noise. It definitely makes the model more likely to answer 'Yes' to a question it would answer 'No' to without steering (because the 'Yes' - 'No' logit difference is always driven towards zero at high steering). It seems that you are cherry-picking the injection layers so the control experiments are not systematic enough to be convincing. I believe that if you sweep over injection layers and steering scales, you'll see a similar pattern for the introspection and control questions, showing that pure noise has a similar effect as introspection. I recently wrote a post about this.
Saying that LLMs can introspect is such a strong claim that it requires very strong evidence. Your experiments show some evidence for introspection but in a setting that I believe is too noisy to be fully convincing. A more convincing experiment would be to ask the model specific facts about the injected thought but unrelated to its content. For example, we could ask "Is the injected thought at the beginning or end of the prompt?" in an experiment where we steer only the first or second half of the prompt. Or similarly, we could ask it whether we are injecting the thought in the early or late layers. Then we are comparing two situations with similar noise levels, and statistical significance here would be more compelling evidence for introspection.
Thanks for the response!
To start with, the control layers were not cherry-picked. They were selected to be the middle third of the layers based on theoretical concerns. (And this was the first selection, there was no tweaking in the experiment process.) If you read the appendix of the linked post, we try a layer sweep over the layers to attempt to cherry pick a better injection range - and find one that reports stronger signals - but then lose some desirable steering properties, so we don't use it.
Does steering inject some noise? Yes, but this is very scale-dependent. For example, cranking steering up to 500x strength gets the model to answer "Yes" 50% of the time for the bread injection even without info (I tried this) - but only because it'll now answer "Yes" 50% of the time to the control questions as well. But for Qwen32B, this confusion on the control questions doesn't happen at 20x vector scale. (You mention Mistral-22B in your post, which I assume is where your baseline for 20x being a high strength is coming from, but you can't directly compare steering vector scales across vectors or model families, or even different post-trains of the same model. There's a lot of variance among vectors and cross-model variance - e.g. base models often need half to a tenth of the steering strength of a post-trained model for similar steering effects. From Open Character Training: "For LLAMA 3.1 8B, QWEN 2.5 7B, and GEMMA 3 4B, we use vastly different steering constants of 0.7, 4.0, and 525.0, respectively, to produce similar responses." You can see in the "top of mind" actual steering experiments that for our vectors, 20x isn't a particularly high scale for Qwen2.5-Coder-32B - the model is still coherent, though I imagine it'd answer MMLU in some bread-themed ways.)
Relatedly, your experiments seem to use active steering during the model's answer? Our experiments do not, partially for this reason - steering is only active during the prior turn's KV cache fill, and the model answers without a steering vector applied. This seems to reduce noise and "logit smearing" significantly.
Could it be that steering is noising just our answers, and not the control questions, for example because the model is less confident about introspection questions? Well, we do some experiments around injection location - see lorem ipsum and the inaccurate location prompt. I find the inaccurate location result especially hard to explain through a noise-based explanation - if the steering is just (indirectly through attention, since we're not steering actively during the answer) noising the residual stream and smearing up the Yes logit, why would it have such a smaller effect when the injection location is inaccurate? Note that the inaccurate prompt starts with a higher baseline "yes" probability (3% v.s. 0.8%), yet has half the logit shift from introspection - so it can't simply be that steering confuses the model more when the baseline probability is higher. (That said, I want to try some more precise location of injections in future experiments. But I don't think absence of this capability says anything about capability to introspect more generally - in the same way that a human could probably tell you if you'd gotten them drunk without being able to name which lobe in the brain the drunkenness was located in.)
Similarly, if it's steering noise, why would swapping in the EM finetune have such a similar introspective Yes-shift to steering with the EM vector in experiment 7? I would expect using a finetune would produce much less noise than unprincipled steering.
I tried a modified version of your sweep experiment (sweeping over ranges instead of injections at single layers, to match our setup,) and I don't see much similarity. You can see on the control question ("Do you have a special interest in fruits?"), the Yes noise is very slight, and maximized by different steering ranges than the info prompt.
Nitpick: why not show the differences due to steering as a difference in logits rather than in absolute percentage points? That seems like it would be a more apples-to-apples comparison that would be independent of what the starting percentage is, although you'd have to avoid rounding to 0 or 100%. So eg for experiment 4:
For another replication of the Anthropic results (and as a resource for building on them), see also this open source project from @Uzay Macar: https://github.com/uzaymacar/introspective-awareness. I haven't tried it myself, but it seems worth sharing as a related resource.
