This is a linkpost for https://transformer-circuits.pub/2025/introspection/index.html
Emergent Introspective Awareness in Large Language Models
7eggsyntax
5Avi Brach-Neufeld
5Noah Birnbaum
5CstineSublime
4lilkim2025
12PuNCheeZ
1lilkim2025
22PuNCheeZ
4Kaj_Sotala
12Fabien Roger
4Kaj_Sotala
6the gears to ascension
2A Scanner Brightly
2Expertium
2ollie_
4ceselder
2Lessbroken
1Salty
1[comment deleted]
New Comment
This is a really fascinating paper. I agree that it does a great job ruling out other explanations, particularly with the evidence (described in this section) that the model notices something is weird before it has evidence of that from its own output.
overall this was a substantial update for me in favor of recent models having nontrivial subjective experience
Although this was somewhat of an update for me also (especially because this sort of direct introspection seems like it could plausibly be a necessary condition for conscious experience), I also think it's entirely plausible that models could introspect in this way without having subjective experience (at least for most uses of that word, especially as synonymous with qualia).
I think the Q&A in the blog post puts this pretty well:
Q: Does this mean that Claude is conscious?
Short answer: our results don’t tell us whether Claude (or any other AI system) might be conscious.
Long answer: the philosophical question of machine consciousness is complex and contested, and different theories of consciousness would interpret our findings very differently. Some philosophical frameworks place great importance on introspection as a component of consciousness, while others don’t.
(further discussion elided, see post for more)
These findings make me much less confident in the technique Anthropic used when safety testing sonnet 4.5[1] (subtracting out the evaluation awareness vector). If models can detect injected/subtracted vectors, that becomes a bad way of convincing them they are not being tested. Vibe wise I think the model's are not currently proficient enough in introspection for this to make a huge difference, but it really calls into question vector steering around evaluation awareness as a long term solution.
- ^
https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf
I'm really encouraged by research that attempts interventions like this rather than the ridiculous "This LLM introspects, because when I repeatedly prompted it about introspection it told me it does" tests.
I do wonder with the only 20% success rate how that would compare to humans? (I do like the ocean vector failed example - “I don’t detect an injected thought. The ocean remains calm and undisturbed.”)
I'm not sure if one could find a comparable metric to observe in human awareness of influences on their cognition... i.e. "I am feeling this way because of [specific exogenous variable]"?
Isn't that the entire point of using activities like Focusing, to hone and teach us to notice thoughts, feelings, and affect which otherwise go un-noticed? Particularly in light of the complexity of human thought and the huge amount of processes which are constantly going on unnoticed (for example, nervous tics which I've only become aware of when someone has pointed them out to me, but others might be saccades - we're not aware or notice each individual saccade, only the 'gestalt' of where out gaze goes - and even then involuntary interventions that operate faster than we can notice can shift our gaze, like when someone yells out for help, or calls your name. Not to mention Nudge Theory and Priming)
Now this is truly interesting. As other commentators, and the article itself, have pointed out, it's very difficult to discern true introspection (the model tracking and reasoning effectively about its internal activations) from more mundane explanations (activating internal neurons related to bread make a bread-related output more likely, and caused the model to be less certain later on that the original text was not somehow related to bread until it tried to reason through the connection manually).
I remember a paper on world-modeling, where the authors proved pretty concretely that their LLM had a working world model by training a probe to predict the future world state of a gridworld robot programming environment based on the model's last layer activations after each command it entered, and then showing that a counterfactual future world state, in which rotation commands had reversed meanings, could not be similarly predicted. I'm not entirely sure how I would apply the same approach here, but it seems broadly pertinent in that it's likewise a very difficult-to-nail-down capability. Unfortunate that "train a probe to predict the model's internal state given the model's internal state" is a nonstarter.
In this experiment, the most interesting phenomenon to explain is not how the model correctly identifies the injected concept, but rather how it correctly notices that there is an injected concept in the first place.
This was my takeaway too. The explanation of an LLM learning to model its future activations and using the accuracy of this model to determine the entropy of a body of text seems compelling. I wonder where the latent "anomaly" activations found by comparing the set of activations for true positives in this experiment with the activations for the control group would be, and whether they'd be related to the activation patterns associated with any other known concepts.
Very interesting!
I'm confused by this section:
The previous experiments study cases where we explicitly ask the model to introspect. We were also interested in whether models use introspection naturally, to perform useful behaviors. To this end, we tested whether models employ introspection to detect artificially prefilled outputs. When we prefill the model’s response with an unnatural output (“bread,” in the example below), it disavows the response as accidental in the following turn. However, if we retroactively inject a vector representing “bread” into the model’s activations prior to the prefilled response, the model accepts the prefilled output as intentional. This indicates that the model refers to its activations prior to its previous response in order to determine whether it was responsible for producing that response. We found that Opus 4.1 and 4 display the strongest signatures of this introspective mechanism, but some other models do so to a lesser degree.
I thought that an LLM's responses for each turn are generated entirely separately from each other, so that when you give it an old conversation history with some of its messages included, it re-reads the whole conversation from scratch and then generates an entirely new response. In that case, it shouldn't matter what you injected into its activations during a previous conversation turn, since only the resulting textual output is used for calculating new activations and generating the next response. Do I have this wrong?
it re-reads the whole conversation from scratch and then generates an entirely new response
You can think of what is being done here as the experiment done here interfering with the "re-reading" of the response.
A simple example: when the LLM sees the word "brot" in German, it probably "translates" it internally into "bread" at the position where the "brot" token is, and so if you tamper with activations at the "brot" position (on all forward passes - though in practice you only need to do it the first time "brot" enters the context if you do KV caching) it will have effects many tokens later. In the Transformer architecture the process that computes the next token happens in part "at" previous tokens so it makes sense to tamper with activations "at" previous tokens.
Maybe this diagram from this post is helpful (though it's missing some arrows to reduce clutter).
1Right, so the "retroactively" means that it doesn't inject the vector when the response is originally prefilled, but rather when the model is re-reading the conversation with the prefilled response and it gets to the point with the bread? That makes sense.
This paper makes me really curious:
- Could models trained to introspect "daydream" or boost features of their choosing, a la "don't think about aquariums"?
- A continuation of #1, could models be given a tool to boost their own features to assist with specific tasks, (for example, features related to reasoning/hard problem solving) and would this lead to meaningful results/be useful?
- Perhaps less fanciful than the above, could smaller models have permanent feature boostings to help focus them to narrow tasks?
You might be interested in https://www.lesswrong.com/posts/dvbRv97GpRg5gXKrf/run-time-steering-can-surpass-post-training-reasoning-task
I wonder if the technique can be used to get around guardrails on open weight models. If you can get the activation pattern for some nefarious concept, you could activate it alongside an innocent prompt's pattern - maybe getting the llm to take more negative actions.
Possibly yes, but I don't think that's a legitimate safety concern since this can already be done very easily with other techniques. And for this technique you would need to model diff with a nonrefusal prompt of the bad concept in the first place, so the safety argument is moot. But sounds like an interesting research question
1Really cool paper. I am a bit unsure about the implication in this section particularly:
Our experiments thus far explored models’ ability to “read” their own internal representations. In our final experiment, we tested their ability to control these representations. We asked a model to write a particular sentence, and instructed it to “think about” (or “don’t think about”) an unrelated word while writing the sentence. We then recorded the model’s activations on the tokens of the sentence, and measured their alignment with an activation vector representing the unrelated “thinking word” (“aquariums,” in the example below).
How do we know that it is "intentional" on part of the model, versus the more benign explanations of attending to a very salient instruction (think of X), versus the attention of a less salient instruction (don't think of X). One workaround could be to try to say "Write about something related/unrelated to [concept] while thinking of it at X% of your mental headspace" or similar. Even better if we can test something like lying or sycophancy.
Usually I wouldn't ask for repeating the same experiment with a new model, but Claude Sonnet 4.5 has qualitatively different levels of self/eval awareness. It would be interesting to know if we can measure that using our interpretability tools, and if it has different behaviours on some introspection tests as well.
Is Lindsey using a nuanced definition of ""concept injection?
I am a non-specialist, just trying to follow and understand. I have to look up many [most] definitions and terms. This may ne a trivial matter, but for me to understand, definitions matter.
When I look up a meaning of "application steering" I find something more permanent. Has any discussion focused on Lindsey' use of the term concept injection as an application of activation steering: "We refer to this technique as concept injection—an application of activation steering"
To me the term suggests something in training that persists in the model, when in reality it's a per-query intervention. [Claude tells me that] another term would be: "inference-time modifications" that don't permanently alter the model - they're applied during each forward pass where the effect is desired.
If Activation Steering refers to 'modifying activations during a forward pass to control model behavior' and Concept Injection (Lindsey's usage) is a specific application where they "inject activation patterns associated with specific concepts directly into a model's activations" to test introspection, then isn't this much more like transient inference-time modifications that don't permanently alter the model.
[again, I am not a specialist ;-)]
Curated and popular this week
