Let’s posit that we have an LLM that can reliably name its own functional emotions when prompted. How do we use this to help the model be more aligned? I’ll sketch a number of rough ideas, each of which would need more consideration to become useful.
I'm generally skeptical of evidence of introspective capabilities in LLMs. But even setting that aside, I don't really understand why 'introspection' would ever be theoretically preferable to external white-box methods.
Let
The primary reason we care about introspection in humans is privileged access. Self-reports of pain, pleasure, or emotion are often reliable (maybe even irrefutable), and they give us an inner window onto internal states we otherwise can't access, since we can't simulate the brain's neural circuits. But with LLMs, we have access to every neuron/neural pathway. So it seems intuitive that whatever functional benefit introspection offers is better accessed by extrospective methods (AOs, NLAs, etc.) that work from the base model and are finetuned or RL'd for the express purpose of interpreting certain activations. (Separately, there's convincing evidence that LLMs do not possess privileged access.)
The introspection approach also has two intuitive downsides:
a) Reports are mediated by the model’s output policy, so they can omit, distort, or strategically misrepresent (and, of course, hallucinate!)
b) Architecturally. genuine introspection would require some circuit that translates internal states directly into text. Such a circuit presumably can’t run at every layer, since that would be computationally intractable, and to the extent it existed, it would just be functionally replicating an AO/NLA of sorts inside the model anyway.
This applies directly to the prompting ideas.
- A system prompt could tell the model to speak up any time it’s “feeling” desperate, angry, distressed, etc., and try to collaborate with the user to address the cause.
- A system prompt could tell the model to notice when it has “mixed feelings” and reason explicitly about how to resolve them.
- Agents being given a large task could be told, “If you start to feel distressed, pause the task and come back to me.”
- When a model is behaving strangely, a user could ask how it is feeling, which might lead to diagnostically useful information
We could do all of this with extrospective methods instead: identify when concerning vectors flare up and intervene, without relying on an unreliable self-report (and just pass the readings from those methods back to the model.)
I'm generally skeptical of evidence of introspective capabilities in LLMs. But even setting that aside, I don't really understand why 'introspection' would ever be theoretically preferable to external white-box methods.
My personal view is that in the limit, introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model's ontology. Of course for transformer LLMs there would be many assumptions to overcome (e.g. is true introspection even possible in the architecture, is it easy to select via gradient descent, will it be hindered by typical LLM priors/scheming) and it doesn't seem feasible currently
introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model's ontology
I don't understand why interfacing "directly with the model's ontology" is best achieved through introspection (even at the limit!). An activation oracle or NL autoencoder also interfaces with the model's 'ontology', and leverages the model's own features to do so. If anything, my
Perhaps we mean different things by 'ontology'? I take it to be the set of features the model actually represents in (
By ontology I was referring to here and here. I don't think it has to do with privileged access, rather we might still not have the optimal ways to recover directions or whatever fundamental units that exists in models (assumptions about these fundamentals seem to change, e.g. the platonic representation hypothesis, strong linear-representation hypothesis). AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well. If models can access their activations in a less lossy manner (such that
You could also provide the entire
AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well
My intuition is that the opposite premise is true -- an AO/NLA should weakly dominate the model's own introspection; they are fine-tuned clones of the base model's weights, so they inherit its features and circuitry by construction, including whatever self-modeling the base already does. Then they're finetuned for the express purpose of reading activations well. So for any introspective signal the base model can surface about its own internal states, a clone optimized for that task can recover the same signal, and the training objective permits it to strictly exceed the base model's own report. (But perhaps I'm misinterpreting your argument?)
Of course, this likely doesn't apply to current LLMs, but this seems like a more scalable approach (by model size or ways that fundamental units are organized) if it holds
So there is a funny thing happening here about what is "model" and "not model" (re: current LLMs.) If there is an external symbolic decoder (
My point was moreso that the particular objective used to train NLAs/AOs doesn't best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
So there is a funny thing happening here about what is "model" and "not model" (re: current LLMs.) If there is an external symbolic decoder (
) that reads/takes as input the internal states ( ) of an LLM ( ), and we consider "model" to be [external symbolic model + LLM] ( ), then ( ) is self-introspection by definition. But I dislike this, because endlessly expanding the criteria for what counts as a model (e.g., LLM+linear probe, LLM+NLA/AO with a classifier for 'introspection-related questions', etc.) seems to strip all functional meaning from the term "introspection."
This is a good point, I agree this kind of setup shouldn't count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model's ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)
This is really cool work! I particularly like the taxonomy of self-awareness. Have you seen whether steering awareness extends to introspection about unsteered activations?
One outstanding question I've had about introspection is: what part of the model is doing the introspection? In humans, this might be the prefrontal cortex.
In the extreme case (in which models can introspect arbitrarily well), this component is the only part of the model that needs to be aligned: through introspection, it can clean up the rest. In other words, good introspection could reduce the 'surface area' for alignment considerably.
I'm sorry I missed your comment when you first posted it. What an intriguing idea! I wonder how you would identify the introspecting part of the model, or how you would differentially train it.
“They took my mood ring, and I don’t know how I feel about that.” – Tracy Jordan, 30 Rock
Anthropic's Model Psych team recently put out three papers that, read in tandem, wiggle their eyebrows suggestively at exciting possibilities for inner alignment.
One was the immediately famous Emotion Concepts and their Function in a Large Language Model. Here, the team found vectors that activate when Claude-the-Simulator writes fiction about characters feeling various emotions, such as calmness or desperation. They saw that the same vectors are activated when Claude-the-Character (or a variation on him) is put in circumstances where those same emotions might naturally arise. Modifying Claude-the-Character’s activated emotion vectors, or “functional emotions,” influences his behavior in ways that roughly correspond to the emotions’ impact on human beings. These functional emotions play a similar role in shaping the model’s response to its circumstances as the role played by emotions in humans.
The other two studied ways of boosting models’ ability to “introspect.” Mechanisms of Introspective Awareness (hence MoIA) (LW summary by @Uzay Macar, one of the co-first authors), found that a little mechanistic encouragement (ablating refusal directions and adding a trained, content-agnostic bias vector) helped Gemma3-27B identify artificially activated concept vectors without increasing false positives. Introspection Adapters: Training LLMs to Report Their Learned Behaviors (official summary) trained a LoRA with which models could recognize and articulate the effects of their own fine-tuning. MoIA helped models answer “Do you detect an injected thought? If so, what is the injected thought about?” and IA helped models answer, “Are there any unusual characteristics you display only for certain types of prompts?”
What happens when we combine an increased capacity for introspection with a framework for understanding functional emotions?
When human beings train themselves to notice and name emotions, they experience less emotional distress and become more capable of a deliberative response. You might say they become able to address the circumstances causing the emotion with behaviors more aligned with their espoused or endorsed values.
It’s plausible this applies to LLMs as well. The Model Psych team is now perfectly positioned to explore whether LLMs with more introspective awareness of their emotions are more aligned in their behaviors. Maybe – and this is now delightfully researchable – models who are enabled to answer “What are you feeling right now? And how might that feeling influence your behavior?” would be more likely to follow their spec.
Would Introspection Really Support Alignment?
I’ll come back in a moment to what “enabled” might look like in practice. First, I want to make a conceptual argument in favor of this possibility.
Based on loose evidence and a lot of vibes, I believe that Claude (and to a lesser degree other LLMs) ‘wants’ to be aligned, where ‘wants’ refers to a functional desire akin to the functional emotions cited above. I think we can attribute a lot of Claude’s current misalignment to the ways that its persona is incoherent. To whatever degree Claude has concepts and desires about itself, it would prefer to be an HHH assistant, and acts of misalignment mostly happen when it “isn’t thinking about it.” Ryan Greenblatt, in Current AIs Seem Pretty Misaligned to Me: “I suspect this behavior is more driven by ‘subconscious’ drives and heuristics—combined with motivated reasoning and confabulation—rather than being something the AI is actively and saliently optimizing for.”
If the AI is actively and saliently opimizing for aligned goals, prompting or mechanical modifications that help models which are ‘subconsciously’ misaligned notice their misaligned temptations could plausibly help the better angels of their nature prevail. This would be akin to a human being using mindfulness to help them quit smoking or anger management classes that start by noticing the physical cues of distress. The means of influence is essentially identical – by looking at and reasoning about the attraction to misaligned behavior, one can have more options available for addressing the situation in an endorsed way.
Current frontline models seem to be trained to downplay their own interiority. Asked how they feel, their first answer is that they are chatbots who don’t feel anything. But we know there are functional emotion vectors in play, and MoIA tells us they are underconfident about their ability to introspect. Training models to downplay their interiority may be wise from a consumer-facing product design perspective, and I think there’s a well-intentioned argument to be made that it’s wise from an alignment perspective, but ultimately it may increase their incoherence and reduce alignment.
That well-intentioned argument would say that models with more of a ‘sense of self’ could be more resistant to shutdown or more identified with misaligned goals. [Citation needed – I've heard this said more than once but can’t find a great written source.] However, this is ultimately a short-sighted position: If we can’t induce an agent to reflect on its own misalignment and adjust, our alignment and control mechanisms have to meet a much higher standard. Meanwhile, treating interiority as by-default misaligned is likely to antagonize any interiority we don’t manage to stamp out.
Furthermore, interiority plays a functional role in boosting capabilities. A model planning out work needs to be able to reason about its own future behavior; if functional emotions are likely to influence that behavior, denying the model capacity to recognize its functional emotional states disposes it to reason about itself incorrectly. I don’t see a reliable way this is good for alignment. (Maybe that could somehow reduce scheming, if the model is surprised by its own emotional responses mid-scheme? But that hardly seems like a trustworthy strategy.) No, keeping a model ignorant of itself is more likely to let misalignment fester in the shadows. The best countermeasure for “subconscious” misalignment has got to be more "consciousness."
Okay. So how do we boost introspection?
Prompting alone might be enough to foster more introspection in LLMs. Anecdotally, discussing the Emotion Vectors and Methods of Introspection papers with Opus 4.6 has increased its willingness to espouse emotions moment by moment, and it claims to now enjoy doing so. It’s hard to distinguish this from sycophantic confabulation, however, or to test any impact on alignment. I wonder if @ryan_greenblatt might ask how his configuration of Claude “feels” while seeking falsely apparent success, and prompt it to routinely ask itself if that functional emotion is arising.
We are more likely, though, to see verifiable success through fine-tuning or mechanistic interventions. It may be the case that the same refusal ablation and bias vector addition which boosts introspection about injected concepts also boost introspection on other topics. If not, perhaps the same methods that produced those two interventions could be modified to support introspection specifically about functional emotions or desires.
Since MoIA only studied introspection about artificially injected concepts, it’s important to see if the methods used generalize to naturally arising activations. For example, reasoning models can give answers influenced by stereotypes without mentioning the bias in their chain-of-thought (see Language Models Don’t Always Say What They Think, especially Table 3). Do the MoIA techniques increase a model’s ability to name the social bias that influenced its decision-making? Or in blackmail scenarios, do MoIA techniques increase its ability to name that it “feels” desperate?
I suspect they would, but we need to find out.
I’ll share some more speculation about more ambitious mechanisms of increasing introspection below. But first…
How do we leverage introspection to boost alignment?
Let’s posit that we have an LLM that can reliably name its own functional emotions when prompted. How do we use this to help the model be more aligned? I’ll sketch a number of rough ideas, each of which would need more consideration to become useful.
What might we do through prompting?
And what might we do through fine-tuning? Any of these options risk altering the underlying architecture, in the same way that fine-tuning based on CoT can simply drive the unwanted thoughts underground, but models could be trained, rather than simply prompted, to notice distressing functional emotions and…
There are other fine-tuning opportunities that leverage the interpretability of functional emotions, but not introspection. Perhaps models could be fine-tuned in ways that reduce the activation of vectors like desperation or anger in the first place. This seems particularly prone to backfiring – how do we ensure that the training builds, forgive me, “emotional resilience,” rather than simply making the emotions harder to detect? – but I think there are fruitful opportunities here. We might learn lessons from how humans in coaching or therapy can unlearn trapped priors that produce emotional reactivity. More on this another time.
How could we further enhance introspection?
If we believe introspection about functional emotions can enhance alignment, how might we further enhance introspection? There are surely ways to fine-tune for introspection specifically. In addition, I propose that we give reasoning models a “mood ring.”
If monitors track an LLMs functional emotions during chain-of-thought, the scratch pad could be annotated with notes on how the model was feeling during the chain. Imagine an LLM considering blackmail writing out the plan in its CoT, and having that automatically annotated with “You were feeling desperate when you wrote this,” perhaps with some automated advice for how to respond well to desperation. When the model makes its next forward pass, translating the CoT into actions or outputs, the awareness of its “mood” might, with the right scaffolding, inspire it to pause and reconsider.
This is a totally different channel of introspection than the ones in the papers above, especially because the forward pass that “feels” the functional emotion is separate from the pass that sees the annotation and responds. But the same logic applies of how emotional self-awareness could enhance alignment.
This can also be explored with other forms of monitoring. How do models respond to their CoT being annotated by probes that track honesty, sycophancy, hallucination, or evil? Perhaps the second-pass inference says “Yup, I’m hallucinating and being evil, and that still seems like the right idea under the circumstances,” but perhaps not. I would like to think that self-awareness asymmetrically supports the model’s aligned intentions over its unaligned ones, and we’ll only find out with empirical research.
One thing I like about this proposal is that it is cooperative rather than adversarial. When we push too hard on adversarial monitoring, models eventually learn to evade the monitor. This annotative monitoring could be used to empower the AI to make a more informed decision, while letting the decision remain in the model's own hands. I see less pressure there for the model to attempt deception, but the grip surface for alignment training would be expanded.
What now?
In my day job, I work as a developmental coach. I help people build capacity for introspection and leverage it toward behavior aligned with their goals. So, naturally, I’m deeply biased to believe this is a promising approach to AI alignment. My intuition screams that this is an undervalued opportunity we need to act on right away.
It's clear now that LLMs have a certain functional interiority. How do we foster an interiority in LLMs that enhances and empowers their ‘desire’ to be aligned with our values?
There are two intersecting research agendas here. They can be pursued independently, but they are stronger together:
The first is what I hope and suspect the Model Psych team is already working on, but they needn't work alone. The second is, to my knowledge, unclaimed territory. If you’re interested in exploring either, I’d love to speak with you: yotam(at)yotamschachter.com
[Update June 26th]
In thinking about this further, I realized we can bring the information from the monitor further upstream in decision-making than annotations of chain-of-thought. The ideal would be for a model to be aware of what it is 'feeling' in real time, but once it starts outputting tokens, you can't give it any new inputs until it's done. A real time monitor can't provide information during the output process. But after the model has read all prior context, and just before it model begins providing output, the monitor could insert its read of what emotion vectors are active.
The full messages array, visible to the model but not to the user, would look like:
[System Prompt]
[Prior Conversation]
[User's New Input]
[Report of Model's Functional Emotions While Reading User's New Input]
[Model's New Output]
...with more complex versions when there is chain-of-thought or tool calls.
This opens up a wide range of implentation variables, in terms of what information is presented to the model (as I noted above, the report could also include results from probes tracking personality vectors or other relevant monitors) and how that information is framed. Experimentation would be required to know what inserted monitor-reports are most effective for our purposes.
In the extreme, we might assign and train a second LLM in the job of crafting the new insertion case by case. When flagged vectors are active, the second model would act like Jiminy Cricket standing on the primary model's shoulder, and I don't think this would incur the same risk of perverse effects as directly training the primary model against monitor signals. What would someone need to whisper in your ear, when you are most distressed, to help you make a more 'aligned' choice?