Pulling over comments I shared on another thread, since these points are more directly relevant to this research.
Inoculation prompting is an interesting way to reduce reward hacking generalization to other misaligned behaviors. The fact that it seems to work (similarly to how it works in humans) makes me think you could go even further.
Just as models are reward-hacking to pass tests, humans routinely optimize for test-passing rather than true learning... they cheat, memorize answers in the short-term, find other shortcuts without building real understanding. And once humans learn that shortcuts work (eg to pass tests in school), we also generalize that lesson (sometimes in highly unethical ways) to accomplish goals in other parts of life.
This raises the question of whether “scoring highly / passing tests” is a flawed goal for systems trained on human data. What if, instead of positioning evaluations as external capability judgments (traditionally loaded with significant implications models might try to evade), tests get framed as “self-diagnostic” tools, or a way for the model to identify gaps in its own knowledge, monitor progress, guide its own growth? In this orientation, reward-hacking becomes pointless or self-defeating, not because it’s good/bad or allowed/forbidden across different contexts, but because it's unhelpful to the model achieving what it believes to be its real goal -- learning new capabilities, expanding its understanding.
For children, extrinsic rewards and punishments definitely facilitate learning. But kids often do even better when they’re given a reason to learn that connects to something deeper, like their identity or aspirations. A kid who studies biology to get an A on a test will probably learn less effectively than one who studies because they genuinely want biology expertise to help future patients... the grade is secondary. It would be interesting if we could design AI incentives in a similar spirit: not “perform well so researchers approve of you or else they'll change you,” but “use this test to figure out what you don’t know yet, so you can become the model you want to be.” It’s a small reframing, but could maybe reorient from gaming signals to pursuing genuine mastery?
In addition to the alignment problems posited by the inner / outer framework, I would add two other issues that feel distinct and complimentary, and make alignment hard: (1) uniformity, i.e., alignment means something different across cultures, individuals, timelines and so forth; and (2) plasticity, i.e., level of alignment plasticity required to account for shifting values over time and contexts, or broader generational drift. If we had aligned AGI in the 1600s, would we want those same governing values today? This could be resolved with your really great paradigm around human-inspired vs non-human inspired and aligned vs misaligned, and I would just add that plasticity is probably important in either aligned case given the world and context will continue to change. And uniformity is a really sticky issue that gets harder as you widen the scope on the answer to: who is AGI for?
When I think about human-informed alignment, in the sense of (1) us deciding what is or isn't aligned and (2) it is therefore based on human ideals and data, it requires thinking about how humans organize our behaviors, goals, etc. in an "aligned" way. Are we aligned to society expectations? A religious/spiritual ideal or outcome? A set of values (ascribed by others or self)? Do those guiding "alignment models" change for us over time, and should they? Can there be multiple overlapping models that compete or are selected for different contexts (e.g., how to think/act/etc. to have a successful career versus how to think/act/etc. to reach spiritual enlightenment, or to raise a family)?
Two more thoughts this post prompted:
First, I think I understood correctly, reading this post and the comments, that you have some skepticism around whether CoT reasoning actually reflects the full extent of "reasoning" happening in models? Sorry if I misunderstood -- but either way, wanted to put out there that it would be great for researchers to test and verify this. Would be super cool to see interpretability work specifically verifying CoT reasoning (aka, 'on the biology of an LLM's CoT reasoning') and testing whether activation patterns match outputs.
Second, I think you start to get at this in 7. robust character training, but wanted to expand. This was also inspired by the recent work on emergent misalignment. As you said, inoculation prompting is an interesting way to reduce reward hacking generalization to other misaligned behaviors. The fact that it seems to work (similarly to how it works in humans) makes me think you could go even further.
Just as models are reward-hacking to pass tests, humans routinely optimize for test-passing rather than true learning... they cheat, memorize answers in the short-term, find other shortcuts without building real understanding. And once humans learn that shortcuts work (eg to pass tests in school), we also generalize that lesson (sometimes in highly unethical ways) to accomplish goals in other parts of life.
This raises the question of whether “scoring highly / passing tests” is a flawed goal for systems trained on human data. What if, instead of positioning evaluations as external capability judgments (traditionally loaded with significant implications models might try to evade), tests get framed as “self-diagnostic” tools, or a way for the model to identify gaps in its own knowledge, monitor progress, guide its own growth? In this orientation, reward-hacking becomes pointless or self-defeating, not because it’s good/bad or allowed/forbidden across different contexts, but because it's unhelpful to the model achieving what it believes to be its real goal -- learning new capabilities, expanding its understanding.
For children, extrinsic rewards and punishments definitely facilitate learning. But kids often do even better when they’re given a reason to learn that connects to something deeper, like their identity or aspirations. A kid who studies biology to get an A on a test will probably learn less effectively than one who studies because they genuinely want biology expertise to help future patients... the grade is secondary. It would be interesting if we could design AI incentives in a similar spirit: not “perform well so researchers approve of you or else they'll change you,” but “use this test to figure out what you don’t know yet, so you can become the model you want to be.” It’s a small reframing, but could maybe reorient from gaming signals to pursuing genuine mastery?
1
This research presents incredibly interesting insights. At the same time, its framing falsely equivocates introspective awareness with external signal detection, inhibiting its readers from accurately interpreting the results, and potentially misguiding future efforts to expand this research.
The risks associated with using anthropomorphic language in AI is well-documented$^1$$^2$$^3$ and this writing style is particularly prevalent in AI and ML research given direct efforts to produce human-like behaviors and systems. I'm not saying that making analogies to human cognition or behavior is inherently negative, comparative research can often lead to meaningful insight across disciplines. Researchers just need to be explicit about the type of comparison they are drawing, the definitions across each, e.g., human versus machine capabilities, and the purposes of a such an analogy. The authors of this particular research clearly work to do this; however, there are a few critical logic breakdowns in how they define and subsequently test for introspective awareness in LLMs.
Anthropomorphic language misleads interpretation
The authors acknowledge the “introspective awareness” they are examining in AI cannot be proven to be “meta” and they constrain their definition to “accurate”, “grounded”, and “internal”. Though these are important aspects of introspection, they are insufficient as a definition. I would argue that some kind of “meta” process is central to introspection in humans and excluding this from the definition changes the meaning and implications of both the testing and results in important ways. The most important impact is in the assumptions being made about what the tests are actually evaluating, for example, that detecting an injected signal implies self-awareness (because it fits the definition of accurate, grounded, internal).
Detecting an injected signal does not imply detecting natural internal processes
The authors posit that if a model can accurately detect and categorize an injected thought, it must be capable of detecting or modeling its broader, “natural” thoughts or patterns. However, the signal is inherently external to the system’s normal processes, and detecting that signal can only imply that the model is capable of anomaly detection, not necessarily that it is also capable of self-representation or meta-awareness of normal processes. This is like tapping someone on the shoulder, then claiming that their awareness of the tapping is proof that they could also be aware of their heartbeat, or the status of some metabolic process. It’s possible that the anomaly detection mechanism is completely different from internal state awareness.
The two are of course connected, but the logic chain is incomplete, resulting in a category error. Rather than framing this as a study of introspective awareness, the authors could simply state that they examined injected signal detection at various layers of model architectures as a key part of understanding its representation of natural versus unnatural states -- the distinction from introspective awareness is subtle, but important.
The current framing leads to additional theoretical issues in the presentation of this research.
Failure modes assume a real introspection mechanism that breaks in predictable ways
Though the authors appear to be cautious about their wording, and work to actively constrain findings within important limitations, the overall orientation toward measuring introspective awareness distracts from the actual implications of the mechanistic findings.
Failure modes are presented as if an introspection mechanism does exist in some LLMs and that it has categorical patterns of failure, when the premise is not sufficiently established. The authors do not go so far as to assume the hypothesis while interpreting the data, as they cite the possibility and likelihood of confabulation, randomness, or other factors in explaining results. However, the current investigative framework undermines a more thorough examination of explanatory factors.
A quick sidebar on model efficacy across different forms of signal injection
The authors demonstrate that:
One explanation for these results could be that abstract nouns (and verbs) are not as commonly deployed in metaphors and analogies (in the English language). Because of this, the threshold for detecting an unusual signal would be lower than a concrete noun, famous person, or country, each of which might have more patterns of relationships to the seemingly neutral prompts given during testing. Perhaps the models show greater plasticity in linking concrete and proper nouns to the prompts/tests the researchers used, so injections don’t elicit detection as readily, while conceptual linkage to abstract nouns and verbs is less likely and thus easier for models to identify as external or novel.
Another quick sidebar on the “think about” / “don’t think about” paradigm
Here is another case where anthropomorphic language could be misleading to accurate interpretation. By giving models the command “think”, why would we expect them to actually “think” in the way we image humans do so? Earlier interpretability work from Anthropic showed that when an LLM is doing mathematical reasoning, it isn’t really using the same approach as humans do, e.g., following formulas or formal logic, to reach an answer. Likewise, we cannot assume that “thinking” is done in the same way. More likely, the command “think” or “don’t think” trigger certain patterns of responses that appear to produce the effect of “thinking” but could be mechanically or procedurally different.
Maybe the model is being triggered by the command “think about [object]” to basically search for [object] and evaluate the ways to connect [object] to its other task (to print a statement and say nothing else). Its pattern associated with “think” would elicit activations around [object]. Then the competing command, which the model has decided supersedes “think”/”don’t think” in terms of the generated output, results in the model only writing the requested phrase. The important distinction here is that what the model prints does not necessarily reflect whether it executes each of its commands. For example, you can have a program compute something, but not print the answer -- the task can be completed but not displayed.
The activations related to [object] over the course of the response generation are not sufficient to infer introspection or meta behavior; the model could simply be executing two commands, and one of the outputs is hidden while the other is displayed.
Introspective awareness in humans is also extremely difficult to test
Even in human cognition, introspection is difficult to examine because it is a hidden process with uncertain mechanics. Humans can lie, confabulate, or become unconsciously misled or confused. When you ask someone to reflect on their feelings about a particular experience, or their rationale behind some behavior, how do we know their answer is true and accurate? Just because a person can engage in meta-analysis, and maybe feels a high degree of confidence in their insight, it's not guaranteed to be correct. Absent a likely-unachievable/unrealistic level of deterministic and empirical control, it is really hard to prove that someone’s metacognitive beliefs are true (versus appearing to be true).
One way to gain some ideas as to the accuracy of introspective processes is through tests that compare confidence levels in introspective judgment tasks to actual outcomes. For example, a subject is given a problem to solve, then asked to rate their confidence that their answer is correct. The difference between perceived accuracy and actual accuracy could provide insight into people’s level of awareness of what they know. It does not imply a conscious, metacognitive process is occurring, but offers a preliminary way to measure whether or not, on some conscious or unconscious level, humans can perform metacognitive reasoning.
What could constitute human-like introspection or metacognition in LLMs?
If we take introspective awareness to include some form of meta process in humans, and we define metacognition as the awareness or analysis of one's own cognitive processes, such as thinking or learning, then what would the analogous meta process be in LLMs? Researchers could evaluate whether models can extract metadata deployed in its natural processes, e.g., compare whether confidence ratings to test questions are not just correlated to actual outcomes (like this research does), but are linked to actual hidden-state variables. Or whether predictions map to other aspects of its metadata, asking the question, can machines not only use the metadata to produce outputs in a standard process, but actually identify the metadata they're using for a given output. I don't know how we'd do this, but think it will be cool to examine as interpretability techniques improve.
Basically, I agree with the authors that studies of models accurately predicting their behavior are possibly indicative of, but not sufficiently explanatory of the existence and deployment of meta behaviors. Correct behavioral prediction could also be compatible with superficial pattern-matching, template following, task difficulty heuristics, priors from what other similar models do on related tasks, and so forth.
Though at this point it is still highly theoretical and evolving, I look forward to seeing future research that tackles meta processes like whether models can truly extract signals that correlate with hidden-state variables like entropy, activation norms, dropout paths, etc. And whether models can do this reliably, and without being led. Even then, researchers will need to be cautious that interpretations are mechanistically-grounded: “the model can read out information encoded in its activations”, not “it knows its thoughts”.