Responses to o4-mini-high's final criticisms of the post:
Criticism: "You're treating hyper-introspection (internal transparency) as if it naturally leads to embedded agency (full goal-driven self-modification). But in practice, these are distinct capabilities. Why do you believe introspection tools would directly lead to autonomous, strategic self-editing in models that remain prediction-optimized?"
Response: Yes, these are distinct, and one won't necessarily lead to the other - but both are being developed by the same groups in order to deploy them. There's a reasonable question about how linked they are, but I think that there is a strong case that self-modifying via introspection, even if only done during training and via internal deployment would lead to much more dangerous and hard to track deception.
Criticism: "You outline very plausible risks but don’t offer a distribution over outcomes. Should we expect hyper-introspection to make systems 10% more dangerous? 1000%? Under what architectures? I'd find your argument stronger if you were more explicit about the conditional risk landscape."
Response: If we don't solve ASI alignment, which no-one seems to think we can do, we're doomed once we build misaligned. This seems to get us there more quickly. Perhaps it even reduces short term risks, but I think timelines are far more uncertain than the way the risks will emerge if we build systems that have these capabilities.
Criticism: "Given that fully opaque systems are even harder to oversee, and that deception risk grows with opacity too, shouldn't we expect that some forms of introspection are necessary for any meaningful oversight? I agree hyper-introspection could be risky, but what's the alternative plan if we don’t pursue it?
Response: Don't build smarter than human systems. If you are not developing ASI, and you want to monitor current and near future but not inevitably existentially dangerous systems, work on how humans can provide meaningful oversight in deployment instead of tools that enhance capabilities for accelerating the race - because without fixing the underlying dynamics, i.e. solving alignment, self-monitoring is a doomed approach.
Criticism: "You assume that LLMs could practically trace causal impact through their own weights. But given how insanely complicated weight-space dynamics are even for humans analyzing small nets, why expect this capability to arise naturally, rather than requiring radical architectural overhaul?"
Response: Yes, maybe Anthropic and others will fail, and building smarter than human systems might not be possible. Then strong interpretability is just a capability enhancer, and doesn't materially change the largest risks. That would be great news, but I don't want to bet my kid's lives on it.
Epistemic Status: Speculative, but grounded in current trends in interpretability and self-supervised learning. This was initially written as an outline at some point in 2024, but written/rewritten more fully in response to Anthropic’s announcement that it was pursuing this general direction. (Drafting was assisted by GPT4o, but the direction and rewriting to fix its misunderstandings are my own.)
Most alignment conversations treat LLMs as black boxes that can be prompted, steered, or fine-tuned by external agents. But as alignment research increasingly focuses on interpretability, and is increasingly co-managed by the systems themselves, it seems that the safety plan isn’t just better outer control—but better inner transparency. That is, we imagine LLMs that can not only tell you when they’re uncertain or wrong, but accurately identify why are where they went wrong. That might include what prior associations led to the mistake, where in the weights the issue is, or what training data shaped the answers in a given direction.
This post posits and explores this idea of hyper-introspection, which is that future models might develop or be given the capacity to explicitly inspect themselves, create some implicit clear self-model that is based on this information, detect and debug their own reasoning chains, and rather than the oft-noted concerns about designing their successor, will propose or implement self-modifications. This would be intended to allow error detection, causal traceability, and self-correction all bundled into a system[1] that can self correct, notice when it is failing, and propose changes.
But this isn’t just promising—it’s also incredibly risky. Giving models the tools to model and modify themselves could accelerate both prosaic near-term alignment, the focus on Anthropic's current interpretability research, and also deceptive longer-term misalignment.
Our understanding of the systems is crude enough that we are unlikely to detect when systems are misaligned, or becoming misaligned, especially when they have a better understanding of themselves than the humans overseeing them. We don't have anything better than rough heuristics for when and why fine-tuning works, and we should expect that we won't have any better understanding when the model is being used for the analysis or for proposing changes.
Currently, LLMs can sometimes notice contradictions or errors via chain-of-thought prompting, or when asked to double check their work. But successful self-correction without pretty explicit prompting on the part of humans is not common, and is certainly not reliable. The models lack self-awareness to realize how to introspect, much less the ability to meaningfully access their own training history or weights, so they are not only a black box, but one they cannot even see. In some cases, researchers with access to the weights have tools that allow them to interpret the model’s actions, but the model itself does not use those tools. Similarly, researchers can access things like relative probabilities of tokens, but the model itself only “sees” the tokens which are realized.
Current research on interpretability is essentially capabilities-focused, including work on detecting inconsistencies and hallucinations to improve the model’s performance. Interpretability tools can be used to that end, though I’m unsure how or whether this is done in practice for reasoning models, and in general the approach seems to be to use such tools only during training. However, even without interpretability tools, it seems that models have some degree of functional introspective ability. (For example, early testing showed that models seem to have a rudimentary and low-accuracy but predictive ability to estimate the temperature of a generated output.)
I’m positing future work that enables what I will call Hyper-Introspection, a capability set encompassing at least tool usage to inspect its own weights and perform interpretability work on them, which is then used to diagnose and understand its behavior. This might be used for the detection of internal contradictions or failures, and tracing such issues or failures back to latent associations, subnetworks, or training examples. Essentially, this would look something like a combined debugger, therapist, and historian living inside the model.
I spent at least a decade absolutely certain that I hated mushrooms. At some point, I connected this in my mind to the story that I had heard countless times about how, when I was three, I ate a mushroom in the back yard, my sister ran and told my parents, and I had my stomach pumped. At some point, I realized my dislike of mushrooms presumably dated to that event, and actually tried some. In fact, I now really like mushrooms.
More generally, a hyper-introspective human might notice some preference or trait that they don’t like, trace it back to childhood experiences, or books they read, or some other factor, and then actively reshape their mental model. Now imagine that—but as an internal loop running inside an LLM-based system.
What I mean when I talk about "hyper-introspection" is models that can access, analyze, and explain their own internal workings with enough fidelity that they can trace behaviors to causes, and recommend changes, or more concerningly, intervene on their own weights.
More precisely, hyper-introspection would include at least low level access, the capability to diagnose or research specific cases, and the ability to diagnose and propose changes.
Low level access means that the model has and uses tools which can query its own internal state — not just outputs or token-level activations, but weights, attention patterns, activations, gradients. Diagnosing and research cases means that given a particular output, bias, or failure, the model can trace the origin using the low level tools. It would answer questions like "Which subnetworks or weight clusters most contributed to this mistaken conclusion?" or "Was this behavior more likely shaped by pre-training data, fine-tuning, or emergent optimization?" Diagnosis and proposing changes means that the model can propose causal stories about how these internals led to specific behavior - not in the current chain-of-thought post-hoc rationalization sense, but via mechanisms. And based on introspection, the model would suggest (or even implement) targeted changes. For example, removing harmful or spurious associations, adjusting internal weighting between conflicting concepts, or reweighting experts in the mixture of experts. To do this the model would address some amount of the embedded agency problem, most worryingly, to be used in reinforcement learning without clear segregation of goals. That is, it would create and update a working self-model of its own tendencies, failure modes, and likely causes for both, but update that self-model partly to pursue its goals.
In other words, hyper-introspection is what you would get if you embedded an interpretability researcher, a debugger, and a slightly neurotic therapist inside the model, with real-time access to the model's own mind, but didn’t give it a way to protect its goals or define that goal clearly. This is a much more dangerous tool than simple confidence estimates or chain-of-thought prompting. It turns a passive black box into something more like an active epistemic agent — one that not only acts, but studies and edits itself.
Of course, true hyper-introspection may require architectural changes, not just better prompting or training. But this is exactly the direction that Anthropic seems to be pursuing. Of course, it might not be possible to do, and requires substantial progress on interpretability, but that is what is being planned - alongside current work building the type of agentic system that could implement the types of work I’ve suggested.
Given that, I don’t want to suggest more about how this could happen, or why it might be valuable for capabilities. Instead, I will note that there is a case which will likely be made that this capability would be accessible externally, so that hyper-introspection can radically improve oversight. This is deeply dangerous, and leads to almost exactly the same failure mode as OpenAI discusses when talking about training chain-of-thought. Any selection pressure based on monitoring this type of internal oversight would lead to strong optimization for deception. And even the internal oversight and recommendations themselves would likely select for capabilities that are self-deceptive and hide goals. Lastly, automating any such capacity would be a direct path to dangerous self-modification and improved deception.
Hyper-introspection is plausibly useful for monitoring human-level systems, but also a tool that can enhance capabilities (including deception and autonomy) in ways that explicitly evade our ability to meaningfully monitor and control them. And we should expect that control and monitoring methods are limited by human capacity, so that when the models are far faster or, more challenging, far smarter than humans, there will no longer be capacity for meaningful oversight. So I claim that both edges of this double edged sword are aimed at dangerous outcomes.
When outlining this originally last year, I was hesitant to speculate about this goal and approach, but at this point it seems that it was overtaken by events. To be somewhat hyperbolic, I would be concerned about warning people not to build the torment nexus, but they have already started doing this - so this post is trying to bring attention to how dangerous pursuing such work will be and repeat what evidently needs repeating - we don’t have a workable plan for aligning, monitoring, or controlling smarter than human systems.
Edit to add fotnote: This doesn't imply the base model itself does this, but that it has tools and oversight models that enable it, creating a system-of-systems. And the system of systems is then a concerning coherent actor, because we haven't solved robust scalable oversight generally.