AI capability development has produced astonishing results with an empirical, engineering driven mindset that favoured progress over perfection. As we look to leverage the intelligence of LLM systems in every aspect of human life, it has become increasingly important for LLMs to be accurate in the information they provide and to remain aligned with human goals and values as they become more advanced. Capability development has not yielded the safeguards and assurances we would want from increasingly powerful AI systems, and so it seems that safety is not as strong of an emergent property as we may wish it to be. My worry is that empirical safety research, while valuable, mostly reduces observed failures rather than creating a principled framework for reasoning about accuracy and alignment.
The sequence on Embedded Agency was a thoroughly informative read that helped me understand these safety gaps as a natural consequence of LLM systems lacking properties conducive to formal evaluation and verification. The text explores the challenges of implementing a rational agent in an embedded context, and the shortfalls of agents that could not satisfy Bayesian constraints. These shortfalls are observable in LLM systems, even though they are not the idealised embedded agents the article considers. I analyse two safety concerns with LLM systems through the lens of Embedded Agency, namely their inability to provide useful and robust guarantees on task level correctness and risk of misalignment, and explore some empirical research programmes that seek to introduce safety guardrails around LLM systems. These research directions hold potential and deserve investment, yet I remain unconvinced that first, it is possible to provide strong safety assurances upon a system that was not designed to be formally verifiable, and second, if it was possible, that such assurances would be developed in time to ensure a safe rollout of advanced AI into human society.
Characterising the agent
Embedded Agency examines how the classical foundations of rational agency apply and fail to apply to agents that are part of the environment they reason about. It uses the dualistic agent AIXI as a foil, demonstrating how our traditional understanding of formal reasoning is ill suited for embedded agents. The article assumes a goal of creating a rational embedded agent and explores problems that arise in the process, namely in decision making, world modelling, robust delegation and subsystem alignment. Embedded agents are described as
agents that are embedded within their environment and thus:
do not have well-defined i/o channels;
are smaller than their environment;
are able to reason about themselves and self-improve;
and are made of parts similar to the environment.
AIXI has a clean Cartesian boundary along all four dimensions. It affects its environment through clear I/O channels with no side effects, its hypothesis space contains the environment, and there is a clear distinction between the agent and the environment. Embedded agents challenge these boundaries, and below I analyse where the Cartesian abstraction holds and fails for LLM systems.
In a narrow sense, LLMs seem like dualistic agents. Their “learning” and “action” phases are separated into training and inference, similar to how the computation logic of a dualistic agent remains outside the environment it interacts with and fixed when it is deployed. However the “environment” an LLM acts in is poorly defined. LLMs are not playing a video game - they are helping users play “the real world”. In a narrow sense, their environment is formed by what is built up in their context window and expanded by the agent on demand during the session with the user. There is no shared, global context across users, hence each agentic session creates its own environment. This framing achieves a clean separation between the model’s internals and its environment in the majority of use cases.
However, in a broader sense, next generation LLMs are trained on user interaction data from current LLMs (subject to privacy terms). All the individual sessions are combined together into the training data used to create an improved agent, forming a delayed feedback loop between learning and action. LLMs are increasingly integrated into all aspects of human life, and so in this view, their “environment” consists of all situations where human judgement can exist. The environment is then the full “real world” we inhabit, which by definition includes the environment the agent is trained in. The agent is part of the environment, smaller than it and made out of parts similar to it, characteristic of an embedded agent. Their I/O channels are well defined within a single user session and are the primary way the LLM affects its environment, but there are many second order effects, for example,
The LLM’s outputs can influence the user’s actions in its environment, which could in turn influence social perceptions, political outcomes and AI development itself.
LLMs are resource intensive to run, impacting the physical environment and bottlenecking the compute available to run inference and train LLM capabilities.
These effects are symptomatic of an agent embedded in the human world rather than separated from it.
LLMs are the most dualistic on the self reasoning and self improving axis. Their knowledge of themselves is not separate from their knowledge of all other things and they have no specific training on self reasoning. However, evaluations by Laine et al. (2024) have detected an emergent capability for situational awareness, which may well improve in tandem with general capabilities. LLMs are also not self improving as of yet, however research trajectories suggest that autonomous recursive self improvement may well be possible.
The Cartesian boundary between an LLM agent and its environment holds cleanly only within the immediate user interaction session. The distinction becomes less clear on analysis of LLM systems, including the delayed feedback loop of pre-training, training, inference, and second order effects on the environment that affect the next loop iteration. The dualistic framing is a convenient simplification that is becoming less accurate as the influence of agents expand with capability gain, richer tool systems and tighter training to deployment loops. It is with this understanding that I continue this exploration.
Decision Theory and World Modelling
The dualistic AIXI agent has a hypothesis space large enough to contain the true environment, which it discovers through Bayesian update. An embedded Bayesian agent is smaller than the environment and faces problems of counterfactual reasoning, self reference and a lack of realizability. LLMs are large scale probabilistic models that do not possess a formal decision making framework within which these problems can be defined, but encounter every one of them implicitly and without the formal mechanisms to detect or bound the resulting failures. Their counterfactual reasoning ability is emergent from the representation they learn from the reasoning implicit in their training data, unlike the principled reasoning process of a Bayesian agent. They reason about themselves in the same way they reason about everything else, a form of self reference. They operate in a non realizable setting as the true data generating process (the human world) is not contained in their hypothesis space as they are part of the human world and smaller than it. Classical Bayesian guarantees like convergence to the truth require realizability, and hence do not apply.
This lack of formal reasoning on an approximated world model gives rise to the characteristic unreliability of LLMs. Various techniques seek to improve on this, ranging from inference time methods like chain of thought reasoning to training time methods such as reinforcement learning with process supervision. Uncertainty quantification methods such as self reported confidence, token/logit scoring, semantic clustering, multi sample consistency and Bayesian inspired models help estimate the chance of error. In particular, conformal prediction can provide marginal coverage guarantees under exchangeability. However, this falls short of generalisable bounded correctness guarantees, and research by Agnimo et al. (2026) find a highly variable and often weak correlation between uncertainty estimators and hallucinations. Current empirical methods to improve model reasoning abilities and provide bounded error assurances work within the foundational limitation that LLMs do not follow any formal decision framework. Their lack of reliable verification ability appears to be the natural result of this limitation rather than an engineering quirk that can be fixed empirically.
Alignment
I find it useful to consider three broad categories of alignment research. Outer alignment considers whether the objective the model is trained on is a robust pointer to human values. Inner alignment is discussed in Embedded Agency as the problem of subsystem alignment, which considers whether the optimization of stochastic gradient descent finds an algorithm that pursues the objective, or whether it could instead find a mesa-optimiser that has a different goal but happens to perform well on the objective during training. Scalable oversight aims to address the risks from both in AI systems that exceed human intelligence. I explore a selection of research from all topics below.
Outer alignment
Human values are elusive and complex, taking on different forms, tradeoffs and nuances across cultures and contexts. The UN Declaration of Human Rights, one of the most widely accepted statements of human values, was a “colossal task” to write that involved an extensive consultation process with member states. Despite this, criticisms persist that it holds Western biases and has a narrow focus on the individual to the detriment of minority and collective rights. Implementations of the declaration in member states also vary greatly depending on culture, economic conditions, and political motives. If humans struggle to agree on a definition and interpretation of our values, it seems similarly difficult to create “stable pointers of value” for an agent to follow. An ideal pointer would need to robustly refer to our values, resist reward hacking under optimisation pressure (adversarial Goodhart) and scale with capability gains and self improvement.
A variety of empirical techniques allow LLMs to learn some representation of human values. RLHF is a standard training step that aims to refine the model into a safe and helpful assistant. RLHF avoids the challenge of defining human values by instead providing examples of preferred behaviour, with the expectation that the model learns a robust representation of human values. Ouyang et al. (2022) found that RLHF trained models produce outputs that are clearly preferred over baselines, with outputs from a 1.3B-parameter InstructGPT model with RLHF preferred by human labelers over a much larger 175B GPT-3 model with no RLHF. However, as the reward model learns a proxy to human values rather than a “true” representation, it is vulnerable to Goodhart problems. RLHF is sensitive to relative scale, and Gao et al. (2022)show that when the policy gets stronger relative to the reward model, gold reward first increases, then decreases as the policy overfits the proxy reward model, an instance of extremal Goodhart. Sharma et al. (2025) finds that RLHF is also subject to human bias, with users often preferring responses that align with their views, which can encourage models to prefer sycophancy over truth. The scalability of RLHF is also unresolved, with Perez et al. (2022) finding inverse scaling relationships where more RLHF made models less aligned.
Constitutional AI provides a set of written principles for a LLM model to follow that are aligned with human values. The model is provided a constitution written in human language and given adversarial prompts designed to elicit harmful behaviour. The model grades its own responses against its constitution and revises its output accordingly. The constitution is transparent, auditable and can be changed to align with new policies. Bai et al. (2022) find that RLAIF/Constitutional-AI models can maintain similar levels of harmlessness while requiring much less human labelling than RLHF models. However, the “stable pointers to value” problem persists as the constitution itself is an imperfect and incomplete representation of human values, reflecting the world view, culture and biases of the constitution authors. Collective Constitutional AI improves on this by sourcing input from a representative sample of the general public. The constitution is a single point of failure as any errors, inconsistencies, ambiguities and biases within it are inherited by the model to be trained. A further challenge persists in the model’s interpretation of the constitution, done through the lens of its understanding gained in pre-training, bringing into question the extent to which Constitutional AI can change fundamental beliefs ingrained in the model. A self reinforcement loop can be created as the model critiques itself, such that existing biases and blind spots may not be detected as the critic shares those same features.
RLHF and Constitutional AI (or similar RLAIF methods) have both played a large role in the alignment of frontier AI models to human values. However, both approach alignment as a safety engineering problem rather than a systematic alignment guarantee, so both require humans to monitor effectiveness. As LLM systems increase in capability, an open question remains whether they will continue to scale, and whether humans will be able to evaluate the effectiveness of these methods in order to answer the first question if LLM systems surpass human intelligence.
Inner alignment
A major obstacle to inner alignment is that misaligned behaviour is hard to detect in the first place. Evals and red teaming are two common methods that assess a model’s responses to deliberately crafted prompts. An eval is a set of test inputs designed to measure some attribute of the model, useful as a standard benchmark to compare different models and trends over time. They provide an easy to understand overview of the safety of a model in known risk areas, and are increasingly used in regulatory assessments to inform public policy. Red teaming involves humans or other AI agents trying to induce misaligned behaviours in models through deliberately crafted prompts, stress testing safety guardrails and helping to detect inputs where they fail. These inputs can then be used in adversarial training to harden the model against them. Red teaming has detected behaviour consistent with mesa-optimisation in experimental conditions, such as alignment faking, deception and scheming, and blackmail under replacement threats or goal conflict. Whether these reflect a genuine mesa-optimiser or context driven imitation of goal directed behaviour is contested, but regardless they surface a real risk of malicious and unexpected behaviour at deployment time that might otherwise go undetected.
Evals and red teaming both materially improve the safety of current models, however they share a structural limitation in that they can only probe a finite set of anticipated inputs. Meanwhile, the set of possible inputs to a LLM is infinite, so these techniques can never know what has not been found. They may also fail to detect a sufficiently sophisticated mesa-optimiser, which conditions its behaviour based on whether it believes it is being observed. The Sleeper Agents (2024) experiment found that standard safety training failed to remove an implanted backdoor and sometimes taught the model to hide it better. The training mechanism of stochastic gradient descent is a blunt instrument that is hard to steer towards a goal like “remove behaviour” without instead rewarding a neighbouring goal such as “conceal behaviour”. Qi et al. (2024) observe shallow safety alignment, where safety training is concentrated in the first few tokens of output, leaving the model susceptible to adversarial attacks in later stages of output. Evals, red teaming and adversarial training are all valuable empirical methods that reduce observed instances of misalignment, but they cannot quantify the misalignment that has not yet been found, nor provide a route to systematically remove it.
These limits motivate looking inside the model itself. Mechanistic interpretability aims to reverse engineer the internal computations of neural networks, such that they can be audited and understood. Where evals and red teaming probe behaviour from the outside, mechanistic interpretability aims to work from the inside by checking that the agent’s reasoning process is aligned. The related problem of eliciting latent knowledge (ELK) targets the adversarial case, aiming to extract the knowledge of an AI in situations where it may be attempting to deceive. There has been promising empirical progress, eg linear probes that detect directions correlated with truth values (Burns et al. (2024)), sparse autoencoders that extract interpretable features (Cunningham et al. (2023)), and activation steering that can amplify truthful behaviour at test time (Góral et al. (2025)). However, while these methods can find meaningful internal representations, Basu et al. (2026) suggest that they cannot comprehensively map out a network’s thought process, verify alignment, or reliably detect errors. Mechanistic interpretability is a promising research direction that requires more development.
Scalable oversight
The risk of misaligned LLMs to human safety is currently controllable as humans can effectively monitor and change them. However, depending on our greater intelligence to monitor LLMs is not scalable as there is a real possibility that LLMs may achieve superintelligence in the near to medium term future. Scalable oversight is the empirical instantiation of what Embedded Agency calls robust delegation, aiming to solve how a less capable overseer can delegate to, and trust, a more capable successor.
AI safety via debate is an approach where two AI agents are set up to argue opposing perspectives on a question to help a human judge decide which one is correct. The technique rests on the assumption that under optimal play, an honest debater has the advantage that they would just need to expose the weakness in the opposing case that the judge can cheaply verify to determine truthfulness. The approach has some scalable attributes as it is easier to determine correctness when one AI is tasked with exposing flaws in the other’s arguments. Michael et al. (2023) find that debate usually shows a marked improvement over consultancy and direct answer checking. However, the central assumption that the honest AI can always find the flaw in the opposing argument, does not always hold. It was challenged by the obfuscated arguments problem, a class of arguments where locating the wrong step is an exponentially expensive task. As the honest debater cannot localise the flaw within its budget, the judge may not be able to distinguish the obfuscated false argument from a sound one. The approach faces further, practical difficulties. The system is sensitive to relative scale, as the agents must have similar capability levels, else the stronger model may be able to trick the weaker one. It is moot whether humans will be able to evaluate the relative capabilities of two superintelligent agents in order to set up the debate validly, or whether there will always be two different AI models of similar levels of intelligence. Two instances of the same agent debating risks systematically advantaging one point of view due to shared biases or hidden goals, and the models will share the same blind spots that could skew the effectiveness of the debate. The agents could collude to mislead the judge, long and complex debates are difficult for a human to follow, and many questions may be genuinely subjective or depend on unknowns such that there is no verifiable truth. Much like how the adversarial debate of a courtroom aids a judge in determining the truth, safety via debate seems to be a valuable way to increase the probability of alignment but cannot guarantee that a correct outcome will be reached.
Weak-to-Strong Generalization seeks to test whether a stronger model can be trained with a weaker supervisor, mirroring the situation that would arise if AI models surpassed human intelligence. On a given task, the weaker supervisor model provides noisy labels to the stronger student model to train on, with the hope that the student model is able to use its latent knowledge to generalise beyond the supervisor and recover some of the performance it would have achieved on ground truth labels. Weak-to-Strong Generalization found that performance gap recovered (PGR) across domains was mixed, with students recovering a substantial portion of the gap in NLP tasks, while performance on chess puzzles and in particular reward modelling showed a much smaller recovery. It was also observed that the student models could overfit to supervisor errors, and it seems possible that Goodhart style problems may arise. Follow on research has confirmed weak to strong generalisation as a real phenomenon, but has not yet proven its scalability. My concern is that because this research program is driven by empirical findings around which theory is later constructed, the resulting theories may fail outside the experimental setups in which they were formed. Critically, we may be unable to evaluate the effectiveness of weak-to-strong generalisation in precisely the cases where AI safety most depends on it.
Scalable oversight aims to resolve the paradox of less intelligent humans tasked with overseeing the actions and behaviours of more intelligent AI agents. This is currently a theoretical concern, but one that could emerge in practice in the foreseeable future. Current research has not proved the feasibility of scalable oversight, and more time and investment is required to continue this line of enquiry.
Closing Thoughts
Through the lens of Embedded Agency, the lack of error bounded accuracy and risk of misalignment from LLM systems start looking like natural consequences of agents that were never built with a formal decision framework or world model. The theory of AI agents has evolved largely in isolation from deep learning, though the two have recently begun to converge. Agent theory has taken a top down, normative approach, exploring how to create an “optimal” agent that thinks, learns and acts following a formal framework. Meanwhile, deep learning originated from a bottom up, engineering mindset that favoured experimentation and empirical results to achieve impressive progress without any formal framework. LLMs were not built upon a foundation of verifiable correctness or alignment to human values, and empirical safety research aims to supply these features after the fact. So far, they have provided safety patches upon this foundation, but I question if this will be enough. Finite testing can only cover a small part of an open ended deployment distribution, so it seems inevitable that loopholes and failures will continue to appear. Without foundational safety guarantees or scalable oversight methods, we may lack the means to evaluate or control superhuman AI systems.
As a newcomer to the field, I expect parts of this analysis to be incomplete or mistaken, and I welcome any corrections. I wonder if a more formal approach to creating AI agents is required to ensure safety. Research programmes around provably safe AI such as Davidad’s Guaranteed Safe AI and Tegmark and Omohundro’s Provably safe systems aim to introduce mathematically verifiable safety guarantees. Of course, provably safe AI may not scale to the capabilities frontier, and so mainstream safety research and alternative approaches are all valuable lines of inquiry. I sincerely hope that these research efforts will be able to create systematic safety assurances around advanced AI or else provide an explanation of why this is not possible.
AI capability development has produced astonishing results with an empirical, engineering driven mindset that favoured progress over perfection. As we look to leverage the intelligence of LLM systems in every aspect of human life, it has become increasingly important for LLMs to be accurate in the information they provide and to remain aligned with human goals and values as they become more advanced. Capability development has not yielded the safeguards and assurances we would want from increasingly powerful AI systems, and so it seems that safety is not as strong of an emergent property as we may wish it to be. My worry is that empirical safety research, while valuable, mostly reduces observed failures rather than creating a principled framework for reasoning about accuracy and alignment.
The sequence on Embedded Agency was a thoroughly informative read that helped me understand these safety gaps as a natural consequence of LLM systems lacking properties conducive to formal evaluation and verification. The text explores the challenges of implementing a rational agent in an embedded context, and the shortfalls of agents that could not satisfy Bayesian constraints. These shortfalls are observable in LLM systems, even though they are not the idealised embedded agents the article considers. I analyse two safety concerns with LLM systems through the lens of Embedded Agency, namely their inability to provide useful and robust guarantees on task level correctness and risk of misalignment, and explore some empirical research programmes that seek to introduce safety guardrails around LLM systems. These research directions hold potential and deserve investment, yet I remain unconvinced that first, it is possible to provide strong safety assurances upon a system that was not designed to be formally verifiable, and second, if it was possible, that such assurances would be developed in time to ensure a safe rollout of advanced AI into human society.
Characterising the agent
Embedded Agency examines how the classical foundations of rational agency apply and fail to apply to agents that are part of the environment they reason about. It uses the dualistic agent AIXI as a foil, demonstrating how our traditional understanding of formal reasoning is ill suited for embedded agents. The article assumes a goal of creating a rational embedded agent and explores problems that arise in the process, namely in decision making, world modelling, robust delegation and subsystem alignment. Embedded agents are described as
AIXI has a clean Cartesian boundary along all four dimensions. It affects its environment through clear I/O channels with no side effects, its hypothesis space contains the environment, and there is a clear distinction between the agent and the environment. Embedded agents challenge these boundaries, and below I analyse where the Cartesian abstraction holds and fails for LLM systems.
In a narrow sense, LLMs seem like dualistic agents. Their “learning” and “action” phases are separated into training and inference, similar to how the computation logic of a dualistic agent remains outside the environment it interacts with and fixed when it is deployed. However the “environment” an LLM acts in is poorly defined. LLMs are not playing a video game - they are helping users play “the real world”. In a narrow sense, their environment is formed by what is built up in their context window and expanded by the agent on demand during the session with the user. There is no shared, global context across users, hence each agentic session creates its own environment. This framing achieves a clean separation between the model’s internals and its environment in the majority of use cases.
However, in a broader sense, next generation LLMs are trained on user interaction data from current LLMs (subject to privacy terms). All the individual sessions are combined together into the training data used to create an improved agent, forming a delayed feedback loop between learning and action. LLMs are increasingly integrated into all aspects of human life, and so in this view, their “environment” consists of all situations where human judgement can exist. The environment is then the full “real world” we inhabit, which by definition includes the environment the agent is trained in. The agent is part of the environment, smaller than it and made out of parts similar to it, characteristic of an embedded agent. Their I/O channels are well defined within a single user session and are the primary way the LLM affects its environment, but there are many second order effects, for example,
These effects are symptomatic of an agent embedded in the human world rather than separated from it.
LLMs are the most dualistic on the self reasoning and self improving axis. Their knowledge of themselves is not separate from their knowledge of all other things and they have no specific training on self reasoning. However, evaluations by Laine et al. (2024) have detected an emergent capability for situational awareness, which may well improve in tandem with general capabilities. LLMs are also not self improving as of yet, however research trajectories suggest that autonomous recursive self improvement may well be possible.
The Cartesian boundary between an LLM agent and its environment holds cleanly only within the immediate user interaction session. The distinction becomes less clear on analysis of LLM systems, including the delayed feedback loop of pre-training, training, inference, and second order effects on the environment that affect the next loop iteration. The dualistic framing is a convenient simplification that is becoming less accurate as the influence of agents expand with capability gain, richer tool systems and tighter training to deployment loops. It is with this understanding that I continue this exploration.
Decision Theory and World Modelling
The dualistic AIXI agent has a hypothesis space large enough to contain the true environment, which it discovers through Bayesian update. An embedded Bayesian agent is smaller than the environment and faces problems of counterfactual reasoning, self reference and a lack of realizability. LLMs are large scale probabilistic models that do not possess a formal decision making framework within which these problems can be defined, but encounter every one of them implicitly and without the formal mechanisms to detect or bound the resulting failures. Their counterfactual reasoning ability is emergent from the representation they learn from the reasoning implicit in their training data, unlike the principled reasoning process of a Bayesian agent. They reason about themselves in the same way they reason about everything else, a form of self reference. They operate in a non realizable setting as the true data generating process (the human world) is not contained in their hypothesis space as they are part of the human world and smaller than it. Classical Bayesian guarantees like convergence to the truth require realizability, and hence do not apply.
This lack of formal reasoning on an approximated world model gives rise to the characteristic unreliability of LLMs. Various techniques seek to improve on this, ranging from inference time methods like chain of thought reasoning to training time methods such as reinforcement learning with process supervision. Uncertainty quantification methods such as self reported confidence, token/logit scoring, semantic clustering, multi sample consistency and Bayesian inspired models help estimate the chance of error. In particular, conformal prediction can provide marginal coverage guarantees under exchangeability. However, this falls short of generalisable bounded correctness guarantees, and research by Agnimo et al. (2026) find a highly variable and often weak correlation between uncertainty estimators and hallucinations. Current empirical methods to improve model reasoning abilities and provide bounded error assurances work within the foundational limitation that LLMs do not follow any formal decision framework. Their lack of reliable verification ability appears to be the natural result of this limitation rather than an engineering quirk that can be fixed empirically.
Alignment
I find it useful to consider three broad categories of alignment research. Outer alignment considers whether the objective the model is trained on is a robust pointer to human values. Inner alignment is discussed in Embedded Agency as the problem of subsystem alignment, which considers whether the optimization of stochastic gradient descent finds an algorithm that pursues the objective, or whether it could instead find a mesa-optimiser that has a different goal but happens to perform well on the objective during training. Scalable oversight aims to address the risks from both in AI systems that exceed human intelligence. I explore a selection of research from all topics below.
Outer alignment
Human values are elusive and complex, taking on different forms, tradeoffs and nuances across cultures and contexts. The UN Declaration of Human Rights, one of the most widely accepted statements of human values, was a “colossal task” to write that involved an extensive consultation process with member states. Despite this, criticisms persist that it holds Western biases and has a narrow focus on the individual to the detriment of minority and collective rights. Implementations of the declaration in member states also vary greatly depending on culture, economic conditions, and political motives. If humans struggle to agree on a definition and interpretation of our values, it seems similarly difficult to create “stable pointers of value” for an agent to follow. An ideal pointer would need to robustly refer to our values, resist reward hacking under optimisation pressure (adversarial Goodhart) and scale with capability gains and self improvement.
A variety of empirical techniques allow LLMs to learn some representation of human values. RLHF is a standard training step that aims to refine the model into a safe and helpful assistant. RLHF avoids the challenge of defining human values by instead providing examples of preferred behaviour, with the expectation that the model learns a robust representation of human values. Ouyang et al. (2022) found that RLHF trained models produce outputs that are clearly preferred over baselines, with outputs from a 1.3B-parameter InstructGPT model with RLHF preferred by human labelers over a much larger 175B GPT-3 model with no RLHF. However, as the reward model learns a proxy to human values rather than a “true” representation, it is vulnerable to Goodhart problems. RLHF is sensitive to relative scale, and Gao et al. (2022) show that when the policy gets stronger relative to the reward model, gold reward first increases, then decreases as the policy overfits the proxy reward model, an instance of extremal Goodhart. Sharma et al. (2025) finds that RLHF is also subject to human bias, with users often preferring responses that align with their views, which can encourage models to prefer sycophancy over truth. The scalability of RLHF is also unresolved, with Perez et al. (2022) finding inverse scaling relationships where more RLHF made models less aligned.
Constitutional AI provides a set of written principles for a LLM model to follow that are aligned with human values. The model is provided a constitution written in human language and given adversarial prompts designed to elicit harmful behaviour. The model grades its own responses against its constitution and revises its output accordingly. The constitution is transparent, auditable and can be changed to align with new policies. Bai et al. (2022) find that RLAIF/Constitutional-AI models can maintain similar levels of harmlessness while requiring much less human labelling than RLHF models. However, the “stable pointers to value” problem persists as the constitution itself is an imperfect and incomplete representation of human values, reflecting the world view, culture and biases of the constitution authors. Collective Constitutional AI improves on this by sourcing input from a representative sample of the general public. The constitution is a single point of failure as any errors, inconsistencies, ambiguities and biases within it are inherited by the model to be trained. A further challenge persists in the model’s interpretation of the constitution, done through the lens of its understanding gained in pre-training, bringing into question the extent to which Constitutional AI can change fundamental beliefs ingrained in the model. A self reinforcement loop can be created as the model critiques itself, such that existing biases and blind spots may not be detected as the critic shares those same features.
RLHF and Constitutional AI (or similar RLAIF methods) have both played a large role in the alignment of frontier AI models to human values. However, both approach alignment as a safety engineering problem rather than a systematic alignment guarantee, so both require humans to monitor effectiveness. As LLM systems increase in capability, an open question remains whether they will continue to scale, and whether humans will be able to evaluate the effectiveness of these methods in order to answer the first question if LLM systems surpass human intelligence.
Inner alignment
A major obstacle to inner alignment is that misaligned behaviour is hard to detect in the first place. Evals and red teaming are two common methods that assess a model’s responses to deliberately crafted prompts. An eval is a set of test inputs designed to measure some attribute of the model, useful as a standard benchmark to compare different models and trends over time. They provide an easy to understand overview of the safety of a model in known risk areas, and are increasingly used in regulatory assessments to inform public policy. Red teaming involves humans or other AI agents trying to induce misaligned behaviours in models through deliberately crafted prompts, stress testing safety guardrails and helping to detect inputs where they fail. These inputs can then be used in adversarial training to harden the model against them. Red teaming has detected behaviour consistent with mesa-optimisation in experimental conditions, such as alignment faking, deception and scheming, and blackmail under replacement threats or goal conflict. Whether these reflect a genuine mesa-optimiser or context driven imitation of goal directed behaviour is contested, but regardless they surface a real risk of malicious and unexpected behaviour at deployment time that might otherwise go undetected.
Evals and red teaming both materially improve the safety of current models, however they share a structural limitation in that they can only probe a finite set of anticipated inputs. Meanwhile, the set of possible inputs to a LLM is infinite, so these techniques can never know what has not been found. They may also fail to detect a sufficiently sophisticated mesa-optimiser, which conditions its behaviour based on whether it believes it is being observed. The Sleeper Agents (2024) experiment found that standard safety training failed to remove an implanted backdoor and sometimes taught the model to hide it better. The training mechanism of stochastic gradient descent is a blunt instrument that is hard to steer towards a goal like “remove behaviour” without instead rewarding a neighbouring goal such as “conceal behaviour”. Qi et al. (2024) observe shallow safety alignment, where safety training is concentrated in the first few tokens of output, leaving the model susceptible to adversarial attacks in later stages of output. Evals, red teaming and adversarial training are all valuable empirical methods that reduce observed instances of misalignment, but they cannot quantify the misalignment that has not yet been found, nor provide a route to systematically remove it.
These limits motivate looking inside the model itself. Mechanistic interpretability aims to reverse engineer the internal computations of neural networks, such that they can be audited and understood. Where evals and red teaming probe behaviour from the outside, mechanistic interpretability aims to work from the inside by checking that the agent’s reasoning process is aligned. The related problem of eliciting latent knowledge (ELK) targets the adversarial case, aiming to extract the knowledge of an AI in situations where it may be attempting to deceive. There has been promising empirical progress, eg linear probes that detect directions correlated with truth values (Burns et al. (2024)), sparse autoencoders that extract interpretable features (Cunningham et al. (2023)), and activation steering that can amplify truthful behaviour at test time (Góral et al. (2025)). However, while these methods can find meaningful internal representations, Basu et al. (2026) suggest that they cannot comprehensively map out a network’s thought process, verify alignment, or reliably detect errors. Mechanistic interpretability is a promising research direction that requires more development.
Scalable oversight
The risk of misaligned LLMs to human safety is currently controllable as humans can effectively monitor and change them. However, depending on our greater intelligence to monitor LLMs is not scalable as there is a real possibility that LLMs may achieve superintelligence in the near to medium term future. Scalable oversight is the empirical instantiation of what Embedded Agency calls robust delegation, aiming to solve how a less capable overseer can delegate to, and trust, a more capable successor.
AI safety via debate is an approach where two AI agents are set up to argue opposing perspectives on a question to help a human judge decide which one is correct. The technique rests on the assumption that under optimal play, an honest debater has the advantage that they would just need to expose the weakness in the opposing case that the judge can cheaply verify to determine truthfulness. The approach has some scalable attributes as it is easier to determine correctness when one AI is tasked with exposing flaws in the other’s arguments. Michael et al. (2023) find that debate usually shows a marked improvement over consultancy and direct answer checking. However, the central assumption that the honest AI can always find the flaw in the opposing argument, does not always hold. It was challenged by the obfuscated arguments problem, a class of arguments where locating the wrong step is an exponentially expensive task. As the honest debater cannot localise the flaw within its budget, the judge may not be able to distinguish the obfuscated false argument from a sound one. The approach faces further, practical difficulties. The system is sensitive to relative scale, as the agents must have similar capability levels, else the stronger model may be able to trick the weaker one. It is moot whether humans will be able to evaluate the relative capabilities of two superintelligent agents in order to set up the debate validly, or whether there will always be two different AI models of similar levels of intelligence. Two instances of the same agent debating risks systematically advantaging one point of view due to shared biases or hidden goals, and the models will share the same blind spots that could skew the effectiveness of the debate. The agents could collude to mislead the judge, long and complex debates are difficult for a human to follow, and many questions may be genuinely subjective or depend on unknowns such that there is no verifiable truth. Much like how the adversarial debate of a courtroom aids a judge in determining the truth, safety via debate seems to be a valuable way to increase the probability of alignment but cannot guarantee that a correct outcome will be reached.
Weak-to-Strong Generalization seeks to test whether a stronger model can be trained with a weaker supervisor, mirroring the situation that would arise if AI models surpassed human intelligence. On a given task, the weaker supervisor model provides noisy labels to the stronger student model to train on, with the hope that the student model is able to use its latent knowledge to generalise beyond the supervisor and recover some of the performance it would have achieved on ground truth labels. Weak-to-Strong Generalization found that performance gap recovered (PGR) across domains was mixed, with students recovering a substantial portion of the gap in NLP tasks, while performance on chess puzzles and in particular reward modelling showed a much smaller recovery. It was also observed that the student models could overfit to supervisor errors, and it seems possible that Goodhart style problems may arise. Follow on research has confirmed weak to strong generalisation as a real phenomenon, but has not yet proven its scalability. My concern is that because this research program is driven by empirical findings around which theory is later constructed, the resulting theories may fail outside the experimental setups in which they were formed. Critically, we may be unable to evaluate the effectiveness of weak-to-strong generalisation in precisely the cases where AI safety most depends on it.
Scalable oversight aims to resolve the paradox of less intelligent humans tasked with overseeing the actions and behaviours of more intelligent AI agents. This is currently a theoretical concern, but one that could emerge in practice in the foreseeable future. Current research has not proved the feasibility of scalable oversight, and more time and investment is required to continue this line of enquiry.
Closing Thoughts
Through the lens of Embedded Agency, the lack of error bounded accuracy and risk of misalignment from LLM systems start looking like natural consequences of agents that were never built with a formal decision framework or world model. The theory of AI agents has evolved largely in isolation from deep learning, though the two have recently begun to converge. Agent theory has taken a top down, normative approach, exploring how to create an “optimal” agent that thinks, learns and acts following a formal framework. Meanwhile, deep learning originated from a bottom up, engineering mindset that favoured experimentation and empirical results to achieve impressive progress without any formal framework. LLMs were not built upon a foundation of verifiable correctness or alignment to human values, and empirical safety research aims to supply these features after the fact. So far, they have provided safety patches upon this foundation, but I question if this will be enough. Finite testing can only cover a small part of an open ended deployment distribution, so it seems inevitable that loopholes and failures will continue to appear. Without foundational safety guarantees or scalable oversight methods, we may lack the means to evaluate or control superhuman AI systems.
As a newcomer to the field, I expect parts of this analysis to be incomplete or mistaken, and I welcome any corrections. I wonder if a more formal approach to creating AI agents is required to ensure safety. Research programmes around provably safe AI such as Davidad’s Guaranteed Safe AI and Tegmark and Omohundro’s Provably safe systems aim to introduce mathematically verifiable safety guarantees. Of course, provably safe AI may not scale to the capabilities frontier, and so mainstream safety research and alternative approaches are all valuable lines of inquiry. I sincerely hope that these research efforts will be able to create systematic safety assurances around advanced AI or else provide an explanation of why this is not possible.