TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata.
A Theory of What?
The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment[1], and use this model to build a long-term plan. Moreover, it has to continually update the model and revise the plan with new information and additional processing time. Additionally, it has to accomplish all of that while making efficient use of the available information (statistical efficiency) and computing resources (computational efficiency).
The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI we are trying to avoid or counteract; and (iii) the aligned ASI we are trying to create (such as the GDS). Rocks, bacteria, and thermostats are non-examples: while a bacterium or a thermostat can be regarded as steering the environment, both are very limited in the variety of patterns in the environment they can exploit. For instance, if you switch the "cool" and "heat" outputs of a thermostat, it will not understand what has gone wrong and adapt its behavior. Animals and contemporary AIs are marginal cases (it's not obvious whether they should count).
Given that agents are at the core of both the threat and the proposed solution, the theoretical foundation for ASI alignment requires a theory of agents. Now, concepts adjacent to "agent" have a considerable history of academic study, but different research communities approach the subject very differently—compare the frameworks used in decision theory (and, by extension, in game theory and economics), Hutter's "universal intelligence", and reinforcement learning theory. It is therefore important to understand what the scope and level of abstraction of the theory of agents relevant for ASI alignment should be. To this end, I propose several desiderata.
Descriptive
The theory should be descriptive, i.e. applicable to all our main examples of agents. For any such example, the theory should ascribe specific preferences, beliefs, and any other alignment-relevant attributes to the agent[2]. These ascriptions should be grounded in the general philosophical soundness of the theory, not in domain knowledge about the specific examples; due to hypercomplexity, it is unlikely that domain knowledge will suffice[3]. In particular, if we imagine an alien civilization facing the AI alignment problem, or an AI agent facing the alignment problem for successor agents it wants to create, the theory should be about equally useful to them[4].
This means that agent theory should deconfuse us about philosophical questions pertaining to human preferences, such as, for example, the Caplan-Alexander debate on whether mental illness should be thought of as "unusual preferences". This doesn't mean that the answer to any particular question of this kind must be obvious from the theory, as it might come down to idiosyncratic nuances of the human brain. It does mean that if we also knew all pertinent facts about the physical world, we would be able to answer the question (or at least quantify the residual uncertainty), at least in principle. Actually inferring the answer from these facts might still be hard, just as explaining emergent physical phenomena can be hard even if all the relevant fundamental laws are known.
On the other hand, descriptiveness doesn't require that concepts such as "human preferences" be perfectly unambiguous. It requires only that, to the extent they remain ambiguous, this ambiguity should not allow critical errors when Goodhart's Curse is accounted for. It is not obvious that these concepts can be made perfectly precise, but it is clear that the alignment problem is fundamentally normative (i.e. it is rooted in our preferring certain AI futures over other AI futures). When zooming in to a resolution at which our preferences are ill-defined, it becomes immaterial which choice to make, since our preferences are the only impetus for distinguishing these choices. Any imprecision that is inevitable is also tolerable.
For example, imagine that Alice prefers to eat apples approximately 3 times as often as bananas, but there is no fully principled answer to the question "does Alice prefer to eat apples 2.99 times as often as bananas, or 3.01 times, or some number in the middle?" Then, it doesn't matter whether the AI aligned to Alice according to our theory will interpret her preference as 2.99 or 3.01—both are acceptable outcomes.
In the mathematical formalism of the theory, this ambiguity might manifest as something like the choice of the universal Turing machine (UTM) in Solomonoff induction. Different UTMs correspond to priors that differ merely by a multiplicative constant, but for particular pairs of UTMs this constant can be arbitrarily large. Then, a reasonable choice might be any machine that we would intuitively consider "simple" or "natural". This doesn't nail down a single UTM, but might come close enough for the differences not to matter in practice[5].
This situation is not fundamentally different from that of most concepts in science. For example, astrophysicists find no difficulty discussing the "center of mass of the Sun", even though strictly speaking it is ill-defined: the boundary of the Sun is fuzzy, particle positions are only well-defined down to the scale of their Compton wavelength, and even the notion of space itself becomes questionable in quantum gravity.
Notably, a descriptive theory must neither underspecify nor overspecify what counts as an agent. For example, a theory that mandates that "agents" use stochastic gradient descent would probably exclude humans, and is hence overspecified. On the other hand, a theory that accepts anything that maximizes the expectation of some utility function as an "agent" would conclude that even a rock is an agent and would severely underspecify agent attributes (see e.g. Shah 2018). The correct theory should fall into neither extreme.
Metareflective
The theory should allow us to describe agents that learn, know, or use facts from agent theory itself. For example, it should be possible to:
Describe an agent that plans to construct a successor agent but faces an alignment problem, and show how such an agent can overcome the alignment problem by learning and applying agent theory (all within the framework of agent theory itself).
Describe an agent that identifies other agents in the universe and infers their preferences, beliefs, or other attributes.
Describe an agent learning about a bound satisfied by a particular algorithm pertinent to agent theory, e.g. a statistical complexity bound.
If a theory satisfies this desideratum, I say that it is metareflective[6].
There are several ways to see why metareflectiveness is important:
If our theory is not metareflective, it might mean that it doesn't apply to humans, and hence it also isn't descriptive.
The straightforward approach to solving alignment grounding is for the AI agent to infer the user's attributes by using (parts of) agent theory in its own reasoning.
Metareflectiveness seems necessary to properly model recursive self-improvement. Recursive self-improvement is a factor that exacerbates AI risk, and hence the failure to model it might be indicative of a crucial gap in the theory.
Classical learning theory is not metareflective: there is no standard way to describe how learning algorithms can learn learning theory itself. On the other hand, formal logic has at least some metareflective flavor, as it is possible to, for example, represent the notion of "proving a theorem in Peano Arithmetic" within Peano Arithmetic itself. However, such metareflection has inherent limitations because of Goedel's theorems. This is the origin of the "Loebian obstacle" in MIRI's investigation of tiling agents. The theory of agents should do better.
Comparative
The theory should be comparative, meaning that it allows comparing agents along two (possibly entangled) dimensions:
How similar are attributes of these agents such as preferences and priors?
How powerful are the agents in terms of their ability to construct sophisticated models and plans?
Indeed, we cannot formalize alignment if there is no way to compare the user agent to the AI agent. For the theory to be useful, it needs to be able to describe agents that are similar in their preferences and other relevant attributes, and at the same time far apart in their power (for the AI agent to implement GDS, it needs to substantially surpass the user). Moreover, the threat model is rooted in a comparison as well: the danger comes from an AI agent whose power is superhuman and whose attributes are not especially human-like.
In particular, as I argued in the previous article, the comparability of attributes requires some form of metaphysical realism: there needs to be a single ontology—the external reality—in which the preferences and beliefs of different agents are grounded. This is in contrast to most existing frameworks for agency, which take the "cybernetic" approach of grounding all attributes in the agent's direct actions and observations. It is insufficient to have a theory which meets the "descriptive" desideratum by ascribing such cybernetic attributes to real-world agents.
Another consequence is that computational resource bounds must play a central role—it is insufficient to use idealized models like AIXI. Indeed, without computational bounds every optimization problem is brute-forceable, so why would any one agent be more powerful than another?[7] As an analogy, suppose we are trying to create a formal theory of chess-playing-power. Without computational constraints, it's trivial to write an algorithm that finds the perfect move every time. So, there are essentially only two levels of chess-playing-power: perfect play and everything else. Only when we introduce computational constraints do the finer gradations we track in real life become natural.
In a computationally unbounded universe, any notion of "agent" would be qualitatively different. In a universe in which resource bounds exist but have a profoundly different nature (e.g. pure space bounds or Turing degrees), relevant phenomena might exist, but the analogy with our universe is suspect at best.
Aggregative
The theory of agents should not only describe individual agents but also give an account of the possible interactions of agents, and the possible outcomes or equilibria produced by such interactions. In particular, it should also describe conditions under which a collective of agents can be regarded as a single agent (see also Kulveit 2024 and my review thereof). I will call such a theory aggregative.
There are multiple reasons why this is necessary. The most critical reason, in my view, is that we can never assume that the agent is perfectly aligned. This is because the need to make decisions under time pressure and the possibility of irreversible mistakes imply that the agent might make critical decisions without having enough information for perfect alignment. This means that the effects of residual misalignment between the user and the AI agent need to be understood.
Another way to view it is, the residual uncertainty of the AI agent about the user means that the agent needs to align itself to the aggregate of possible counterfactual users. We therefore need to understand how to view this collective as a single alignment target. For example, suppose the AI agent has a belief represented by a probability distribution over a parameter space which parametrizes the unknown user utility function . It is sometimes assumed that the agent should effectively use the utility function (see e.g. Hadfield-Menell et al. 2016). However, this prescription is suspect: a utility function is only defined up to a positive affine transformation, so the user's utility function can be described equally well by , for any functions and . This is a problem because is not, in general, an affine transformation of , and hence the naive prescription is ill-posed. While we can try to salvage it using some ad hoc normalization, an aggregative theory should suggest a principled solution instead.
There are also other reasons to care about aggregation:
If multiple superintelligent agents emerge at the same time, understanding their interaction can be important. For example, if these agents aren't all aligned to each other (whether because some of them are not human-aligned at all, or because different agents are aligned to different humans), will they wage destructive wars, possibly devastating humanity as collateral damage? (See also Clifton 2019.)
It is potentially desirable to align AI agents to multiple human users, maybe even to entire populations or to all of humanity. For such a solution, we need to account for the interaction between the human agents, such as the resulting mechanism design challenges.
Superintelligent agents might engage in acausal trade, and we need to understand the consequences. This might seem like a fantastical possibility; however, we inherently need to be prepared for the "fantastical" when dealing with superintelligence. As our own abilities and knowledge would seem fantastical to people who lived centuries ago, so might the abilities and knowledge of a superintelligence that can make scientific, technological, and arguably even philosophical progress much faster than human beings.
There is of course a vast literature on the topic of multi-agent rationality, most notably game theory. However, we need to be prepared to reexamine some of the common assumptions in this literature. In particular, most of game theory is centered on the concept of Nash equilibrium. However, the Nash equilibrium is suspect for multiple reasons:
It is PPAD-complete and therefore likely to be computationally intractable.
Learning is a central ingredient in agency, which motivates special attention to repeated games with long time horizons. However, the folk theorems imply that in such games the -Nash condition is very weak, justifying any payoff vector above an extremely permissive threshold. (This does make finding an -Nash equilibrium computationally tractable in such games, but that is somewhat cold comfort.)
The fundamental justification for Nash equilibria comes from Causal Decision Theory, which has itself been the subject of criticism in the context of agent theory motivated by AI alignment (see Yudkowsky and Soares 2017).
Another field of study concerned with multi-agent interactions is multi-agent reinforcement learning (see e.g. Zhang 2019). However, most of the work in that field is concerned with the construction of agents with specific aggregative properties, often making assumptions such as a common utility function, centralized control, or at least a single algorithm. While these assumptions are understandable for the more mundane applications, they are mostly unworkable for our purpose. Instead, we need a theory that is simultaneously descriptive and aggregative, one that can explain the emergence of collective behaviors from separate agents of different nature and origin, rather than merely engineering them. Indeed, while we can engineer our AI agent, we cannot engineer humans, or the counterfactual users imagined by the AI agent, or acausal trade partners, or even other competing AI agents necessarily.
Constructive
Last but not least, the theory should be constructive, i.e. produce practical algorithms that can be implemented as real-world AI systems. Moreover, these algorithms must be equipped with quantitative performance guarantees. It is not enough for the theory to provide abstract or idealized models. In particular, this is another reason to focus on computational resource constraints, beyond comparability.
Indeed, if the theory did not yield practical algorithms, how would we apply it? We may be tempted to rely on heuristics, but these heuristics are liable to invalidate our safety guarantees. At the very least, the theory has to produce a formal specification such that any heuristic output that certifiably meets the specification is safe. (That would be the "formal verification" approach, advocated by e.g. Dalrymple et al. 2024.) Such a formal specification would already put the problem in , and hence involve substantial grappling with computational resource constraints. Moreover, I'm currently unaware of a promising path to place the problem in (i.e. create a sound and complete efficiently-verifiable specification) which doesn't bring it all the way down to (i.e. create a provably-correct efficient algorithm for the agent).
In particular, the treatment of computational constraints cannot be a mere reduction to certain types of "oracles", as is popular in the contemporary machine learning theory literature (see e.g. Dudik et al. 2016). Such a reduction passes the buck to how these oracles are supposed to be implemented, and what guarantees we have (or can at least rigorously conjecture) about the implementation. (Of course, this can still be insightful, as it allows us to narrow down where the hard part might be located.)
Nor is it sufficient to make do with asymptotic guarantees (e.g. the guarantees typically studied in singular learning theory) without at least a rough quantitative analysis. Without results such as quantitative statistical complexity bounds, it is impossible to guarantee alignment convergence. We would be taking our chances with agents that would converge to alignment in some imaginary asymptotic limit, but take catastrophic actions long before approaching that point.
The constructive nature of the theory does imply that it is potentially dual-use: it can make AI safer, but it might also make AI more powerful. This seems inevitable: the deep understanding of any phenomenon is inherently relevant to all applications of the phenomenon. It requires the researchers to be prudent about which results should be revealed to which parties at what time.
Conclusion
I have described my five primary desiderata for a theory of agents: the theory should be descriptive, metareflective, comparative, aggregative, and constructive. Hopefully, this gives readers a preliminary understanding of what sort of theory we are looking for, which will help motivate the subsequent (more technical) exposition. In the following articles we will go through a high-level description of computational agent theory and see how it corresponds with these desiderata. We will also discuss the Computational Superimitation proposal and see how it leverages these desiderata to address the core challenges of the alignment problem.
I thank Roman Malov, Vinayak Pathak, and my beloved spouse @Marcus Ogren for diligently reviewing drafts. Their feedback greatly improved this article. I also thank Claude Opus 5.5 for helping to find sources and Claude Sonnet 5.5 for copyediting. All remaining flaws are entirely my own fault.
Humans create models; all algorithms from RL theory (both model-based and even the so-called model-free) create something that can be called a model; and contemporary AI can at least answer questions about the world with a small inference budget. The good regulator theorem is an attempt to formalize this claim, but for the purpose of this article I only use "model" as an informal intuition pump. ↩︎
There doesn't have to be a binary dichotomy between "agents" and "non-agents". For example, it's conceivable to have a scalar measure of how agentic a system is, and the more agentic it is, the more unambiguously we can ascribe various attributes to it. ↩︎
Domain knowledge such as neuroscience can still be valuable to help validate the theory and/or shape the prior or inductive bias of the agent to make alignment learning more efficient and robust. ↩︎
Perhaps for an AI agent the problem would be easier since such an agent can easily access its own source code and internal state. ↩︎
To be clear, this is just an illustrative metaphor. I don't claim that Solomonoff induction is "correct" in some objective sense. Neither do I claim that there are no contexts in which something akin to choosing the UTM should be done in some other manner (e.g. having the AI learn it from the user). ↩︎
I use the word "metareflective" rather than just "reflective" to avoid confusion with reflective stability, a different desideratum that I will touch on in later articles. ↩︎
This is not to say that ASI must have more compute than the human brain. It can have less compute but use it more efficiently. But if the resources are unbounded, then efficiency doesn't matter. ↩︎
TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata.
A Theory of What?
The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment [1] , and use this model to build a long-term plan. Moreover, it has to continually update the model and revise the plan with new information and additional processing time. Additionally, it has to accomplish all of that while making efficient use of the available information (statistical efficiency) and computing resources (computational efficiency).
The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI we are trying to avoid or counteract; and (iii) the aligned ASI we are trying to create (such as the GDS). Rocks, bacteria, and thermostats are non-examples: while a bacterium or a thermostat can be regarded as steering the environment, both are very limited in the variety of patterns in the environment they can exploit. For instance, if you switch the "cool" and "heat" outputs of a thermostat, it will not understand what has gone wrong and adapt its behavior. Animals and contemporary AIs are marginal cases (it's not obvious whether they should count).
Given that agents are at the core of both the threat and the proposed solution, the theoretical foundation for ASI alignment requires a theory of agents. Now, concepts adjacent to "agent" have a considerable history of academic study, but different research communities approach the subject very differently—compare the frameworks used in decision theory (and, by extension, in game theory and economics), Hutter's "universal intelligence", and reinforcement learning theory. It is therefore important to understand what the scope and level of abstraction of the theory of agents relevant for ASI alignment should be. To this end, I propose several desiderata.
Descriptive
The theory should be descriptive, i.e. applicable to all our main examples of agents. For any such example, the theory should ascribe specific preferences, beliefs, and any other alignment-relevant attributes to the agent [2] . These ascriptions should be grounded in the general philosophical soundness of the theory, not in domain knowledge about the specific examples; due to hypercomplexity, it is unlikely that domain knowledge will suffice [3] . In particular, if we imagine an alien civilization facing the AI alignment problem, or an AI agent facing the alignment problem for successor agents it wants to create, the theory should be about equally useful to them [4] .
This means that agent theory should deconfuse us about philosophical questions pertaining to human preferences, such as, for example, the Caplan-Alexander debate on whether mental illness should be thought of as "unusual preferences". This doesn't mean that the answer to any particular question of this kind must be obvious from the theory, as it might come down to idiosyncratic nuances of the human brain. It does mean that if we also knew all pertinent facts about the physical world, we would be able to answer the question (or at least quantify the residual uncertainty), at least in principle. Actually inferring the answer from these facts might still be hard, just as explaining emergent physical phenomena can be hard even if all the relevant fundamental laws are known.
On the other hand, descriptiveness doesn't require that concepts such as "human preferences" be perfectly unambiguous. It requires only that, to the extent they remain ambiguous, this ambiguity should not allow critical errors when Goodhart's Curse is accounted for. It is not obvious that these concepts can be made perfectly precise, but it is clear that the alignment problem is fundamentally normative (i.e. it is rooted in our preferring certain AI futures over other AI futures). When zooming in to a resolution at which our preferences are ill-defined, it becomes immaterial which choice to make, since our preferences are the only impetus for distinguishing these choices. Any imprecision that is inevitable is also tolerable.
For example, imagine that Alice prefers to eat apples approximately 3 times as often as bananas, but there is no fully principled answer to the question "does Alice prefer to eat apples 2.99 times as often as bananas, or 3.01 times, or some number in the middle?" Then, it doesn't matter whether the AI aligned to Alice according to our theory will interpret her preference as 2.99 or 3.01—both are acceptable outcomes.
In the mathematical formalism of the theory, this ambiguity might manifest as something like the choice of the universal Turing machine (UTM) in Solomonoff induction. Different UTMs correspond to priors that differ merely by a multiplicative constant, but for particular pairs of UTMs this constant can be arbitrarily large. Then, a reasonable choice might be any machine that we would intuitively consider "simple" or "natural". This doesn't nail down a single UTM, but might come close enough for the differences not to matter in practice [5] .
This situation is not fundamentally different from that of most concepts in science. For example, astrophysicists find no difficulty discussing the "center of mass of the Sun", even though strictly speaking it is ill-defined: the boundary of the Sun is fuzzy, particle positions are only well-defined down to the scale of their Compton wavelength, and even the notion of space itself becomes questionable in quantum gravity.
Notably, a descriptive theory must neither underspecify nor overspecify what counts as an agent. For example, a theory that mandates that "agents" use stochastic gradient descent would probably exclude humans, and is hence overspecified. On the other hand, a theory that accepts anything that maximizes the expectation of some utility function as an "agent" would conclude that even a rock is an agent and would severely underspecify agent attributes (see e.g. Shah 2018). The correct theory should fall into neither extreme.
Metareflective
The theory should allow us to describe agents that learn, know, or use facts from agent theory itself. For example, it should be possible to:
If a theory satisfies this desideratum, I say that it is metareflective [6] .
There are several ways to see why metareflectiveness is important:
Classical learning theory is not metareflective: there is no standard way to describe how learning algorithms can learn learning theory itself. On the other hand, formal logic has at least some metareflective flavor, as it is possible to, for example, represent the notion of "proving a theorem in Peano Arithmetic" within Peano Arithmetic itself. However, such metareflection has inherent limitations because of Goedel's theorems. This is the origin of the "Loebian obstacle" in MIRI's investigation of tiling agents. The theory of agents should do better.
Comparative
The theory should be comparative, meaning that it allows comparing agents along two (possibly entangled) dimensions:
Indeed, we cannot formalize alignment if there is no way to compare the user agent to the AI agent. For the theory to be useful, it needs to be able to describe agents that are similar in their preferences and other relevant attributes, and at the same time far apart in their power (for the AI agent to implement GDS, it needs to substantially surpass the user). Moreover, the threat model is rooted in a comparison as well: the danger comes from an AI agent whose power is superhuman and whose attributes are not especially human-like.
In particular, as I argued in the previous article, the comparability of attributes requires some form of metaphysical realism: there needs to be a single ontology—the external reality—in which the preferences and beliefs of different agents are grounded. This is in contrast to most existing frameworks for agency, which take the "cybernetic" approach of grounding all attributes in the agent's direct actions and observations. It is insufficient to have a theory which meets the "descriptive" desideratum by ascribing such cybernetic attributes to real-world agents.
Another consequence is that computational resource bounds must play a central role—it is insufficient to use idealized models like AIXI. Indeed, without computational bounds every optimization problem is brute-forceable, so why would any one agent be more powerful than another? [7] As an analogy, suppose we are trying to create a formal theory of chess-playing-power. Without computational constraints, it's trivial to write an algorithm that finds the perfect move every time. So, there are essentially only two levels of chess-playing-power: perfect play and everything else. Only when we introduce computational constraints do the finer gradations we track in real life become natural.
In a computationally unbounded universe, any notion of "agent" would be qualitatively different. In a universe in which resource bounds exist but have a profoundly different nature (e.g. pure space bounds or Turing degrees), relevant phenomena might exist, but the analogy with our universe is suspect at best.
Aggregative
The theory of agents should not only describe individual agents but also give an account of the possible interactions of agents, and the possible outcomes or equilibria produced by such interactions. In particular, it should also describe conditions under which a collective of agents can be regarded as a single agent (see also Kulveit 2024 and my review thereof). I will call such a theory aggregative.
There are multiple reasons why this is necessary. The most critical reason, in my view, is that we can never assume that the agent is perfectly aligned. This is because the need to make decisions under time pressure and the possibility of irreversible mistakes imply that the agent might make critical decisions without having enough information for perfect alignment. This means that the effects of residual misalignment between the user and the AI agent need to be understood.
Another way to view it is, the residual uncertainty of the AI agent about the user means that the agent needs to align itself to the aggregate of possible counterfactual users. We therefore need to understand how to view this collective as a single alignment target. For example, suppose the AI agent has a belief represented by a probability distribution
There are also other reasons to care about aggregation:
There is of course a vast literature on the topic of multi-agent rationality, most notably game theory. However, we need to be prepared to reexamine some of the common assumptions in this literature. In particular, most of game theory is centered on the concept of Nash equilibrium. However, the Nash equilibrium is suspect for multiple reasons:
Another field of study concerned with multi-agent interactions is multi-agent reinforcement learning (see e.g. Zhang 2019). However, most of the work in that field is concerned with the construction of agents with specific aggregative properties, often making assumptions such as a common utility function, centralized control, or at least a single algorithm. While these assumptions are understandable for the more mundane applications, they are mostly unworkable for our purpose. Instead, we need a theory that is simultaneously descriptive and aggregative, one that can explain the emergence of collective behaviors from separate agents of different nature and origin, rather than merely engineering them. Indeed, while we can engineer our AI agent, we cannot engineer humans, or the counterfactual users imagined by the AI agent, or acausal trade partners, or even other competing AI agents necessarily.
Constructive
Last but not least, the theory should be constructive, i.e. produce practical algorithms that can be implemented as real-world AI systems. Moreover, these algorithms must be equipped with quantitative performance guarantees. It is not enough for the theory to provide abstract or idealized models. In particular, this is another reason to focus on computational resource constraints, beyond comparability.
Indeed, if the theory did not yield practical algorithms, how would we apply it? We may be tempted to rely on heuristics, but these heuristics are liable to invalidate our safety guarantees. At the very least, the theory has to produce a formal specification such that any heuristic output that certifiably meets the specification is safe. (That would be the "formal verification" approach, advocated by e.g. Dalrymple et al. 2024.) Such a formal specification would already put the problem in
In particular, the treatment of computational constraints cannot be a mere reduction to certain types of "oracles", as is popular in the contemporary machine learning theory literature (see e.g. Dudik et al. 2016). Such a reduction passes the buck to how these oracles are supposed to be implemented, and what guarantees we have (or can at least rigorously conjecture) about the implementation. (Of course, this can still be insightful, as it allows us to narrow down where the hard part might be located.)
Nor is it sufficient to make do with asymptotic guarantees (e.g. the guarantees typically studied in singular learning theory) without at least a rough quantitative analysis. Without results such as quantitative statistical complexity bounds, it is impossible to guarantee alignment convergence. We would be taking our chances with agents that would converge to alignment in some imaginary asymptotic limit, but take catastrophic actions long before approaching that point.
The constructive nature of the theory does imply that it is potentially dual-use: it can make AI safer, but it might also make AI more powerful. This seems inevitable: the deep understanding of any phenomenon is inherently relevant to all applications of the phenomenon. It requires the researchers to be prudent about which results should be revealed to which parties at what time.
Conclusion
I have described my five primary desiderata for a theory of agents: the theory should be descriptive, metareflective, comparative, aggregative, and constructive. Hopefully, this gives readers a preliminary understanding of what sort of theory we are looking for, which will help motivate the subsequent (more technical) exposition. In the following articles we will go through a high-level description of computational agent theory and see how it corresponds with these desiderata. We will also discuss the Computational Superimitation proposal and see how it leverages these desiderata to address the core challenges of the alignment problem.
I thank Roman Malov, Vinayak Pathak, and my beloved spouse @Marcus Ogren for diligently reviewing drafts. Their feedback greatly improved this article. I also thank Claude Opus 5.5 for helping to find sources and Claude Sonnet 5.5 for copyediting. All remaining flaws are entirely my own fault.
Humans create models; all algorithms from RL theory (both model-based and even the so-called model-free) create something that can be called a model; and contemporary AI can at least answer questions about the world with a small inference budget. The good regulator theorem is an attempt to formalize this claim, but for the purpose of this article I only use "model" as an informal intuition pump. ↩︎
There doesn't have to be a binary dichotomy between "agents" and "non-agents". For example, it's conceivable to have a scalar measure of how agentic a system is, and the more agentic it is, the more unambiguously we can ascribe various attributes to it. ↩︎
Domain knowledge such as neuroscience can still be valuable to help validate the theory and/or shape the prior or inductive bias of the agent to make alignment learning more efficient and robust. ↩︎
Perhaps for an AI agent the problem would be easier since such an agent can easily access its own source code and internal state. ↩︎
To be clear, this is just an illustrative metaphor. I don't claim that Solomonoff induction is "correct" in some objective sense. Neither do I claim that there are no contexts in which something akin to choosing the UTM should be done in some other manner (e.g. having the AI learn it from the user). ↩︎
I use the word "metareflective" rather than just "reflective" to avoid confusion with reflective stability, a different desideratum that I will touch on in later articles. ↩︎
This is not to say that ASI must have more compute than the human brain. It can have less compute but use it more efficiently. But if the resources are unbounded, then efficiency doesn't matter. ↩︎