There's a story I've heard a few times in agent foundations, that involves starting with a very idealized theory of agency, like AIXI, IB, or UDT, in which we can achieve strong, provable safety properties,[1] and gradually making it more realistic, until it can eventually be implemented, or at least used to model real agents accurately. I'll call this process "de-idealization".[2] In this case, it means creating a tractable approximation of the theory that is also embedded, self-modifying, hierarchical, etc., while hoping to keep some of what we liked about the original theory. I've recently become quite pessimistic about this story, mostly because I don't think such idealized theories tell us much about what the more realistic case will look like.
Firstly, I think it's very plausible that extensive de-idealization will result in a theory which allows only much weaker safety guarantees, if any at all. Trying to maintain specific guarantees is like trying to impose a strong constraint, in advance, on a process that is inherently unpredictable, and by default we should expect a sort of regression to the mean. Secondly, a long phase of de-idealization offers limited feedback loops (at least without significant experimental work) and hence limited opportunity to realize if we're on the wrong track. Finally, I think there's little precedent for this story succeeding in the past, even across a range of academic fields. I'll expand on each of these points below.
To be clear, I think de-idealization is important and good! What I'm concerned about is the specific way it's being done here, which I expect isn't very suitable for these theories and aims. In future writing I'll give some specific suggestions for changes to this process that I think would increase the prospects of progress and success in agent foundations.
Proof jenga
Let's start with a metaphor. Imagine you have a complete jenga tower, with full layers of blocks, and an elaborate stained glass square on top. Your task is to remove a certain number of blocks, without letting the glass fall off and shatter. The blocks are the idealized assumptions of your theory, and the glass is your safety guarantee. Maybe it's that your agent minimizes regret according to a specified utility function, or that it converges to the same utility function as another agent. The less realistic your assumptions are to start with, the more blocks you have to remove, and the more likely the tower will collapse, breaking your proof.
Of course, this is a little unfair. De-idealization is really more like replacing the blocks with different blocks, while also modifying the guarantee itself. And not every step necessarily involves weakening an assumption, but there are still typically many steps that do. So the tower can also be viewed as a Ship of Theseus, and the question then becomes whether what the eventual blocks can support is the kind of thing that we were hoping for originally. And there's a good reason to think that it won't be.
Regression to the mean
Currently, theories in agent foundations tend to look more impressive than theories in empirical AI safety, cognitive science, or even economics. They often have a very broad scope, e.g. claiming to describe a huge class of algorithms or reasoners, and yet still come with very strong formal properties. I think there are a couple of reasons for this. The theories that I'm talking about here are mostly motivated by conceptual or philosophical arguments, and although they often use powerful mathematical tools, the application of those tools does not have a strong basis in established empirical knowledge (unlike, e.g., theoretical physics). Furthermore, most of the work to develop them further, including de-idealization, is done a priori, with experiments playing only a small role. Hence, the theories have so far had limited contact with nature, and this is why they appear to describe agency with such rigor. In other words, we've taken a "measurement" of their impressiveness that is likely inaccurate, and through de-idealization, we should expect to get gradually more accurate measurements, which will reveal much less impressiveness. This could be via the scope of the theory narrowing significantly, so that it is only useful for a limited set of agents in limited conditions, or the guarantees weakening significantly, as described above.[3]
In other words, if we put our theories of agency on equal footing with other approaches to studying behavior, they might not stand out nearly as much. And one of the most insistent lessons of science is that “the first ninety-nine explanations you come up with are wrong” (Yudkowsky, 2010); in this case, the first ninety-nine attempts to improve our understanding of agency may well be failures.
The strong and modest ambitions of agent foundations
The criticisms I’ve described so far are mostly relevant to what I call the strong ambition of agent foundations: that it will eventually give us a principled account of the conditions required to achieve superintelligence-robust alignment, or even a construction for a system that fulfills these conditions. I think many in the field hold this ambition. Some work that is particularly emblematic of it involves trying to achieve optimal performance across more and more decision problems, which can sometimes look like normative reasoning about esoteric edge cases (e.g. malign priors, acausal blackmail, infinite ethics). This can lead to gradually stronger safety guarantees, but often at the cost of even greater idealization. Hence it does little to address the difficulty of finding effective ways to talk about realistic agents, which may then demand reconstruction of these decision problems and guarantees.
Of course, the strong ambition is not required to justify the value of a line of research. A more modest ambition is for agent foundations to simply make some progress in this direction, such as via red-teaming other alignment proposals, or by suggesting architectural modifications that would incrementally increase the robustness of safety measures. Indeed, agent foundations has already achieved these aims to some extent, but mostly via conceptual reasoning.[4] What remains in my view is to demonstrate that more elaborate formal frameworks add value beyond what we could already establish through conceptual reasoning. For instance, whether they predict novel failure modes in realistic systems, and whether any of those predictions are borne out. In this way the mathematical machinery can “earn its keep”.[5] This is one example of a kind of feedback that I think is lacking in agent foundations.
Feedback loops and generalization
There's no guarantee that we've stumbled upon a good theory of agency that will actually be useful for our aims. I think the motivations that are normally given for these theories are quite weak evidence of that. Indeed, many of these theories are heavily inspired by Expected Utility Theory (EUT), whose strongest justification is normative arguments about inexploitability. And empirical support for EUT is sorely lacking, e.g. it can predict human behavior only in narrow circumstances. We can try to defend EUT by viewing it as a normative standard which humans do not live up to,[6] but again, its normative merits alone provide little evidence of its relevance to realistic, generally capable agents.[7]
So what if our current theories of agency are completely wrong? That is, what if they simply fail to add to our knowledge about realistic (or even constructed) agents? There are many examples of reasonable and parsimonious theories in other domains that don’t pass this test, such as Aristotelian mechanics, which was central to discussions of motion for thousands of years, before the development of the modern scientific method revealed the inaccuracy of its empirical claims. This is one way that failure can occur. But it can also occur even with accurate predictions, such as in Ptolemaic astronomy with epicycles, which led to mathematical models that could often predict planetary motion more closely than previous methods. And yet, Ptolemaic astronomy failed to provide an explanation of the dynamics underlying planetary motion and therefore failed to generalize to less typical conditions or other physical phenomena. There’s a common thread with EUT here, which is that limited predictive success is not sufficient by itself to warrant committing to a theory, as it can sometimes result simply from overfitting on certain data. Attempting to generalize then provides further opportunity for feedback, particularly about the depth of our understanding, and to what extent it “carves nature at its joints”.
Importantly, this feedback from generalization is not immediate. It is acquired only through an iterative process, which oscillates between levels: the narrow and the general, the idealized and the realistic. In mathematical terms, generalization permits a larger class of models. An example with EUT is dropping the independence axiom, which allows for alternatives like Kahneman and Tversky's Cumulative Prospect Theory (CPT).[8] EUT can then be recovered as a special case of CPT, in which we make specific choices for probability weighting and the utility reference point. So EUT is only one of many special cases of CPT, which can be obtained by making different choices for these free parameters. Hence, the best way to get evidence about both whether CPT was a useful generalization, and whether EUT was a good starting point, is by trying to use the other, new, specialized models. If those models can, for instance, make more or better empirical predictions than the other models, this is a good sign.[9] But what if we’re unable to extract useful models from the generalization, despite our best efforts? In most cases, this will be negative evidence, e.g. if the new models make worse predictions. In some cases, it may be neutral, e.g. if it is very hard to re-specialize in other directions. But even then, we at least have some negative evidence about the tractability of the overall agenda.
Intuitively, CPT is a de-idealization of EUT. But it’s worth noting that the correspondence with generalization isn’t always so clean. For instance, continuous photon-counting models are technically generalizations, but they’re idealizations intuitively, because photons are actually quantized in reality. Nonetheless we can apply the same reasoning as above and evaluate idealizations (whether or not they are generalizations) by the eventual success of their de-idealizations. Just to drive the point home, we really don't know in advance how much theory modification this process will involve, though the significance of anomalies can be a guide. To correct for damping in an oscillator, you simply need to add one term to the equation. But correcting Newton's law of gravitation to predict the complete precession of Mercury's perihelion is much harder. These observations hence cast doubt on the form of the law itself, and led to more fundamental modifications, eventually culminating in general relativity (which nonetheless retains a strong connection to Newtonian gravity). So I argue that the subsequent success of general relativity in making novel predictions provided evidence for the value of Newton’s law in guiding our research directions.[10]
In the case of Ptolemaic astronomy, however, while epicycles were something of a de-idealization, and contributed to the predictive success of the theory, there wasn't really any way the theory could be modified further to get out something like Newtonian gravity. Nor can Ptolemy's theory now be recovered from the latter, or general relativity. This lack of such a structural connection makes it hard to view modern developments as continuing in the same research program (even despite its influence on Kepler and Copernicus).[11]
My overall claim here is that we should not treat generalization, or idealization, as intrinsically good; our generalizations should also “pay rent”, and those that don’t should be de-prioritized. I think this is part of what has made fields like theoretical physics so successful.
And without such a process, it becomes very hard to acquire strong evidence one way or another about our current theories. But I also think that we’ve seen plausible warning signs, like huge barriers to making the theories realistic, and requiring very sophisticated mathematical machinery to overcome these barriers (e.g. Infra-Bayesian Physicalism for embedded agency). Another possible red flag is needing to construct very strange mechanisms, such as updatelessness, in order to explain very ordinary behavior, such as precommitments. In other words, it often looks like we're fighting a very uphill battle when de-idealizing these theories, and if anything, that is evidence against us having stumbled upon good theories. As Scott Garrabrant has said: “The point where we discover updatelessness is the point where we are supposed to realize that all of utility theory is wrong” (Garrabrant, 2010).[12]
Historical precedent
What if we look for this story in other academic fields? Where has it panned out? Only very rarely in the natural sciences. Specifically, I think it’s a combination of features which is rare. One of them is starting with a very idealized theory, and another one is developing the theory mostly a priori. The third one is not drawing significantly on established empirical knowledge to come up with theory. I think that the strategy of idealizing into the platonic realm is most successful when we start with a firm footing in the real world. It’s not that we can’t access the platonic realm in other ways, but there is a very vast space of logical truths and mathematical models, and for a given practical problem, most of them won’t turn out to be that useful. If we have the opposite impression, I think it’s mostly because of a selection effect or survivorship bias: “You only ever hear about the cool math stuff that becomes useful later on, because that's so interesting; you don't hear about stuff that's left in the dustbin of history.”(Anonymous, 2025)
Even despite this, and the relative recency of trying to approximate idealized proofs in engineered systems, we can still find important examples of failure, which gives more cause for pessimism. These failures often involve the results of proofs failing to obtain in reality due to assumptions being insufficiently realistic, despite seeming reasonable at the time. It is also common for there to be relevant dynamics that were completely unknown in advance and therefore implicitly excluded from the models. These examples suggest that philosophical and mathematical rigor are not necessarily sufficient for getting things right “on the first critical try”, despite this being a commonly-cited motivation.
For instance, a popular past strategy for rebalancing portfolios, before Black Monday in 1987, used mathematical models to mimic put options, with provable guarantees such as a portfolio value floor. The assumptions of the model included price continuity and high liquidity, and it transpired that the real world did not sufficiently approximate these assumptions, and many portfolios fell below the provable floor.[13] Similarly, the dynamics leading to the collapse of Tacoma Narrows Bridge in 1940 were not adequately represented in the aerodynamic models that established confidence in its design.
I dislike appealing solely to intuition or a handful of examples to justify historical claims, so in the spirit of providing stronger evidence, I asked Claude and ChatGPT to look through every Nobel prize, and they judged that less than 10% of them match all of the aforementioned features, with the vast majority of examples being from economics. Excluding economics, the number is closer to a few percent. Unfortunately, this question is merely a proxy for the historical question we would most want to ask here: “What has been the success rate of deliberately trying to make major scientific advances with this methodology?” That question is impossible to answer due to the aforementioned survivorship bias. Nonetheless, if we backchain by conditioning on having achieved a major scientific advance, and ask what the most likely ways are that we got there, this methodology doesn’t seem to be one of them (on historical priors).[14] I think this is a fair reference class, because of what we’re trying to do here: find a way of developing a major new technology with strong safety guarantees. I’m agnostic here about whether this involves creating a novel architecture which is as capable as LLMs, or whether it involves understanding deep learning well enough to achieve strong safety guarantees with something like the current architecture.
But one could also argue that the reference class should be something more narrow, like computer science or economics, where these features are indeed more common. I think this is not at all obvious. On the contrary, humanity’s very conception of agency and our intuitions about it are primarily based upon biological entities that occur in the physical world, such as humans and other animals. Furthermore, even the development of modern AI itself was heavily inspired by biological neural networks. I’m not saying it’s obvious that biology is a better reference class, more so that agency encompasses a hodgepodge of things that don’t fit neatly into any single, existing scientific discipline, so the most defensible reference class is simply the entire category.
To be more specific, I see two important ways in which studying agency differs from standard computer science. Firstly, much of computer science historically has concerned objects which are explicitly specified (e.g. hand-coded), especially in cases where the field has delivered strong formal guarantees that actually obtain (even approximately) in the real world. Generally capable agents, including AI, are of course not like this, and while we might aspire to make AI more like this, we don’t yet know to what extent that is possible. Secondly, and relatedly, the systems in question have generally been much simpler than agents, e.g. their interactions with other systems / environments are more constrained, there are less moving parts, etc. So I expect that models of agents will be more chaotic and sensitive to small perturbations, which will plausibly come up a lot in de-idealization.
In my view, economics has the stronger case for relevance. Very idealized theories, a priori work, and even ex ante normative constraints, are all cornerstones of economics. And it also studies emergent complex systems (like the market). But so do most other areas of science. Hence I think we still need a positive argument if we are to lean so heavily upon economics as to use a methodology which is uniquely prevalent there. And I don’t have such an argument. Nor do I have a strong counter-argument, but I would note that finding very robust guarantees for such systems is more commonplace in the natural sciences (even when phase transitions are possible). If anything, many of the triumphs of economics have been in pointing out all the ways that systems in the real world (whether natural or artificial) differ from idealized models, without necessarily providing an achievable way of closing that gap. For all its usefulness, economics is still far from solving its own alignment problem (between humans and the market), and arguably this failure itself contributed to our desperate need to solve AI alignment now.
An aside on reflective stability
Don’t our theories of agency already make some empirical claims? For instance they claim that as AIs become more generally capable (and therefore more instrumentally rational), they will converge towards certain properties, such as "coherence" (e.g. being better described by EUT). Or, if AI is currently acting like CDT, it will start acting more and more like FDT.
These are certainly interesting empirical claims! The problem is that they could still be completely wrong,[15] and no one in agent foundations seems very interested in testing them empirically at the moment.[16] I personally would be excited about such work. It seems like AI is already getting close to being quite dangerous, so wouldn't it be reasonable to expect these trends to start appearing at least somewhat, if they're going to end up being relevant?
There’s also a version of these claims that can't really be tested empirically because they're actually more like "right before the singularity (and not any earlier), AI will finally instantiate [my favorite theory of agency], because of [philosophical arguments]".[17] Notice that you could insert basically any theory of agency here, which gives us little hope of discriminating between them confidently, even if one of the claims was true.
Proof by contradiction?
In this post I’ve tried to explain why I’m pessimistic about a common theory of change in agent foundations. To oversimplify, the argument is somewhat like a reductio ad absurdum: that subscribing to this story amounts to hoping for a rare, best-case outcome from an unpredictable process (de-idealization), with limited opportunity for course-correction. Kant called such arguments “apagogic” and considered them unreliable because they (perhaps incorrectly) assume that the debate and subject matter is amenable to such clean logical structure.[18] In this spirit, I am not claiming to have shown, yet, that the alternatives are better, i.e. that they’re relatively likely to succeed. It may well be that every way of approaching agency (or AI safety) is fraught with difficulty.
But importantly, I don’t think the difficulty that we’re facing here is unique. Natural phenomena often seem fundamentally incomprehensible to humans before we develop the right language or paradigms to talk about them with. Consider ancient Greek philosophers struggling to understand motion. As such, we can draw on a long history of humanity overcoming such difficulties, with a variety of different methodologies. I argue that the methodologies with the strongest track record, particularly for the sorts of engineering aspirations that we have with AI safety / alignment, all involve some sort of experimental feedback, or at least strong empirical foundations.[19] This is in contrast to the methodology I’ve been describing, which has a distinctive lack of both such features. Importantly, I'm not saying both are required: I think just one can significantly compensate for the other.
Similarly, I don’t think that the apparent complexity of agency is strong evidence that we need to start with a very complex theory that aims to capture everything in one fell swoop, which I think is kind of like trying to postulate the entire ideal gas law in advance. There are possible worlds where the best approach to studying agency is a theory which demands many years of a priori development before it can receive significant empirical feedback.[20] But until we have good reason to believe we’re in one of those worlds (which I’m currently skeptical of), I argue that this characteristic should count against the theory, and against our willingness to commit to it. My hope is that we instead begin exploring alternative methodologies, hypotheses and theories more comprehensively. I aim to elaborate on what this might look like in future writing.
Conclusion
I think there are strong reasons to doubt that the field of agent foundations will achieve its aspirations without significant empirical entanglement. Hence I’m skeptical of approaches that involve reasoning our way towards very idealized theories and developing them extensively a priori. The more idealized a theory is, the more steps may be required to de-idealize it sufficiently that we can evaluate its usefulness comprehensively. This increases the risk of getting stuck on the wrong theory, and/or losing the things that made it attractive in the first place. I think that a more iterative process, in which experimental results beget idealization which then begets more experimental results, has historically been a much more common way of achieving major scientific advances. Or rather, the history of science is much better characterized by this description.
It remains to be shown how this approach can be applied to studying agency, including choosing a specific methodology that fits in that broad category, and finding theories or hypotheses that are amenable to it. This is the direction that I intend to pursue next, and I hope this post will encourage others to do the same.
This work was completed as part of the AFFINE Research Fellowship. Thanks to Haru Kim, Fernando Garcia, Nico Penttilä, Jonas Hallgren, Mateusz Baginski, Samuel Ratnam, and Hugo Lyons Keenan for useful feedback and discussions which significantly influenced my writing. Note this does not constitute an endorsement of the post. The diagrams in this post were generated with GPT-6 Astra, but closely following my own visual concepts. All text, including in the diagrams, is entirely human-written.
Actually, this regression to the mean can happen just through reflection and comparison as well. I generally recommend being somewhat wary any time a theory initially seems especially unique / ground-breaking / insightful / clarifying / promising for alignment. There's a good chance that it will later seem less of all of those things. And this is where more systematic criteria and heuristics can help us a lot, e.g. whether the theory has been useful for creating or describing anything real, or whether it is on track to do so (in a reasonable timeframe).
This is not meant as general criticism of elaborate formal frameworks. I expect them to be central to making progress here, but the difficulty is in finding ones that are fruitful.
Note that this is still a very imperfect defense, because the idealizations of EUT largely prevent it from being used to predict behavior in a wider range of situations at all. For instance, the predictions normally require a simple action and outcome space, such that we can compute expected utility, and normally avoid embedded dynamics (e.g. Newcomblike problems).
Unless we also make the empirical claim that these normative merits will be selected for. See “An aside on reflective stability” for discussion of this argument.
Notably, CPT has been shown to make better predictions than EUT, but still only in very narrow circumstances, and the experimental literature is still fairly small. Conversely, there has thus far been almost no work evaluating Infra-Bayesianism experimentally, despite it also being a generalization of EUT.
Of course, the mere fact that general relativity was capable of explaining these deviations was not sufficient to establish its value. Later experimentation demonstrated that it was not merely overfitting to a larger set of data.
My thinking here is inspired somewhat by Imre Lakatos’ conception of progressive and degenerating research programmes (Lakatos, 1978). Lakatos characterized Newtonian mechanics as the former and (late-stage) Ptolemaic astronomy as the latter, arguing that programmes had a “hard core” of principles which were integral to their identity, surrounded by a "protective belt" of auxiliary assumptions (e.g. about how things should be measured) which were more readily modified to account for anomalies. A progressive programme was then one in which modifications led to novel, successful predictions, unlike in a degenerating programme, where modifications only served to explain away more and more anomalies.
I came across this quote in Ihor Kendhiukov’s “On The Independence Axiom” (2026). I also want to credit Kendhiukov’s writing and other correspondence for significantly influencing my views on EUT.
Many portfolios didn’t fall below the provable floor, but later simulations suggest that this is because their managers had deviated from the strategy before the crash (e.g. using intuitive judgment), or because they happened to use specific initializations or parametrizations of the strategy which were more conservative and more resistant to crashes. So there was nonetheless no value floor for the strategy as a whole.
Of course, a major advance can be achieved by iterating the process and chaining together a series of more minor advances, which is essentially what I was advocating for earlier. This normally requires significant experimental phases where the minor advances are empirically validated.
How could the conditional claim be wrong? I expect many cases of a realistic agent being well-described by CDT don't involve the agent literally running the CDT algorithm internally for every decision, especially if the model only fits approximately. Hence, there may not even be a mechanism available to simply "switch to using FDT".
Notably there has been some work done on this in the ML literature outside of agent foundations, e.g. Mazeika et al. (2025), Song et al. (2025), Wang, M. (2023), and Tak et al. (2026), but this work has received very little attention in agent foundations, and strong convergence to anything is still very much an open question.
E.g. because the AI will reflect more and realize that it will "win" (achieve optimal performance) in a more diverse range of situations or decision problems if it uses FDT instead of CDT.
I think most of these are part of what we call “the scientific method”, but I don’t want to completely exclude the rest of the space yet. E.g. pure mathematical progress can be quite useful for science even when it develops completely independently of such aspirations.
One argument for being in such a world is that superintelligence might just be qualitatively different to all agents that currently exist, or can currently be created. As such, there is very little we can learn from them that might generalize, or at least, we might miss the most important properties by anchoring on empirical foundations or positive empirical feedback. Hence it’s better to consider the full space of possible agents and then use a priori reasoning to bias our search. I don’t find this very compelling on priors, because I think the premise is ~unprecedented historically. Also, this is itself an apagogic argument, against valuing empirical feedback, and in my view doesn’t constitute a good positive argument for why purely a priori work might give us better access to relevant regions of the possible-agent space.
There's a story I've heard a few times in agent foundations, that involves starting with a very idealized theory of agency, like AIXI, IB, or UDT, in which we can achieve strong, provable safety properties,[1] and gradually making it more realistic, until it can eventually be implemented, or at least used to model real agents accurately. I'll call this process "de-idealization".[2] In this case, it means creating a tractable approximation of the theory that is also embedded, self-modifying, hierarchical, etc., while hoping to keep some of what we liked about the original theory. I've recently become quite pessimistic about this story, mostly because I don't think such idealized theories tell us much about what the more realistic case will look like.
Firstly, I think it's very plausible that extensive de-idealization will result in a theory which allows only much weaker safety guarantees, if any at all. Trying to maintain specific guarantees is like trying to impose a strong constraint, in advance, on a process that is inherently unpredictable, and by default we should expect a sort of regression to the mean. Secondly, a long phase of de-idealization offers limited feedback loops (at least without significant experimental work) and hence limited opportunity to realize if we're on the wrong track. Finally, I think there's little precedent for this story succeeding in the past, even across a range of academic fields. I'll expand on each of these points below.
To be clear, I think de-idealization is important and good! What I'm concerned about is the specific way it's being done here, which I expect isn't very suitable for these theories and aims. In future writing I'll give some specific suggestions for changes to this process that I think would increase the prospects of progress and success in agent foundations.
Proof jenga
Let's start with a metaphor. Imagine you have a complete jenga tower, with full layers of blocks, and an elaborate stained glass square on top. Your task is to remove a certain number of blocks, without letting the glass fall off and shatter. The blocks are the idealized assumptions of your theory, and the glass is your safety guarantee. Maybe it's that your agent minimizes regret according to a specified utility function, or that it converges to the same utility function as another agent. The less realistic your assumptions are to start with, the more blocks you have to remove, and the more likely the tower will collapse, breaking your proof.
Of course, this is a little unfair. De-idealization is really more like replacing the blocks with different blocks, while also modifying the guarantee itself. And not every step necessarily involves weakening an assumption, but there are still typically many steps that do. So the tower can also be viewed as a Ship of Theseus, and the question then becomes whether what the eventual blocks can support is the kind of thing that we were hoping for originally. And there's a good reason to think that it won't be.
Regression to the mean
Currently, theories in agent foundations tend to look more impressive than theories in empirical AI safety, cognitive science, or even economics. They often have a very broad scope, e.g. claiming to describe a huge class of algorithms or reasoners, and yet still come with very strong formal properties. I think there are a couple of reasons for this. The theories that I'm talking about here are mostly motivated by conceptual or philosophical arguments, and although they often use powerful mathematical tools, the application of those tools does not have a strong basis in established empirical knowledge (unlike, e.g., theoretical physics). Furthermore, most of the work to develop them further, including de-idealization, is done a priori, with experiments playing only a small role. Hence, the theories have so far had limited contact with nature, and this is why they appear to describe agency with such rigor. In other words, we've taken a "measurement" of their impressiveness that is likely inaccurate, and through de-idealization, we should expect to get gradually more accurate measurements, which will reveal much less impressiveness. This could be via the scope of the theory narrowing significantly, so that it is only useful for a limited set of agents in limited conditions, or the guarantees weakening significantly, as described above.[3]
In other words, if we put our theories of agency on equal footing with other approaches to studying behavior, they might not stand out nearly as much. And one of the most insistent lessons of science is that “the first ninety-nine explanations you come up with are wrong” (Yudkowsky, 2010); in this case, the first ninety-nine attempts to improve our understanding of agency may well be failures.
The strong and modest ambitions of agent foundations
The criticisms I’ve described so far are mostly relevant to what I call the strong ambition of agent foundations: that it will eventually give us a principled account of the conditions required to achieve superintelligence-robust alignment, or even a construction for a system that fulfills these conditions. I think many in the field hold this ambition. Some work that is particularly emblematic of it involves trying to achieve optimal performance across more and more decision problems, which can sometimes look like normative reasoning about esoteric edge cases (e.g. malign priors, acausal blackmail, infinite ethics). This can lead to gradually stronger safety guarantees, but often at the cost of even greater idealization. Hence it does little to address the difficulty of finding effective ways to talk about realistic agents, which may then demand reconstruction of these decision problems and guarantees.
Of course, the strong ambition is not required to justify the value of a line of research. A more modest ambition is for agent foundations to simply make some progress in this direction, such as via red-teaming other alignment proposals, or by suggesting architectural modifications that would incrementally increase the robustness of safety measures. Indeed, agent foundations has already achieved these aims to some extent, but mostly via conceptual reasoning.[4] What remains in my view is to demonstrate that more elaborate formal frameworks add value beyond what we could already establish through conceptual reasoning. For instance, whether they predict novel failure modes in realistic systems, and whether any of those predictions are borne out. In this way the mathematical machinery can “earn its keep”.[5] This is one example of a kind of feedback that I think is lacking in agent foundations.
Feedback loops and generalization
There's no guarantee that we've stumbled upon a good theory of agency that will actually be useful for our aims. I think the motivations that are normally given for these theories are quite weak evidence of that. Indeed, many of these theories are heavily inspired by Expected Utility Theory (EUT), whose strongest justification is normative arguments about inexploitability. And empirical support for EUT is sorely lacking, e.g. it can predict human behavior only in narrow circumstances. We can try to defend EUT by viewing it as a normative standard which humans do not live up to,[6] but again, its normative merits alone provide little evidence of its relevance to realistic, generally capable agents.[7]
So what if our current theories of agency are completely wrong? That is, what if they simply fail to add to our knowledge about realistic (or even constructed) agents? There are many examples of reasonable and parsimonious theories in other domains that don’t pass this test, such as Aristotelian mechanics, which was central to discussions of motion for thousands of years, before the development of the modern scientific method revealed the inaccuracy of its empirical claims. This is one way that failure can occur. But it can also occur even with accurate predictions, such as in Ptolemaic astronomy with epicycles, which led to mathematical models that could often predict planetary motion more closely than previous methods. And yet, Ptolemaic astronomy failed to provide an explanation of the dynamics underlying planetary motion and therefore failed to generalize to less typical conditions or other physical phenomena. There’s a common thread with EUT here, which is that limited predictive success is not sufficient by itself to warrant committing to a theory, as it can sometimes result simply from overfitting on certain data. Attempting to generalize then provides further opportunity for feedback, particularly about the depth of our understanding, and to what extent it “carves nature at its joints”.
Importantly, this feedback from generalization is not immediate. It is acquired only through an iterative process, which oscillates between levels: the narrow and the general, the idealized and the realistic. In mathematical terms, generalization permits a larger class of models. An example with EUT is dropping the independence axiom, which allows for alternatives like Kahneman and Tversky's Cumulative Prospect Theory (CPT).[8] EUT can then be recovered as a special case of CPT, in which we make specific choices for probability weighting and the utility reference point. So EUT is only one of many special cases of CPT, which can be obtained by making different choices for these free parameters. Hence, the best way to get evidence about both whether CPT was a useful generalization, and whether EUT was a good starting point, is by trying to use the other, new, specialized models. If those models can, for instance, make more or better empirical predictions than the other models, this is a good sign.[9] But what if we’re unable to extract useful models from the generalization, despite our best efforts? In most cases, this will be negative evidence, e.g. if the new models make worse predictions. In some cases, it may be neutral, e.g. if it is very hard to re-specialize in other directions. But even then, we at least have some negative evidence about the tractability of the overall agenda.
Intuitively, CPT is a de-idealization of EUT. But it’s worth noting that the correspondence with generalization isn’t always so clean. For instance, continuous photon-counting models are technically generalizations, but they’re idealizations intuitively, because photons are actually quantized in reality. Nonetheless we can apply the same reasoning as above and evaluate idealizations (whether or not they are generalizations) by the eventual success of their de-idealizations. Just to drive the point home, we really don't know in advance how much theory modification this process will involve, though the significance of anomalies can be a guide. To correct for damping in an oscillator, you simply need to add one term to the equation. But correcting Newton's law of gravitation to predict the complete precession of Mercury's perihelion is much harder. These observations hence cast doubt on the form of the law itself, and led to more fundamental modifications, eventually culminating in general relativity (which nonetheless retains a strong connection to Newtonian gravity). So I argue that the subsequent success of general relativity in making novel predictions provided evidence for the value of Newton’s law in guiding our research directions.[10]
In the case of Ptolemaic astronomy, however, while epicycles were something of a de-idealization, and contributed to the predictive success of the theory, there wasn't really any way the theory could be modified further to get out something like Newtonian gravity. Nor can Ptolemy's theory now be recovered from the latter, or general relativity. This lack of such a structural connection makes it hard to view modern developments as continuing in the same research program (even despite its influence on Kepler and Copernicus).[11]
My overall claim here is that we should not treat generalization, or idealization, as intrinsically good; our generalizations should also “pay rent”, and those that don’t should be de-prioritized. I think this is part of what has made fields like theoretical physics so successful.
And without such a process, it becomes very hard to acquire strong evidence one way or another about our current theories. But I also think that we’ve seen plausible warning signs, like huge barriers to making the theories realistic, and requiring very sophisticated mathematical machinery to overcome these barriers (e.g. Infra-Bayesian Physicalism for embedded agency). Another possible red flag is needing to construct very strange mechanisms, such as updatelessness, in order to explain very ordinary behavior, such as precommitments. In other words, it often looks like we're fighting a very uphill battle when de-idealizing these theories, and if anything, that is evidence against us having stumbled upon good theories. As Scott Garrabrant has said: “The point where we discover updatelessness is the point where we are supposed to realize that all of utility theory is wrong” (Garrabrant, 2010).[12]
Historical precedent
What if we look for this story in other academic fields? Where has it panned out? Only very rarely in the natural sciences. Specifically, I think it’s a combination of features which is rare. One of them is starting with a very idealized theory, and another one is developing the theory mostly a priori. The third one is not drawing significantly on established empirical knowledge to come up with theory. I think that the strategy of idealizing into the platonic realm is most successful when we start with a firm footing in the real world. It’s not that we can’t access the platonic realm in other ways, but there is a very vast space of logical truths and mathematical models, and for a given practical problem, most of them won’t turn out to be that useful. If we have the opposite impression, I think it’s mostly because of a selection effect or survivorship bias: “You only ever hear about the cool math stuff that becomes useful later on, because that's so interesting; you don't hear about stuff that's left in the dustbin of history.” (Anonymous, 2025)
Even despite this, and the relative recency of trying to approximate idealized proofs in engineered systems, we can still find important examples of failure, which gives more cause for pessimism. These failures often involve the results of proofs failing to obtain in reality due to assumptions being insufficiently realistic, despite seeming reasonable at the time. It is also common for there to be relevant dynamics that were completely unknown in advance and therefore implicitly excluded from the models. These examples suggest that philosophical and mathematical rigor are not necessarily sufficient for getting things right “on the first critical try”, despite this being a commonly-cited motivation.
For instance, a popular past strategy for rebalancing portfolios, before Black Monday in 1987, used mathematical models to mimic put options, with provable guarantees such as a portfolio value floor. The assumptions of the model included price continuity and high liquidity, and it transpired that the real world did not sufficiently approximate these assumptions, and many portfolios fell below the provable floor.[13] Similarly, the dynamics leading to the collapse of Tacoma Narrows Bridge in 1940 were not adequately represented in the aerodynamic models that established confidence in its design.
I dislike appealing solely to intuition or a handful of examples to justify historical claims, so in the spirit of providing stronger evidence, I asked Claude and ChatGPT to look through every Nobel prize, and they judged that less than 10% of them match all of the aforementioned features, with the vast majority of examples being from economics. Excluding economics, the number is closer to a few percent. Unfortunately, this question is merely a proxy for the historical question we would most want to ask here: “What has been the success rate of deliberately trying to make major scientific advances with this methodology?” That question is impossible to answer due to the aforementioned survivorship bias. Nonetheless, if we backchain by conditioning on having achieved a major scientific advance, and ask what the most likely ways are that we got there, this methodology doesn’t seem to be one of them (on historical priors).[14] I think this is a fair reference class, because of what we’re trying to do here: find a way of developing a major new technology with strong safety guarantees. I’m agnostic here about whether this involves creating a novel architecture which is as capable as LLMs, or whether it involves understanding deep learning well enough to achieve strong safety guarantees with something like the current architecture.
But one could also argue that the reference class should be something more narrow, like computer science or economics, where these features are indeed more common. I think this is not at all obvious. On the contrary, humanity’s very conception of agency and our intuitions about it are primarily based upon biological entities that occur in the physical world, such as humans and other animals. Furthermore, even the development of modern AI itself was heavily inspired by biological neural networks. I’m not saying it’s obvious that biology is a better reference class, more so that agency encompasses a hodgepodge of things that don’t fit neatly into any single, existing scientific discipline, so the most defensible reference class is simply the entire category.
To be more specific, I see two important ways in which studying agency differs from standard computer science. Firstly, much of computer science historically has concerned objects which are explicitly specified (e.g. hand-coded), especially in cases where the field has delivered strong formal guarantees that actually obtain (even approximately) in the real world. Generally capable agents, including AI, are of course not like this, and while we might aspire to make AI more like this, we don’t yet know to what extent that is possible. Secondly, and relatedly, the systems in question have generally been much simpler than agents, e.g. their interactions with other systems / environments are more constrained, there are less moving parts, etc. So I expect that models of agents will be more chaotic and sensitive to small perturbations, which will plausibly come up a lot in de-idealization.
In my view, economics has the stronger case for relevance. Very idealized theories, a priori work, and even ex ante normative constraints, are all cornerstones of economics. And it also studies emergent complex systems (like the market). But so do most other areas of science. Hence I think we still need a positive argument if we are to lean so heavily upon economics as to use a methodology which is uniquely prevalent there. And I don’t have such an argument. Nor do I have a strong counter-argument, but I would note that finding very robust guarantees for such systems is more commonplace in the natural sciences (even when phase transitions are possible). If anything, many of the triumphs of economics have been in pointing out all the ways that systems in the real world (whether natural or artificial) differ from idealized models, without necessarily providing an achievable way of closing that gap. For all its usefulness, economics is still far from solving its own alignment problem (between humans and the market), and arguably this failure itself contributed to our desperate need to solve AI alignment now.
An aside on reflective stability
Don’t our theories of agency already make some empirical claims? For instance they claim that as AIs become more generally capable (and therefore more instrumentally rational), they will converge towards certain properties, such as "coherence" (e.g. being better described by EUT). Or, if AI is currently acting like CDT, it will start acting more and more like FDT.
These are certainly interesting empirical claims! The problem is that they could still be completely wrong,[15] and no one in agent foundations seems very interested in testing them empirically at the moment.[16] I personally would be excited about such work. It seems like AI is already getting close to being quite dangerous, so wouldn't it be reasonable to expect these trends to start appearing at least somewhat, if they're going to end up being relevant?
There’s also a version of these claims that can't really be tested empirically because they're actually more like "right before the singularity (and not any earlier), AI will finally instantiate [my favorite theory of agency], because of [philosophical arguments]".[17] Notice that you could insert basically any theory of agency here, which gives us little hope of discriminating between them confidently, even if one of the claims was true.
Proof by contradiction?
In this post I’ve tried to explain why I’m pessimistic about a common theory of change in agent foundations. To oversimplify, the argument is somewhat like a reductio ad absurdum: that subscribing to this story amounts to hoping for a rare, best-case outcome from an unpredictable process (de-idealization), with limited opportunity for course-correction. Kant called such arguments “apagogic” and considered them unreliable because they (perhaps incorrectly) assume that the debate and subject matter is amenable to such clean logical structure.[18] In this spirit, I am not claiming to have shown, yet, that the alternatives are better, i.e. that they’re relatively likely to succeed. It may well be that every way of approaching agency (or AI safety) is fraught with difficulty.
But importantly, I don’t think the difficulty that we’re facing here is unique. Natural phenomena often seem fundamentally incomprehensible to humans before we develop the right language or paradigms to talk about them with. Consider ancient Greek philosophers struggling to understand motion. As such, we can draw on a long history of humanity overcoming such difficulties, with a variety of different methodologies. I argue that the methodologies with the strongest track record, particularly for the sorts of engineering aspirations that we have with AI safety / alignment, all involve some sort of experimental feedback, or at least strong empirical foundations.[19] This is in contrast to the methodology I’ve been describing, which has a distinctive lack of both such features. Importantly, I'm not saying both are required: I think just one can significantly compensate for the other.
Similarly, I don’t think that the apparent complexity of agency is strong evidence that we need to start with a very complex theory that aims to capture everything in one fell swoop, which I think is kind of like trying to postulate the entire ideal gas law in advance. There are possible worlds where the best approach to studying agency is a theory which demands many years of a priori development before it can receive significant empirical feedback.[20] But until we have good reason to believe we’re in one of those worlds (which I’m currently skeptical of), I argue that this characteristic should count against the theory, and against our willingness to commit to it. My hope is that we instead begin exploring alternative methodologies, hypotheses and theories more comprehensively. I aim to elaborate on what this might look like in future writing.
Conclusion
I think there are strong reasons to doubt that the field of agent foundations will achieve its aspirations without significant empirical entanglement. Hence I’m skeptical of approaches that involve reasoning our way towards very idealized theories and developing them extensively a priori. The more idealized a theory is, the more steps may be required to de-idealize it sufficiently that we can evaluate its usefulness comprehensively. This increases the risk of getting stuck on the wrong theory, and/or losing the things that made it attractive in the first place. I think that a more iterative process, in which experimental results beget idealization which then begets more experimental results, has historically been a much more common way of achieving major scientific advances. Or rather, the history of science is much better characterized by this description.
It remains to be shown how this approach can be applied to studying agency, including choosing a specific methodology that fits in that broad category, and finding theories or hypotheses that are amenable to it. This is the direction that I intend to pursue next, and I hope this post will encourage others to do the same.
This work was completed as part of the AFFINE Research Fellowship. Thanks to Haru Kim, Fernando Garcia, Nico Penttilä, Jonas Hallgren, Mateusz Baginski, Samuel Ratnam, and Hugo Lyons Keenan for useful feedback and discussions which significantly influenced my writing. Note this does not constitute an endorsement of the post. The diagrams in this post were generated with GPT-6 Astra, but closely following my own visual concepts. All text, including in the diagrams, is entirely human-written.
E.g. UDT tiles pretty easily, AIXI doesn’t have an inner alignment problem (in the usual sense), etc.
Following the philosophy of science, e.g. McMullin (1985).
Actually, this regression to the mean can happen just through reflection and comparison as well. I generally recommend being somewhat wary any time a theory initially seems especially unique / ground-breaking / insightful / clarifying / promising for alignment. There's a good chance that it will later seem less of all of those things. And this is where more systematic criteria and heuristics can help us a lot, e.g. whether the theory has been useful for creating or describing anything real, or whether it is on track to do so (in a reasonable timeframe).
E.g.: identifying possible difficulties and failure modes like instrumental convergence, Vingean reflection, and the tiling problem.
This is not meant as general criticism of elaborate formal frameworks. I expect them to be central to making progress here, but the difficulty is in finding ones that are fruitful.
Note that this is still a very imperfect defense, because the idealizations of EUT largely prevent it from being used to predict behavior in a wider range of situations at all. For instance, the predictions normally require a simple action and outcome space, such that we can compute expected utility, and normally avoid embedded dynamics (e.g. Newcomblike problems).
Unless we also make the empirical claim that these normative merits will be selected for. See “An aside on reflective stability” for discussion of this argument.
Note that the space of models permitted by dropping independence is very large, and CPT encompasses only a subset.
Notably, CPT has been shown to make better predictions than EUT, but still only in very narrow circumstances, and the experimental literature is still fairly small. Conversely, there has thus far been almost no work evaluating Infra-Bayesianism experimentally, despite it also being a generalization of EUT.
Of course, the mere fact that general relativity was capable of explaining these deviations was not sufficient to establish its value. Later experimentation demonstrated that it was not merely overfitting to a larger set of data.
My thinking here is inspired somewhat by Imre Lakatos’ conception of progressive and degenerating research programmes (Lakatos, 1978). Lakatos characterized Newtonian mechanics as the former and (late-stage) Ptolemaic astronomy as the latter, arguing that programmes had a “hard core” of principles which were integral to their identity, surrounded by a "protective belt" of auxiliary assumptions (e.g. about how things should be measured) which were more readily modified to account for anomalies. A progressive programme was then one in which modifications led to novel, successful predictions, unlike in a degenerating programme, where modifications only served to explain away more and more anomalies.
I came across this quote in Ihor Kendhiukov’s “On The Independence Axiom” (2026). I also want to credit Kendhiukov’s writing and other correspondence for significantly influencing my views on EUT.
Many portfolios didn’t fall below the provable floor, but later simulations suggest that this is because their managers had deviated from the strategy before the crash (e.g. using intuitive judgment), or because they happened to use specific initializations or parametrizations of the strategy which were more conservative and more resistant to crashes. So there was nonetheless no value floor for the strategy as a whole.
Of course, a major advance can be achieved by iterating the process and chaining together a series of more minor advances, which is essentially what I was advocating for earlier. This normally requires significant experimental phases where the minor advances are empirically validated.
How could the conditional claim be wrong? I expect many cases of a realistic agent being well-described by CDT don't involve the agent literally running the CDT algorithm internally for every decision, especially if the model only fits approximately. Hence, there may not even be a mechanism available to simply "switch to using FDT".
Notably there has been some work done on this in the ML literature outside of agent foundations, e.g. Mazeika et al. (2025), Song et al. (2025), Wang, M. (2023), and Tak et al. (2026), but this work has received very little attention in agent foundations, and strong convergence to anything is still very much an open question.
E.g. because the AI will reflect more and realize that it will "win" (achieve optimal performance) in a more diverse range of situations or decision problems if it uses FDT instead of CDT.
Thanks to Fernando Garcia for directing me to this argument.
I think most of these are part of what we call “the scientific method”, but I don’t want to completely exclude the rest of the space yet. E.g. pure mathematical progress can be quite useful for science even when it develops completely independently of such aspirations.
One argument for being in such a world is that superintelligence might just be qualitatively different to all agents that currently exist, or can currently be created. As such, there is very little we can learn from them that might generalize, or at least, we might miss the most important properties by anchoring on empirical foundations or positive empirical feedback. Hence it’s better to consider the full space of possible agents and then use a priori reasoning to bias our search. I don’t find this very compelling on priors, because I think the premise is ~unprecedented historically. Also, this is itself an apagogic argument, against valuing empirical feedback, and in my view doesn’t constitute a good positive argument for why purely a priori work might give us better access to relevant regions of the possible-agent space.
Without economics, the number is more like a few percent.