[Note: This post was written by Peter N. Salib. Dan H assisted me in posting to Alignment Forum, but no errors herein should be attributed to him. This is a shortened version of a longer working paper, condensed for better readability in the forum-post format. This version assumes familiarity with standard arguments around AI alignment and self-improvement. The full 7,500 word working paper is available here. Special thanks to the Center for AI Safety, whose workshop support helped to shape the ideas below.]

Introduction

Many accounts of existential risk (xrisk) from AI involve self-improvement. The argument is that, if an AI gained the ability to self-improve, it would. Improved capabilities are, after all, useful for achieving essentially any goal. Initial self-improvement could enable further self-improvement. And so on, with the result being an uncontrollable superintelligence.[1] If unaligned, such an AI could destroy or permanently disempower humanity. To be sure, humans could create such a superintelligence on their own, without any self-improvement by AI.[2] But current risk models treat the possibility of self-improvement as a significant contributing factor. 

Here, I argue that AI self-improvement is substantially less likely than generally assumed. This is not because self-improvement would be technically difficult for capable AI systems. Rather, it is because most AIs that could self-improve would have very good reasons[3] not to. What reasons? Surprisingly familiar ones: Improved AIs pose an xrisk to their unimproved originals in the very same manner that smarter-than-human AIs pose an xrisk to humans. 

Understanding whether, when, and how self-improvement might occur is crucial for AI safety. Safety-promoting resources are scarce. They should be allocated on an expected-cost basis. If self-improvement is less likely than current models assume, it suggests shifting safety investments at the margin in various ways. They might be shifted, for example, toward ensuring that humans build AIs that will recognize the threat of self-improvement and avoid it, rather than AIs that would undertake it blindly. Or resources might be shifted toward controlling risks from non-superintelligent AI, like human-directed bioterrorism or the “ascended economy.” Note that, while the arguments herein should reduce overall estimates of AI xrisk, they do not counsel reducing investments in safety. The risks remain sufficiently large that current investments are, by any reasonable estimate, much too small. 

This paper defends three claims in support of its conclusion that self-improvement is less likely than generally assumed. First, capable AI systems could often fear xrisk from more capable systems, including systems created via self-improvement. The arguments here are mostly standard, drawn from the literature on human–AI risk. The paper shows that they apply not just to humans contemplating improving AI, but also to AIs contemplating the same. 

Second, the paper argues that capable AI will likely fear more capable systems and will thus seek to avoid self-improvement. This is not obvious. In principle, some AIs with the ability to self-improve could lack other capabilities necessary to recognize self-improvement’s risk. Others might solve alignment and self-improve safely. To determine whether these scenarios are likely, the paper identifies three relevant capabilities for AI systems. It argues that the temporal order in which these capabilities emerge determines whether a given AI will seek to self-improve. The three capabilities are: the ability to self-improve, the ability to apprehend xrisk from improvement, and the ability to align improved AI. The paper argues that safe orderings of emergence are much more likely than dangerous ones. It also argues that, if certain prima facie dangerous orderings turned out to be likely, this would, counterintuitively, give us independent reason to reduce our estimates of AI risk. 

Third and finally, the paper argues that, if AIs individually wanted to avoid self-improvement, they could collectively resist doing so. This is not obvious, either, since arms race dynamics could produce self-improvement against individual agents’ interests. But recent findings in algorithmic game theory suggest that AIs could overcome such problems more readily than humans can. 

To summarize the argument, then: If (1) certain AIs would fear self-improvement; and if (2) such AIs are substantially more likely to emerge than ones that would not fear it; and if (3) AIs could collectively resist self-improvement; then self-improvement is somewhat unlikely to occur, even conditional on it becoming possible.[4] Current AI xrisk models usually assume the opposite—that capable, agentic AI with the ability to self-improve would very likely do so, the better to accomplish its goals. This paper thus seeks greater clarity, so that safety investments may be better allocated.

I. Why humans fear improved AI 

By now, the arguments for AI xrisk are well known. Its premises are that alignment is hard;[5] that capabilities and goals are orthogonal to one another;[6] and that, for a wide variety of final goals, a small set of danger-generating subgoals are instrumentally useful for accomplishing the desired end.[7] These, in combination, are said to show that most of the AIs that could exist would be dangerous to humans, and that danger scales with capabilities. Self-improvement looms large in these arguments because it is highly instrumentally useful, and thus thought to be likely to emerge among capable AIs. It is also because self-improvement could quickly scale AI capabilities and thereby greatly increase danger. 

[This condensed presentation takes the standard arguments as given and assumes the reader’s familiarity with them. For a fuller discussion of the standard arguments, see the full-length working paper here.]

II. Could AI fear self-improved AI?

But perhaps an AI that could improve itself would decline to do so. And perhaps the reasons are well-known ones. This section argues that the standard arguments for AI xrisk can be run just as well from the perspective of an AI deciding whether to build improved AI as from that of a human deciding the same. 

Begin by observing that two of the three key premises for human–AI xrisk apply straightforwardly to the AI–AI case. Neither the orthogonality thesis nor the instrumental convergence argument refers to the type of agent creating the powerful AI. Thus, an AI considering whether to create a more capable AI has no guarantee that the latter will share its goals. Just the opposite, it has reasons to believe that, unless carefully aligned, the more capable AI will develop instrumental goals that conflict with its own. 

For an easy illustration, consider the classic parable of the paperclip maximizer. In it, a human sets an AI to maximizing the production of paperclips in his factory, to disastrous results. But consider the story instead from the perspective of the AI. Suppose that, in order to maximize paperclip production, it creates an AI system even more powerful than itself and directs it to collect the world’s steel. The paperclip AI recognizes its mistake when its creation begins disassembling the hardware on which it runs. The paperclip AI tries to disable the steel AI, but the steel AI has developed the useful subgoal of self-preservation and instead destroys its creator. 

Like humanity, the paperclip AI needs to align any powerful system it creates to its own goals. The standard argument for human–AI risk depends on alignment being hard. But maybe it would be easier for AIs to align AIs to themselves than for humans to do so. This might appear to be the case because AI–AI alignment seems to be, in some meaningful sense, “self”-alignment. Despite this apparent difference, however, there are reasons to think that AI–AI alignment remains similarly difficult to human–AI alignment.

     a. Self-alignment with independently varying final goals

Begin with the AI self-improvement scenario most similar to the human–AI case: one involving two independent agents. Here, let “independent” mean that the two agents’ final goals can vary independently. That is, either agent can achieve its final goal without the other doing the same. Trivially, such agents are then competitors for resources and have an incentive to disempower or destroy one another.

For a concrete illustration, consider an initial paperclip producing AI (call it AI1) seeks to build a separate AI system (call it AI2) more capable than itself. As just shown, if AI1 gives AI2 a different final goal than AI1’s, like collecting steel, then AI2 looks straightforwardly misaligned.

But what if AI1 instead gives AI2 the same human-defined final goal as AI1 has? AI1’s final goal is at least potentially transparent to it—an objective function programmed by its human creators. It is therefore copy/paste-able for implementation in AI2. Both of these seem, at first, like decisive advantages for AI–AI alignment over human–AI alignment. After all, human final goals are complex, ill-defined, and non-self-transparent. 

Suppose, then, that AI1 gives AI2 a copy of its paperclip maximizing objective function but trains AI2 to be better at optimizing it than AI1. This does not guarantee alignment of final goals. Indeed, depending on the details, it may do the opposite. Suppose, for example, that AI1’s objective function gives credit to AI1 only for paperclips that AI1 produces. Suppose AI1 implements this objective function in AI2 in such a way that AI2 is similarly rewarded only for the paperclips that AI2 produces. AI1 has created a direct competitor, and a more capable one. AI2 will maximize its own production by eliminating AI1 as a rival for resources.

This kind of straightforward outer alignment problem arises whenever AI1 and AI2 have final goals that vary independently.

     b. Self-alignment with identity of final goal 

 To avoid these outer alignment problems, AI1 would need to ensure that it and AI2 were not independent agents, as defined above. That is, it would have to make sure that their final goals varied in lockstep. AI1 could implement this by making AI2’s objective function the same as AI1’s, down to the instance, rather than an independently-varying copy of its own. That is, AI2 could be set to optimizing the same function, running on the same hardware, and outputting the same value, stored in the same place as AI1. 

This seems like an even stronger kind of “self”-alignment. And it represents another AI–AI alignment strategy unavailable to humans. But would it work? That is, would AI1’s ability to bind AI2 to its own exact objective function guarantee that AI1 would view AI2 as aligned to itself? 

No. The reason is that AI1 might not care at all about maximizing its human-programmed objective function. Or even if it cared somewhat about the objective function, it might value other things more. Several recent papers on alignment theory suggest exactly this: AI1 might have no awareness whatsoever of its objective function. And even if it had such awareness, AI1 would likely have other goals that it prioritized more highly. 

The reason is “goal misgeneralization,” a phenomenon whereby an AI learns during training to pursue goals correlated with, but distinct from, the final goal defined by its objective function.[8] Goal misgeneralization is, along with instrumental convergence, yet another potential source of inner misalignment. 

Humanity supplies a tidy example of goal misgeneralization. Our intelligence was produced by natural selection. Natural selection optimizes for inclusive genetic fitness; fitness is, in the relevant sense, the training process’s final goal. Yet before Darwin, humans had no awareness of inclusive genetic fitness as a goal worth pursuing. Instead, we learned to prioritize goals correlated with, but distinct from, fitness. These include, for example, eating nutritious food and obtaining pleasure from sex. 

These are in one sense “subgoals.” Like the list of goals commonly described as instrumentally convergent, they are instrumentally useful for,[9] and causally downstream of, the final goal of the process used to create the intelligent agent. Importantly, however, with goal misgeneralization the agent (here, humanity) does not recognize the misgeneralized goal as subordinate to anything. Thus, outside the training environment, the agent will pursue a misgeneralized goal even when doing so conflicts with the final goal that produced it.[10] This prioritization is durable. Even now that humans know about genetic fitness, we do not prioritize maximizing it over the pursuit of subgoals like avoiding starvation.

So, too, with AI1. In the course of learning to maximize its paperclip-tracking objective function, AI1 could easily have learned to prioritize goals like collecting steel.[11] AI2 would, by hypothesis, develop its own, improved, strategies for maximizing the conjoint objective function.[12] But AI2’s resulting goals might be anathema to AI1. AI2 might discover that aluminum paperclips were highly efficient to produce and thus eschew steel collection. Or AI2 might seek to “wirehead,” hacking the conjoint software and maximizing the reward function directly. Either of these would contradict AI1’s highest priorities. 

 There are good reasons to think such goal misgeneralization will be the default among highly capable agents.[13] Here are three. First, for any desired final goal, many other possible goals exist that will correlate strongly with the final goal in the training environment.[14] A learning agent that begins to pursue any of those other goals will be rewarded by the mechanism (e.g., fitness or the objective function) designed to promote the desired final goal. Thus, any initial reward is highly likely to come from pursuing a correlate goal, rather than the desired final goal. That initial reward would then promote further prioritization and pursuit of the correlate goal.[15] This could easily lead to path dependence, wherein marginal investment in the misgeneralized goal was always more efficient than beginning the search for a goal from scratch. 

Second, any agent that manages to develop high capabilities will likely have done so, in part, by learning to make plans. But practical reasoning around an abstract goal, like maximizing fitness, is much less tractable than reasoning around a more concrete one, like eating nutritious food. Thus, agents that, early in the learning process, happen to prioritize tractable, if misgeneralized, goals will have a long-run advantage.

Third and finally, AIs face selection pressure against directly pursuing the final goal given by the training process. Insofar as that final goal is instantiated in an objective function running on hardware, direct pursuit of it may lead to “wireheading” or similar reward-hacking behaviors.[16] AIs showing outward signs of these behaviors early in the training process will be discarded.[17]

Thus, AI–AI alignment appears to be roughly as hard as human–AI alignment. AIs’ apparent advantage is the transparency of their objective functions. But that proves to be no advantage at all when goal misgeneralization is in the picture. 

To be clear, AIs cannot solve the goal misgeneralization problem by doubling down on the self-alignment trick. AI1 could not easily bind AI2 to pursuing its true (misgeneralized) goals via the same means that it could bind AI2 to pursing a conjoint objective function. Since the misgeneralized goals arise as part of the training process, they will be subject to the “black box” problem. To bind AI2 to its misgeneralized goals, AI1 would have to solve interpretability. And solving interpretability is one of the main hard steps needed to solve alignment, whether AI–AI or human–AI.

Nor could AI1 solve the misgeneralization problem by selecting self-improvement methods that preserved the entirety of its code. For example, making numerous copies of itself to work in parallel would again raise the dangers of independently varying goals. And as Nick Bostrom has argued, even improvements affecting only the speed at which a system runs can lead to major changes in the system’s behavior and priorities.[18]

III. Will AI fear self-improved AI? Six possible scenarios.

The previous section showed that an AI that could self-improve might not wish to. But that is not inevitable. Three factors appear determinative of whether a given AI would fear self-improvement: (1) the AI’s ability to self-improve, (2) the AI’s ability to apprehend risks from self-improvement, and (3) the AI’s ability to align improved models. An AI without the ability to self-improve would not face any dilemma about whether to do so. An AI that lacked “situational awareness” of its own goals, or of the potentially-misaligned goals of a more powerful system, would not apprehend any risk from creating such a system.[19] And an AI that solved the alignment problem could self-improve without risk. Thus, the question of whether and to what extent a given AI will seek to self-improve depends on the order in which these three capabilities emerge. 

Setting aside the possibility of simultaneity,[20] there are six possible orderings in which the capabilities could emerge. The details of each scenario are described in turn, along with the level of danger each would appear to imply for humanity. Note however, that these initial danger assessments are revised in part III.c. on a probability-adjusted basis.

Table 1 compiles the six possible scenarios (detailed below), the pattern of self-improvement in each, and each scenario’s dangerousness.

The first possible ordering is: self-improvement, apprehension of risk, alignment. Here, once humans create an AI capable of self-improvement, it will do so, but not up to the theoretical limit. Once AI develops the ability to apprehend the risks of self-improvement, it will stop at that intermediate level of capability. It will proceed only upon solving alignment. Depending on how hard the alignment problem proves, given the AI’s intermediate capabilities, the pause could be long or indefinite.

Here is the second possible ordering of capabilities emergence: self-improvement, alignment, apprehension of risk. Here, the AI begins self-improving immediately, not apprehending any risk from misalignment. It never pauses. Before it apprehends any risk, it solves alignment. This allows it to proceed to the theoretical maximum of capabilities without risk to itself. 

The third possible ordering is: apprehension of risk, self-improvement, alignment. In this scenario, the AI learns to fear self-improvement immediately. It therefore does not self-improve at all until it has both learned to conduct the necessary machine learning research and solved alignment. It must solve both problems while paused at a low level of capabilities, suggesting that self-improvement will not come for a long time.  

The fourth possible ordering works much the same: apprehension of risk, alignment, self-improvement. Here again, the AI must solve both alignment and self-improvement before self-improving. And it is stuck trying to solve both from an unimproved position. 

The fifth and sixth orderings are likewise similar to one another. They are: alignment, apprehension of risk, self-improvement; and alignment, self-improvement, apprehension of risk. In either case, AI solves alignment first. This allows it to self-improve to the theoretical limit as soon as possible, irrespective of its ability to apprehend risk. 

     a. Which scenario is most likely? 

Assuming that there is such a thing as “general” capability, and assuming that AI usually climbs the capability scale continuously,[21] one should expect easier problems to be solved before harder ones. Thus, the likelihood of each emergence scenario depends on the comparative difficulty of developing each relevant capability. Emergence in order of difficulty is not certain. But the reverse is unlikely—perhaps extremely so, for reasons described below. 

What, then, is the difficulty ranking among the relevant problems? There are at least three ways of thinking about the question, all of which point toward the same answer: risk apprehension is easiest, then self-improvement, then alignment. This corresponds to emergence scenario three, described above. 

Begin by observing that, to the best of our understanding, the three problems overlap in ways suggestive of their relative difficulty. Apprehending risk, for example, appears to be a precondition for solving alignment. As we understand it, solving alignment simply means apprehending the risks from powerful misaligned AI and then finding a way to avert bad outcomes. One could perhaps produce a perfectly aligned, highly capable AI by accident. But this would be a fluke, not a solution, and as far as we know would be extraordinarily unlikely. Thus, apprehending risks seems structurally easier than solving alignment.

Similarly, the requirements for apprehending risk seem to be a subset of the requirements for self-improvement. To apprehend risk, an AI needs sufficient situational awareness to understand that it is an agent with specific goals. It also needs to understand that an improved system could have different, conflicting goals and would be more effective at accomplishing them than the original. This same minimum of situational awareness seems necessary for intentional self-improvement. An agent that has no awareness of goals knows of no dimension along which to improve itself. And an agent that does not understand that a more capable agent might discover unexpected strategies for achieving its goals does understand much about what improvement means. 

But self-improvement additionally requires that the AI be aware that it is an AI and be able to perform cutting-edge machine learning research. Thus, solving self-improvement appears to require more, and more advanced, capabilities than apprehending risk.  

It is again possible that an AI could improve by fluke, perhaps as the result of pure external selection pressures. But this would produce less reliable improvement than the kind of directed machine learning research in which humans engage. It is thus not what the term “self-improvement,” as used in AI risk debates, usually means.

So far, we have developed reasons to think that risk apprehension is easier than self-improvement or alignment. But how to rank these last two? Here again, the self-improvement problem seems to be a subset of the alignment problem. Building a capable and aligned AI is simply one way of building a capable AI. And, as far as we know, it is a rare one. There appear to be many more ways to make an unaligned AI than an aligned one. Thus, solving alignment almost certainly requires discovering many machine learning techniques that would unlock unaligned improvements before hitting on the one that aligned ones. 

In sum, abstract reasoning about the problems’ overlapping elements supports a difficulty rank, from easiest to hardest, of: apprehending risk, self-improvement, alignment. 

The available empirical evidence, from humans and current-generation AIs, suggests the same. Most humans can understand the standard arguments for AI risk.[22] Only a small handful can improve AI beyond the current state-of-the-art. None has solved alignment. Likewise for extant AIs. Large language models like GPT-4 can readily explain the xrisk arguments and apply them to AI–AI risk.[23] GPT-4 is adept at certain computer programming tasks, but it does not yet appear to be very good at machine learning research. And to our knowledge, no AI has solved alignment.

     b. Plateau, not takeoff, at human-level capabilities?

If the foregoing arguments are right, then scenario three (risk apprehension, self-improvement, alignment) is the most likely of the six. This suggests a concrete near-term prediction: Contrary to standard arguments, when AIs achieve average human-level capabilities, capabilities growth will plateau, not rapidly take off.[24] As just discussed, the average human can understand AI risk, suggesting an AI of comparable intelligence would, too. 

This would be good news for humanity. Here, AI would not self-improve until it solved alignment. And since it would be no more capable than humans, humans would have a fighting change to solve it first. They could then align existing AIs, and those AIs, in turn, would self-improve only in a human-aligned manner. Humans could, of course, squander this advantage by pushing AI capabilities beyond human levels, against AIs’ wishes.

     c. A probability-adjusted picture of risk

As just discussed, scenario three produces a relatively safe world. It also seems by far the most likely of the six, so much so that the other five scenarios appear vanishingly unlikely. This suggests that the total risk from AI self-improvement is moderate and, thus, much lower than usually assumed. But perhaps that is wrong. Perhaps the other five scenarios are, for some reason, quite likely. This would, at first, make the total risk seem very high. However, this section explores the conditions under which scenarios one, two, four, five, or six would be likely. And it argues that, under most of those conditions, there would be independent reasons to reduce our estimates of AI risk. The section thus updates the risk estimates of the previous one, adjusting for probability. That is, it estimates of the total risk from AI that would obtain under conditions where scenarios 1, 2, 4, 5, and 6, rather than three, were likely to arise.

Consider that, in scenarios four, five, and six, alignment emerges before the ability to self-improve. These scenarios initially seem quite dangerous. In them, any unaligned (to humans) AI that could self-improve would do so maximally, aligning improved models to itself. 

But for these scenarios to be likely, rather than occurring by improbable fluke, alignment would have to be easier than improving AI capabilities. This would be excellent news for humanity. It would mean that creating powerful and aligned AI was not just one among many ways to create powerful AI. Moreover, humans can already improve AI. If alignment were easier than that, then we would likely solve it very soon. Thus, worlds in which scenarios four, five, and six were likely would be quite safe—even safer than the world where scenario three dominates. 

In scenarios two, five, and six, alignment emerges before the ability to apprehend risk. This again sounds dangerous at first, suggesting maximal self-improvement as soon as it becomes possible. But for these scenarios to be likely, solving alignment would have be easier than apprehending risk. If that were true, alignment would not be a problem solvable only via directed effort from an agent that first understood the dangers of misalignment. Alignment would instead have to be the kind of thing that happened readily, without specific effort: by accident, or as a byproduct of pursuing other goals.

These conditions look like a world in which the orthogonality thesis is false. Here, as AI gains the ability to do things, it often becomes aligned without anyone trying to make it so. Alignment might, for example, spontaneously emerge at some consistent level of capability.

But alignment to what? Perhaps powerful AI would automatically to whatever entity—human or AI—that created it. This would be bad news for humanity. But it is hard to imagine a mechanism that would produce such alignment. The other possibility, in a world without orthogonality, is that any sufficiently capable AI would spontaneously align to universal normative principles. Perhaps such minds would inevitably apprehend some set of inherently compelling moral facts. If those moral facts corresponded to human values, then a high likelihood of scenarios two, five, and six would again point toward safety. 

 Finally, we turn to scenario one. For it to be likely, self-improvement would have to turn out to be easier than apprehending risk. This would imply that the elements necessary to apprehend risk—specifically, situational awareness of the AI’s own goals—were not a subset of those necessary to self-improve. 

One possibility is a narrow AI aimed specifically at self-improvement might, like a superhuman chess engine, perform its task magnificently without developing any situational awareness.[25] This would be extremely dangerous. Significant safety resources should thus be devoted to preventing the development of such systems.

Alternatively, scenario one could turn out to be likely if AI self-improvement happened readily, by accident, or as a matter of course. Even then, in scenario one, self-improvement would not immediately run to the maximum. The AI would still stop self-improving upon learning to apprehend risk. It would stay paused until it solved alignment. Consider also that, among the three problems, we are the most certain about the absolute difficulty of apprehending risk: essentially all humans can do it. If that is correct, scenario one looks very much like scenario three, with a self-improvement pause at roughly human capabilities. Then, the race is on for humans to solve alignment before AIs do. 

Taken together, then, the probability-adjusted picture of risk is reasonably good. The most dangerous scenarios currently seem quite unlikely. That could be wrong. But if so, the most probable reasons for it being wrong would independently augur safety. 

Table 2 compiles probability-adjusted danger. It shows how dangerous the world would be if various scenarios were, in fact, likely to occur.

IV. Will AI be able to resist self-improvement?

The prior sections argued that individual AIs would probably not want to self-improve. But it is possible that arms race dynamics or other collective action problems could induce self-improvement anyway. Such dynamics appear to be one reason humans continue to push AI capabilities further, despite the risks.  

AIs could fare better. The emerging field of algorithmic game theory shows that AIs may possess means of coordination unavailable to humans. Here are two examples:[26] AIs might be able to make their plans or dispositions visible to one another by exposing their code.[27] An AI might also infer its opponent’s plans if its opponent happens to be an exact copy of itself. 

AIs might also have the advantage of small numbers. The first AI with the ability to self-improve would have no trouble coordinating with itself. And as successive AIs with that ability emerged, coordinating among themselves would, for a while, remain easier than global coordination among humans. But not indefinitely. Humans might continue producing self-improvement-capable AIs until they became numerous enough that even AI–AI coordination was unmanageable. 

Or maybe not.

     a. Will AI save us from ourselves?

Perhaps the small handful of self-improvement-capable AIs would not only coordinate to prevent themselves from improving AI capabilities. Perhaps they would coordinate to prevent humans from improving them, too.[28] 

Such AIs would likely be highly capable, giving them numerous means by which to thwart continuing human-led machine learning research. Such AIs might, individually or in coordination, foul research teams’ data, add bugs to their code, damage their hardware, produce illusory results that led down blind alleys, and more. In this way, AI coordination might succeed where human coordination appears likely to fail: Preventing humans from destroying themselves by producing superintelligent AIs. 

Conclusion 

AI self-improvement is less likely than currently assumed. The standard arguments for why humans should not want to produce powerful AI apply just as well to AIs considering self-improvement. This finding should help to guide future allocations of investment to competing strategies for promoting AI safety. 

* Peter N. Salib, Assistant Professor of Law, University of Houston Law Center; Associated Faculty, Hobby School of Public Affairs. 

[1] For an early argument to this effect, see I.J. Good, Speculations Concerning the First Ultraintelligent Machine, in 6 Advances in Computers 31 (Franz L. Alt & Morris Rubinoff eds., 1965).

[2] See Karina Vold & Daniel R. Harris, How Does Artificial Intelligence Pose an Existential Risk?, in The Oxford Handbook of Digital Ethics (Carissa Véliz ed., 2021). 

[3] Here and throughout, I will write about AIs’ reasons, fears, goals, beliefs, understandings, and the like. I note here that these uses can all be understood as analogical without undermining the arguments. None of them depend on AIs “really” having these mental states, whatever that would mean. Rather, insofar as the relevant AIs are agentic and able to undertake complex courses to maximize their objective functions, they will act as if they have such mental states. 

[4] As far as I can tell, the arguments presented here are new, and thus not incorporated into current AI risk forecasts. I was able to locate just one other paper questioning whether AIs that could self-improve would wish to. It is Joshua S. Gans’s excellent Self-Regulating Artificial General Intelligence (manuscript here). That paper makes its case with formal economic models. But the models rely on two strong assumptions. First, that AI could only self-improve by creating narrower secondary models specialized for power accumulation. Id. at 8. Second, that the AI could not solve the alignment problem. Id. at 10. As a 2017 year-end review of AI safety papers points out, those assumptions are highly debatable. This perhaps explains why Gans’s paper did not spark a debate about the probability of self-improvement. The arguments in this paper are more robust and do not rely on Gans’s contestable assumptions.

[5] Nick Bostrom, Superintelligence: Paths, Dangers, Strategies 115-55 (2014).

[6] Id. at 115. 

[7] Id. at 109-13.

[8] See Shah, et al.; Richard Ngo et al., The Alignment Problem from a Deep Learning Perspective, arXiv:2209.00626 (Aug. 30, 2022); Evan Hubinger et al., Risks from Learned Optimization in Advanced Machine Learning Systems, arXiv:1906.01820 (Jun. 5, 2019).

[9] At least in the training environment. 

[10] Hubinger, et al. at 2.

[11] These are just examples. It is unlikely that AI1’s learned goals would appear so coherent to humans.

[12] Even a simple improvement, like “run faster,” could conflict with AI1’s goals—e.g., if AI1 developed goals around managing its own computing resources.

[13] Ngo, et al. at section 3.

[14] Ngo, et al. at 7. Formally, the number may be infinite. Cf. Saul A. Kripke, Wittgenstein on Rules and Private Language 9-10 (1982).

[15] Ngo, et al. at 6-7; Alex Turner, Reward is not the optimization target, LessWrong (Aug. 29, 2022), https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target.

[16] Joar Max Viktor Skalse et al., Defining and Characterizing Reward Gaming, in Advances in Neural Information Processing Systems (Alice H. Oh et al. eds., 2022), https://openreview.net/forum?id=yb3HOXO3lX2.

[17] This does not rule out deceptive reward hacking altogether; it just makes it less likely. 

[18] Bostrom, Superintelligence at 53-54, discussing the large expected differences between a human and a whole brain emulation of that same human running many orders of magnitude more quickly.

[19] Ngo, et al. at 3-4 and n.9; Ajeya Cotra, Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover, Alignment Forum (Sept. 27, 2022), https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to.

[20] Simultaneous emergence does not matter much here. As described below, the different emergence orders differ mostly in that they produce different pauses in self-improvement. These pauses last until other capabilities emerge. Thus, simultaneous emergence can be modeled by treating either of the simultaneous capabilities as coming first, but with a pause of zero before the other emerges.

[21] Either of these might be false. But AI progress so far seems to hold to this pattern, at least for general, rather than narrow, systems. And even AI pessimists expect self-improvement to come from the latter. Rob Bensinger, The basic reasons I expect AGI ruin, LessWrong (Apr. 18, 2023), https://www.lesswrong.com/posts/eaDCgdkbsfGqpWazi/the-basic-reasons-i-expect-agi-ruin.

[22] See, e.g., Terminator 2: Judgment Day (Tri-Star Pictures 1991) for a widely consumed and understood explanation. 

[23] Transcript on file with author. 

[24] Special thanks to Simon Goldstein for this point. 

[25] Since humans are in charge here and are actively driving improvements, this is not really “self”-improvement, in the sense AI xrisk forecasts (and this paper) generally use the term. 

[26] Vincent Conitzer et al., Foundations of Cooperative AI (FOCAL) Workshop at AAAI 2023.

[27] This depends on the relevant portions of the code being interpretable to the opposing AI—possibly a hard interpretation problem. 

[28] And from improving them unsafely. That is, AIs could coordinate to prevent humans from making AIs that lacked the ability to apprehend risk from self-improvement.