I've been kind of moved by Richard Ngo's argument (e.g. here and in this thread) that the seeming naturalness of unipolarity might be a semi-ideological artifact of centralized solutions being easier to imagine. I'd be curious to hear you say how you think Bertrand Russell's similar infamous 1950 prediction of either a global war or a world government (or population collapse) by 2000 went wrong, and what the key dissimilarity to your prediction of unipolarity is.
Tangentially answering to the more general point from Richard Ngo: I think the long-term arc of history in fact points towards unipolarity. In another, related Richard Ngo post, 1a3orn comments;
I generally think rationalists gesture at [multipolar equilibria not being stable long-term] as inevitable more than they actually demonstrate that it is inevitable; i.e., long term multi-polar equilibria has been quite sticky for Westphalian states or for plankton.
I don't know much about plankton, but I think it's notable that at the time of the Peace of Westphalia, there were about 300 sovereign states within the Holy Roman Empire, but by now they consolidated to only ~7 sovereign states on the same territory, almost all being members of the EU and NATO.[1] I don't think historical trends are that favorable to multipolarity!
- ^
Austria is not in NATO, and Liechtenstein is neither EU nor NATO, but realistically both are close allies of the larger block around them.
Didn't know about the plankton thing either and I asked Claude, it's actually pretty interesting.
On the general historical point, it's something that I've been wondering about for a long time - for me, it's on the Pareto frontier of how important of a question it is vs how little explicit study it has received. I think it's more thorny than it might first appear, because while the "maximal size/power" of entities increases (companies and economies of scale, nations and the consolidation process that you mention), there is also some vague intuition that technology and the information age has given some power back to flexible, smaller entities like startups or the individual. (see e.g. this old Claude chat).
As of now, I feel relatively undecided and want to think more about it.
I think Russell's prediction was not that bad. I don't know what probability we should translate to his statement that "unless something quite unforeseen occurs". 90% maybe. And what was the real probability that we would have gotten a nuclear war or effectively a world government by 2000? I think about 50%?
I think Russell in 1951, seeing the ever-increasing pace of technological development during his life since 1872, underestimated the possibility that the world would soon enter the Great Stagnation, and the pace of technological progress would stop increasing and would in many ways slow down. In the 1951 essay we are talking about, he writes:
"The first possibility, the extinction of the human race, is not to be expected in the next world war, unless that war is postponed for a longer time than now seems probable. But if the next world war is indecisive, or if the victors are unwise, and if organized states survive it, a period of feverish technical development may be expected to follow its conclusion."
I don't know how much his prediction of feverish technical development depends on this happening after a war, but I think he would be surprised that in 2000 hydrogen bombs were still the most powerful weapons, and that the equilibrium of both sides having nukes and ICBMs but no one managing to develop reliable ICBM interceptors remained stable for so long.
I think AI will likely end the Great Stagnation and bring us back to ever-accelerating feverish technical progress, and Russell's prediction will come true.
I don't know what probability we should translate to his statement that "unless something quite unforeseen occurs". 90% maybe
I assume you mean 10% on "something unforeseen"? Opus 5 estimates "5% and possibly less", for what it's worth.
I think he would be surprised that in 2000 hydrogen bombs were still the most powerful weapons
To be fair, nuclear bombs in 2000 were a thousand times more powerful than in 1951.
I think you're underselling how much of Russell's concern was the apparent instability of the equilibrium at the time, even without further technological progress:
"If things are allowed to drift, it is obvious that the bickering between Russia and the Western democracies will continue until Russia has a considerable store of atomic bombs, and that when that time comes there will be an atomic war."
It doesn't seem like this is conditional on further technological progress. The passage you're quoting is specifically about human extinction, not about world war.
Yeah, I agree he was pretty wrong. But I think that a big part of the story is that he saw at the time politicians on both sides talking about nuclear war as a winnable fight (he quotes politicians on this extensively in his 1958 speech on the topic), and this made him think that one side would eventually pull the trigger. In reality, the concept of mutually assured destruction and nuclear wars being unwinnable eventually became common wisdom among politicians, and then nuclear sabre-rattling went down. And I think that's downstream of the fact that military tech in fact didn't improve that much, and no one yet managed to build a reliable response to nukes + ICBMs.
So you think the Great Stagnation manifested that quickly (it's generally taken to start in the 70s), and in military tech to boot? (to develop better ICBM interception). That feels a bit implausible to me since nuclear bombs became so much more powerful. It feels more like an offense-defense balance intrinsic to the technology + coordination of exactly the type that he didn't expect (the ABM treaty capping defense systems). But I feel a bit out of my depth here.
What the world did have in 2000 was a "unipolar moment" in which one nation, the United States, was a global hegemon without a serious ideological or economic rival. If not a formal world government, it was a world order in which that one nation defined and policed the world system.
Come on, that's obviously not what Russell meant. His prediction was clearly extremely flawed. This is consensus - Opus 5 agrees with me.
I would love to see a world where a wise party controls the stable unipolarity and governs the world through a Long Reflection before we continue scaling to ASI.
I think "eternal stagnation" happens if the dominant party cares more about maintaining power than making the world better, which seems likely if they're current world or lab leaders. I would trust a reflective AI over those people to govern us through a Long Reflection.
Random thoughts on what makes pausing tricky:
- The leading actor might not want to pause.
- The leading actor might not want to pause, except to maintain their own power. They might actively want power and not care that much about the rest of humanity. In the post, I think you assume that our the balance of our future institutions will be relatively similar to how it is today, with nation-states making decisions. Something that could go wrong is if labs become military powers actively racing to the top.
- The leading actor might want to pause, but think the risks of pausing outweigh the gains. For example, the unipolarity might be "stable" from the perspective of most people, but internally unstable in a way that encourages racing. One party (e.g. the USG) might have stable power, but internal instability within the party (e.g. people within USG vying for power) might incentivize the leader of the party to urge a race ahead, because they don't trust that their goals will be instantiated by someone else within the party.
- The leading actor might not be able to pause.
- E.g. if AI development is unstoppable at that point (we've handed off R&D and have to hope for the best).
I'm rereading what I wrote here, and I think it's a bit confused, but I'm not sure why I feel this way.
Thanks for a great post! In your preferred version of Plan A, what do we do during the pause to raise the probability of scaling to superintelligence safely?
I see the situation roughly like this. AI capable of fully automating the US military (including military R&D and command & control) is capable enough to autonomously enforce the Deal if we give it the affordances it needs to do so. My guess is that in the AI Futures capability ontology, you need TED AI—or maybe slightly weaker AI—to fully automate the US military. In Plan A, we reach TED AI in 2035 and then pause for at least five years, during which time we do tons of automated alignment research, exhaustively study our TED AIs to confirm they're in the basin of good deference, have different TED AIs adversarially audit one another for signs of misalignment, and so on. My guess is that if we spend our five years of pausing at TED AI well, we can have high confidence by the end of the pause that our AIs want to follow human instructions and are not scheming to disempower us.
Now, I take you to be saying that even in this situation, we would not have solved all the problems we need to solve to scale safely all the way to the limits of intelligence. For instance, we wouldn't have solved the problem of ASI inventing an alien ontology or becoming so good at persuasion that the concept of instruction following breaks down. These seem like real problems to me. But it's not clear to me that we could solve them with a longer pause at TED AI. They seem like fundamental problems with aligning an incomprehensibly greater intelligence to a human intelligence.
I should probably engage more deeply with the basin of good reference post. But I currently thank 5-10 years is not nearly enough that "we" can confirm that the AI is in the basin of good deference. Sure, we can maybe prove with high confidence that the AI is not actively scheming against us, and maybe even the voters will be justified in believing the narrow claim from the scientists that the AI is not actively scheming. But otherwise, I very much sympathize with Richard Ngo's point that there is no way for most normal people to understand and consent to this process so quickly. 5-10 is like two election cycles! It's really not very much time! And I'm not just talking about the random voters, I feel that I myself will be very distrustful of the Jupiter-brained creature we are creating for whom the concept of instruction following breaks down.
I don't have great ideas what to do instead, but I think probably gradualism is the best approach. The one type of alignment that people have experience with is raising children. The one type of value evolution people feel relatively fine with is the new generation of people gradually shaping culture in their own way.
So let's just have some kids first who can grow up in a more abundant society, built by the limited involvement of TED AIs. Then we have guided humanity through the times of peril, and solved our own small portion of the alignment problem by lovingly raising a generation of somewhat smarter and better-adjusted people, who can then carry on the torch. Solving the rest of the alignment problem is their job, not ours.
Tentatively, I think that after one or two generations, people should start using a moderate amount of human genetic engineering to shape the next generation to be even smarter, wiser and happier. Then at some point, probably one generation should start merging with the machines and becoming cyborgs. I don't know what happens after that, it's their choice. And throughout this process, I hope that there always remain some people (perhaps not very many, probably keeping only a small sliver of the stars to themselves), who wish not to progress to the next level of development, and wish to remain in their only more-or-less enhance human form, but who will still love the people who progress one step further in the next generation, and who will still be loved by them in turn.
It sounds like your high level plan is to make artificial intelligence smart enough to stabilize the situation and no smarter. Then do relatively chill human intelligence augmentation & cyborgism until eventually the augmented humans/cyborgs are ready to build superintelligence.
I don't think this is a crazy plan. It might be better than Plan A. I guess my main worries would be
- Exogneous xrisks that TED AIs can't manage down to zero get us while we're doing chill intelligence augmentation.
- The TED AIs somehow drift into misalignment.
- The limits of machine intelligence prove to be so far from the limits of augmented human intelligence that we're never ready to bulid superintelligence, and maybe we miss out on some unique benefits that superintelligence would have provided.
I think exogneoue xrisk is probably very small, especially that I'm in favor of starting some chill levels od space colonization (e.g. Mars) pretty soon.
It's not impossible but I would find it very surprising if your TED AI collective was unwise or untrustrworthy enough that it could drift into misalignment over time without stopping itself, but trustworthy and wise enough that we could have handed off the intelligrnt explosion to them, and they would have safely got us to the limits of intelligence.
Yep, the third point is a serious objection, but I feel that if our descendants never feel ready to buikd superintelligence, then maybe they just shouldn't. I think it's plausible that TED AI plus maybe narrow superintelligences (e.g. specialized in space-probe design) are enough for a near maximally good future.
My impression was that one of the reason for continuing the advance in AI capabilities following that level of handoff, is that it would no longer be up to us whether continued AI capabilities growth happenedm and it would no longer be up to us whether we survived. Either we already succeeded in aligning AGI well enough by then to trust it to manage the ascent to ASI (or be wise enough to refuse to build it), or we already doomed ourselves.
I don't think that's true, and I tried to argue in the Pausing after unipolarity section that different level of alignment and wisdom is needed for the two goals.
Imagine that individual AI instances are about as smart, thoughtful and benevolent as the average capabilities researcher at an AI company right now. I think setting up a large army of these instances to maintain the peace might work pretty well without them fully taking over the world. With some clever checks-and-balances mechanisms alluded to in this post, I think this collective of moderately well-meaning AIs could remember their oaths to remain loyal to human democracy and function well. But I'm much more scared of tasking the collective of these AI instances to decide if it's safe to scale further and then solve the alignment of the next generations well.
So I think there is a very wide gap between how wise and aligned the AIs need to be to trust them to do a peace-keeping operation with a narrow scope vs to trust them to build the superintelligence whose values will determine what the galaxies get filled with.
"inject it into everyone"
I assume the resistence would be lesser if it didn't involve syringes, and I doubt this is an unsolvable hard constraint, even though the scare factor seems load-bearing in your model.
Option is pervasive, ambient, environmental nanobot protection (together with 222nm light etc)
For real stability, the AI needs to be powerful enough that if one side tries to pull out of the deal and disable the AI surveillance on their side, they can’t. So the major powers need to trust the AIs enough to give them enough military power that the humans can no longer fight off the AIs, and the AIs are tasked with enforcing by force of arms that no one builds superintelligence or other terrible weapons and none of the major powers invade each other, at least until some condition is fulfilled
Plan A supposes mutually assured compute destruction, pldeging assets much like proof of stake, MAD all the way yo. (When thinking about it, feudal hostage exchange and marriage-based pacts are a kind of MAD as well).
I'm not sure what role the weaponized interdependence trend plays here.
overall, great post. Just noticed some apparent weak points and want to invite you to refine.
This post makes a good point, but this ask might be dominated by asking for a pause today
Asking for a pause now is simpler to explain, not as weird, and much better for safety.
The main trade-off is that this is more expensive for the US, but asking for a pause directly makes it clearer that super intelligence is a threat to humanity.
As I tried to stress throughout the post, I agree that pausing sooner is the better option. But I think people should have contingency plans for pausing at least after unipolarity, instead of just scaling to the limits of intelligence as some currently existing plans suggest.
I think it’s likely that the world will enter military unipolarity within our lifetime.[1]
I think the creation of such unipolarity is an alarming prospect, but at least once it happens, there will be less of an excuse to continue the race towards superintelligence. I think it’s important that we shape our actions and advocacy in such a way that at the very latest when such unipolarity comes to exist, we stop AI development for a long time.[2]
The arrival of unipolarity
Why do I believe that it’s likely we will get to unipolarity?
Well, what is the alternative? If unipolarity is never achieved, that means we forever have hostile great powers, maintaining large armies, pointing missiles at each other, and developing better and better AIs. There is never a big enough gap in AI capabilities for any side to develop a decisive strategic advantage, or everyone decides again and again not to use their advantage to disarm the opponent. Multiple groups go to space, still pointing weapons at each other; they develop literally Jupiter-brained superintelligences, but the balance of power never breaks and there always remain competitive, hostile militaries.
I’m not saying this never-ending cold war is impossible, but it feels unlikely to me. Some kind of unipolar outcome seems much more stable. I think plausible outcomes are:
I’m less confident that it will be within our lifetime that stable unipolarity emerges, but given the pace of AI progress, I expect that so many things will happen in the next 50 years[3] that at some point the current unstable situation will end, and we will enter a stable unipolar regime.
I feel that people often feel squeamish talking about wars or all-powerful treaty-enforcement systems, even if those are the logical outcomes of their beliefs. But I think it’s important to explicitly think through these things, so I will go through Plans S, A, B, C, D developed by the AI Futures Project, and I will write about when unipolarity is achieved in the plan and how I believe people should act after that.[4]
Plan S
Under this plan, countries completely stop further AI development, possibly also dismantling the compute supply chain to make it harder to restart.
I believe that stable unipolarity is still likely to eventually emerge under Plan S. Either the pause breaks and the world switches to another Plan discussed later; or normal human wars create a stable hegemony; or hopefully nations generally become friendlier with one another over time, to the point where a great power war becomes as unimaginable as a France-Germany war would be today.
I think trying to pause dangerous AI development and then steering the world towards peaceful coexistence and wiser decision-making over the longer term is a worthy goal. But given that it either changes into one of the other plans or only leads to unipolarity over a longer timeframe, I will not discuss it further in this particular post.
Plan A
Under this plan, the US and China and other important countries cooperate to slow down AI development and steer it in a safer direction. I think that such cooperation is not super likely, but it’s plausible enough to be worth thinking about, and valuable enough to be worth pushing for.
I will use the particular story presented in AI 2040 as a springboard for analyzing the emergence of unipolarity under this Plan A, but I think my conclusions still hold even if things don’t go exactly as they do in the story.
In the story, military unipolarity is not established until very late in the scenario - for a long time, countries don’t dare to give up their militaries and everyone still has missiles pointed at everyone else. So far, this makes sense. However, at the end of the story, it looks like unipolarity is still achieved, but the story glosses over the details.
At the end of the story, countries hand off power to their aligned AIs, and they are implementing a global space governance scheme together. It doesn’t look like anyone is worried about the Russian president going crazy and nuking the rest of the world, but it’s unclear how this threat has been defused.
Did everyone agree to disarm or hand over power to mutually trusted AIs? I hope that most countries will choose this peaceful approach, and I appreciate that Plan A explicitly mentions compensating nuclear powers for their cooperation, and that Russia also gets a share of the pie. But it feels too optimistic to assume that all nine nuclear powers go along with the plan. And even though AIs’ bioweapon capabilities are tightly controlled under Plan A, it seems likely that the control won’t be perfect. With all the new science and industry coming online in the story, eventually small nations will also be able to build terrifying weapons of mass destruction they might use to defend their sovereignty or try to extract concessions.
So what is Plan A’s plan for making sure no one retains the ability to launch nukes or other terrible weapons?
The two options I see are defense and military intervention.
Maybe your own defensive AIs can develop flawless ICBM interceptors, and protect you against bioweapons and other threats. I’m pretty skeptical that people will accept this as a good outcome. I once asked a very prominent AI safety researcher how he imagines superintelligences defending against bioweapons, and his first response was “well, maybe you need to live in bunkers for a while, but robots can build very big and comfortable bunkers”. When I objected that voters wouldn’t be happy with that, his second proposal was to inject nanobots into everyone, so they can defend us from all illnesses. I think the voters won’t be happy with the mandatory nanobot injections either!
I think that when it is explained to the American people that they will need to let superintelligences develop humanly incomprehensible nanotechnology and inject it into everyone, because that’s the only way to defend against Eritrea’s bioweapon program, then the voters will start asking why we can’t prevent Eritrea from building bioweapons instead.
So I expect that the more likely outcome is that the Consortium of countries developing AI together in Plan A will choose a preventive military intervention so no one can pose a threat to the new world. Once there is a “country of geniuses in a datacenter” it can’t be that hard to use an army of AI geniuses to surveil everything happening in Eritrea and make sure they don’t build any terrible weapons.
Plan A assumes that some privacy-preserving AI-surveillance technologies will exist; hopefully those will be the ones used. Also, keeping with the cooperative spirit of Plan A, hopefully all countries will have a choice to voluntarily agree to this surveillance and weapon non-proliferation regime, and the AIs will guarantee that the sovereignty of every country entering the deal will be respected.[5]
Countries that already have some terrible weapons (most notably, nukes) would be offered a chance to join the deal and dismantle their nuclear arsenal in return for preserving their sovereignty and maybe getting a somewhat larger share of resources than similarly sized countries without potent militaries. If they don’t agree to the deal, then their weapons need to be disabled or defended against: maybe bee-sized drones flying in and cutting the wires of all the nukes at once?
In any case, it really looks like the likely outcome of Plan A is the establishment of stable military unipolarity.
But then my question, which I will discuss in a later section, is: why scale to superintelligence, why not pause at unipolarity?
Plan B and C
I agree with the AI Futures Project team that Plan B and Plan C, where the US and China race near full-speed ahead (while trying to slow each other down with cyberattacks and bombing in Plan B), are less desirable than the coordinated slow-down of Plan A.
The AI Futures Project’s scenario for both Plan B and C ends in a war between the US and China.[6][7] I agree with this assessment. How else could they end? As I said in the previous section, I don’t find it very plausible that hostile countries could race to the limits of intelligence without any side ever getting an overwhelming advantage which they can use to disarm the other.
I don’t like wars, and establishing global hegemony through wars is a scary prospect even if we get lucky and the wars turn out relatively bloodless. But AIFP gives 40% to Plan B or C happening (plus another 30% for the even worse Plan D), so it seems important to plan for them.
It’s possible that the war destroys both sides’ ability to progress with AI development for a long time. It’s also possible that both sides survive the war, but both learn that they need to stop further AI development for a long time under the threat of mutually assured destruction. Plan S through bloodier means.
The other option is that one side wins the war. Then they can have the ability to disarm their opponents and make sure (quite possibly through AI surveillance, as discussed above) that no one else can build powerful weapons and superintelligences.[8]
Hopefully, the victor will be magnanimous in its victory and allow the other nations of the world to retain their sovereignty, in the way Japan was allowed to remain sovereign after World War II.
But from the perspective of the future as a whole, I think the most important question is: if the world chooses to go to war instead of a treaty, and one side establishes military unipolarity through the war, why scale to superintelligence, why not pause at unipolarity?
Pausing after unipolarity
What level of AI capabilities is needed to establish unipolarity?
If all major nuclear powers are cooperative, as in an optimistic version of Plan A, it really might not be that high. If a major nuclear power defects, and the AIs need to disable their arsenal with bee-sized drones or something, that’s harder, but I still feel like “country of geniuses in a datacenter” is probably enough.
Now what level of alignment do we need to safely use AIs to establish stable unipolarity?
That might be a pretty big lift.
In Plan B and C, if human leaders just want to use the AIs to win a war, then in theory, the AI can just focus on technical and scientific work in a lab, building bee-sized drones and whatever weapons are needed, without getting too much autonomy in the outside world. But in practice, capabilities are correlated, and the AIs that can build weapons that can easily defeat other major nuclear powers will probably have very deep situational awareness and general autonomous capabilities. And after the war is won, it’s likely that the AIs will need to be given a lot of affordances to successfully monitor everything in every country to make sure that no one is secretly building actual superintelligence or various terrible weapons. We will need pretty reliable alignment at that level of capabilities and affordances.
In Plan A, the level of alignment might need to be even higher if people want to really ensure that the established new world order is actually stable.
For real stability, the AI needs to be powerful enough that if one side tries to pull out of the deal and disable the AI surveillance on their side, they can’t. So the major powers need to trust the AIs enough to give them enough military power that the humans can no longer fight off the AIs, and the AIs are tasked with enforcing by force of arms that no one builds superintelligence or other terrible weapons and none of the major powers invade each other, at least until some condition is fulfilled.[9]
I think this level of hand-off requires a lot of trust and shouldn’t be done lightly. But still, the AIs’ objective (maintain the peace and prevent smarter AIs from being built) is pretty narrow and not very philosophically complicated. I think if we had mind uploading technology, there are people I would trust that if we created a billion copies of them, they could maintain these basic rules while not overstepping their authority and not interfering with the other decision-processes of humanity.
It seems plausible to me that with a few years of lead-time and experimentation which Plan A grants us - without any very fundamental breakthroughs - we could build human-level LLMs that are similarly trustworthy to uploaded humans, and whose “country of geniuses in a datacenter” collective can be trusted with the relatively narrow goal of maintaining the peace and preventing new superintelligences from being built. This might be doable even with the smaller lead-time of Plan B and C, or even in the rush of Plan D if we are really lucky.
(For more on this, see my older post.)
I’m not saying that safely handing off our militaries to a country of geniuses in a datacenter is easy, but scaling to an aligned superintelligence seems so much harder than that.
If I understand the ending of AI 2040 correctly, Plan A wants to scale up to the limits of intelligence in the 2040s, using fully automated AI R&D on the gargantuan data centers floating in the oceans and then quite possibly build literally Jupiter-brained superintelligences as we go out to space.
As they explain, this requires having AIs that are aligned and wise enough that they can build successors who are also aligned and wise enough, that they can build successors… until the limits of intelligence. During this process the AIs might go through several ontology shifts incomprehensible to humans, and might develop such powerful persuasion capabilities and such precise models of the human mind that the concept of an AI giving impartial advice to humans kind of breaks down.[10]
This seems so much harder and more dangerous than just creating an AI swarm that is barely capable and aligned enough to stabilize the world, maintain the peace and make sure that no other superintelligence gets built for a while.
And as discussed earlier, I don’t think that scaling to superintelligence really makes the odious aspects of “stabilizing the world” that much better. They still need to create an effectively AI-led world government. Maybe the superintelligence can make the disarmament of non-coalition nations a little less bloody and the surveillance a little more reliably privacy-preserving, but I don’t know how big the difference is compared to what the country of geniuses in the datacenter can do.
Conclusion
I find the prospect of creating an AI-powered stable military unipolarity alarming, even if the AI doesn’t directly take over. If possible, I would prefer nations to agree to pause AI development, and let the world develop in its normal trajectory. We have made a lot of progress in our people, countries and international order being better and wiser since 1926, even if the last handful of years sometimes feel kind of like a regression. This makes me more optimistic about continuing the slow march of human progress than trying to rush to a finish line within our lifetime.
But if the mutual pause doesn’t come to pass, and AI develops to the point where one country or coalition can overpower the rest, I hope that the outcome won't be worse than it needs to be.
I hope the nations will first try to build the broadest coalition possible, offering the fairest terms to any holdout nation.
I hope that if a war needs to come, it will be fought by the least highly superintelligent AIs that can still win at an acceptable cost.[11]
And, crucially, I hope that after the war is over, the victors take a long pause and stop needlessly rushing towards higher levels of superintelligence.
There will probably be a lot of temptation to just scale further uninterrupted. The people building and controlling the AIs will have a lot of wealth and power, and they will be people who enjoyed seeing the number go up so far, and would enjoy seeing it go up even more. They might have a lot of credit in the eyes of the public too: they just managed to build powerful AIs that, contrary to the warnings of naysayers, remained loyal, and managed to win the most important war. The AIs themselves will likely be influential advisors at that point, and it’s likely they will also advocate for scaling further.[12]
I think it’s important that we resist this temptation, and at the latest once the geopolitical race is no longer pressing, we stop scaling for a long time, and we keep the role of the smartest AIs in society to a relatively limited, peace-keeping role.
To be clear, probably we should eventually scale further and create smarter AIs so we can reach the stars with them.[13] There is some danger that the world under stable military unipolarity becomes stagnant, and people would never decide to go further. So it seems useful to set some deadlines in advance, setting out some rules like doing the next further scaling after 100 years, and then sending out the first colonizing ship in 200 years by default, unless 80% of the population votes against it when the deadline comes.
But I think that rushing too much to build a superintelligence whose thoughts are incomprehensibly beyond us and for whom the very notion of corrigibility breaks down is a bigger risk than taking the chance with falling into eternal stagnation through a too-long pause. I think going through the Long Self-Correction properly will take plenty of time before we can feel comfortable fully handing power off to such incomprehensible creatures, and we shouldn’t plan to be ready within the natural lifetime of this generation.
More on the exact operationalization of what this means later.
As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.
Whenever I write about what "we" should do, that should be understood as the readers, the AI safety community, or humanity as a whole. None of his should be understood to be about European policy.
And quite possibly much earlier
I’m using AIFP’s scenarios as starting points, because I think they are pretty well thought-through and the five plans carve out the possibility-space relatively well. I also think that the AI Futures people are much less squeamish about the possibility of war and the emergence of military unipolarity than some other writers, and I applaud them for this. However, I feel even they are still not explicit enough about the implications of their vision, especially in Plan A, so I wanted to write more about that.
Except there should probably be some universal exit rights enforced for the population of every country, and the robots should build some nice cities for the refugees somewhere where everyone is fine to let them in. (Worst case, some nice domes in Antarctica.) And maybe a few small dictators can be removed if everyone in the UN Security Council agrees.
The Plan C story, aka AI 2027 slowdown ending, ends with the US AI tricking China into a false deal where the Chinese AI eventually betrays the Chinese government and orchestrates a bloodless coup to install a pro-US government. I think this should basically count as a type of war, and I expect the real ending of following the Plan C race will realistically be a more obvious war than the galaxy-brained trickery depicted in the AI 2027 story.
At the end of Plan B, they also say that an alternative to war is handing off power to the AI. But they don’t say what happens after the hand-off. I think if there is no treaty, then the hand-off still results in war, either between the countries or the AIs that took them over.
There are also various intermediate scenarios, where one side wins the war but doesn’t have the stomach to create a permanent global hegemony, and then maybe the race continues and a new war might happen later. Or there is a localized war between the US and China, but Russia keeps its nuclear arsenal, and that will need to be dealt with later.
E.g. people can start building superintelligence again or disable the current AI surveillance army if all major powers agree.
Also, maybe scary threat-dynamics start happening on the acausal trade front if a sufficiently intelligent agent thinks too hard about certain topics. I think there can be various unknown dangers lurking in the shadows as we are scaling to the limits of intelligence.
Preferably, the lead-time would be used to give these AIs a relatively safe architecture too - for example, I’m partial to the literal “country of geniuses in a datacenter” idea of having a lot of not particularly super-human instances run in parallel.
I have some hopes that the AIs will be trained to be wise and well-calibrated advisors, and they will only recommend further scaling if that’s indeed a good idea. But I’m worried that by default, they will just parrot the prejudices of their scaling-pilled creators.
Though this is not entirely obvious to me. I can imagine that a country of geniuses in a datacenter can eventually build basically every technology we need for a good interstellar future.