This is a linkpost for https://open.substack.com/pub/stevenadler/p/contain-and-verify-the-endgame-of?r=4qacg&utm_medium=ios
My take on this: AGI has existed since 2022 (ChatGPT). There are now multiple companies in America and in China which have AGI-level agents. (It would be good to have a list of countries which are next in line to develop indigenous AGI capability.) Given that we are already in a state of coexistence between human and AGI, the next major transition is ASI, and that means, if not necessarily the end of humanity, the end of human control over human affairs. This most likely means that the dominant intelligence in the world is entirely nonhuman AI; it could also refer to some human-AI symbiosis, but again, it won't be natural humanity in charge.
A world in which the development of AGI is allowed, let alone a world in which there is a race to create ever more powerful AI, is a world which by default is headed towards ASI and the dethronement of natural humanity. That is our world already, even if our tech and government leadership manage to not see it that way.
So the most consequential thing that humans can care about right now is the transition to ASI. You can try to stop it from ever happening. or you can try to shape it in advance. Once it happens, it's over, humans per se have no further say, if they still exist they are at the mercy of the transhuman or posthuman agents now in charge.
Now let me try to analyze your essay from this point of view. Your essay is meant as a critique of the paradigm according to which there is a race between America and China to create powerful AI, as if all that either side needs to care about is getting there first. Your message is that even if America gets to powerful AI (or safe powerful AI) first, the possibility of China (and in fact anyone else) developing the same capability would still remain. I see two main suggestions: a "mutual containment" treaty, in which both countries place bounds on their development of AI along with means of verifying that the bounds are being obeyed; and spreading around defensive measures, which make it harder for powerful AI to impose itself on the world.
My take is that mutual containment really means a mutual commitment to stop the creation of ASI, a commitment which to be meaningful ultimately needs to be followed by everyone on Earth. It is a coherent position, but it's an uphill struggle since current trends are all in the other direction. On the other hand, I regard defense against ASI as impossible. Possibly there are meaningful defensive measures against lesser forms of AI, but ASI's relationship to human intelligence is like that of the best computer chess programs to the best human chess players - the latter simply have no chance in such a game.
On the other hand, the only truly safe form of powerful AI is ASI governed by a value system which, if placed in complete control of the Earth, would still be something we could live with, or even something that is good for us. Anything less, e.g. a legal order in which powerful AI exists but there is a ban on further development, is unstable against further development to the ASI level.
So there is a sense in which development of safe powerful AI really is all that matters, because it really has to mean safe ASI, and that is not something which will stay behind borders. If America, China, or any other country achieves ASI, that is success for everyone. But it does also imply the loss of sovereignty for natural humanity, in favor of the hypothetical benevolent superintelligent agent(s).
Really appreciate the thorough engagement -
My take is that mutual containment really means a mutual commitment to stop the creation of ASI, a commitment which to be meaningful ultimately needs to be followed by everyone on Earth.
I infer that you believe I disagree with this?
My aim is something like 1) if folks are going to race with China, at least recognize how hard it’ll be and what it would actually take to end the competition (don’t miscalculate things as wins that aren’t) and 2) help people notice that the factors that’d be necessary for “winning” would also support cooperation to head off the race, and so it’s more possible than they might think. (Or, winning in the race framework is even hard than they think.)
ASI governed by a value system which, if placed in complete control of the Earth, would still be something we could live with,
That's exactly my point. However, once the value system is defined, it will either lock mankind in or be corrigible. The former case contains options like my take where the AI only provides everyone with access to education and enforces only universally agreed political opinions or[1] the situations where the AI builds the Deep Utopia or governs the world, criminalising social parasitism in the whole world. The race to ASI might also be motivated with the belief that it will empower the creators' state or enforce the state's ethos.
The first example of the ASI's role, unlike the latter two, might be considered not as a lock-in, but as another system where mankind can govern itself.
I don’t think China is willing to accept yielding. I can’t think of any reason that they would.
This is totally a shower thought, and I don’t trust it, but what about a strategy of semi-cooperation? China has been contributing to open source models. Those models have been keeping up with, and catching up to the capabilities of the closed source models.
I wonder if mutual containment could come through having similar capabilities as we both learn from each other’s research. Then neither side has a gross advantage. Maybe it doesn’t have to be zero-sum.
.
Yeah I think an alternative to “mutual yield” would be “only develop powerful AI as an international collaboration.” So, yield in terms of building it unilaterally, but not necessarily on the whole
Semi-cooperation is one way for both sides to learn from each other—but so is poor infosec or even outright espionage. If both countries are leaking or spying enough, that might create a kind of uneasy balance (and transparency), even without formal agreements. It’s not exactly stable, but it could prevent either side from gaining a decisive lead.
In fact, sufficiently bad infosec might even make certain forms of cooperation and mutual verification easier. For instance, if both countries are considering setting up trusted data centers to make verifiable claims about AGI development, the fact that espionage already permeates much of the AI supply chain could paradoxically lower the bar for trust. In a world where perfect secrecy is already compromised, agreeing to “good enough” transparency might become more feasible.
I think state relations is only a dimension to the problem.
There are actual power players who invest and steer the development. They have their international relations. State administrations may be imposing certain pressures, motivating players to aggregate by national identity. In a way, state administrations become players themselves.
So, one should think in the terms of entire ecosystem, from state representatives to individual researchers. US vs China mentality only empowers a state-connected subset of players at the expense of the rest of them.
Some competitions have a clear win condition: In a race, be the first to cross a finish line.
The US-China AI competition isn’t like this. It’s not enough to be the first to get a powerful AI system.
So, what is necessary for a good outcome from the US-China AI competition?
I thought about this all the time as a researcher on OpenAI’s AGI Readiness team: If the US races to develop powerful AI before China - and even succeeds at doing so safely - what happens next? The endgame is still pretty complicated, even if we’ve “won” the race by getting to AGI1first.
I suggest two reframes on the US-China AI race:
By “containment,” I mean that a good outcome for the US might require stopping China from ever reaching a certain level of AI capability. It isn’t enough for the US to get there first. For instance, it’s an issue if China builds AI that can disrupt US nuclear command and control even a small percentage of the time. This is true even if the US has a system that can more reliably disrupt theirs. There are some types of AI the US wants for China never to develop - and likewise, that China wants the US never to develop: the interest in containment is mutual.
By “verifiably yielding,” I mean that the US must be confident that China is not continuing to try to build powerful AI. Otherwise, China might eventually surpass US systems or incur other risks, like losing control over their AI system in the rush to catch up. Unfortunately, methods for “verifiable non-development” - confirming that another party isn’t building AGI - are very understudied. We need to invest heavily in developing these methods and creating treaties that can enforce them: Otherwise, even if we “win” the race to certain powerful abilities, we won’t have good ways to confirm that China has given up on pursuing AGI. (These methods can also be useful for slowing or avoiding the race ahead-of-time, if countries can verify that the other is not developing AGI.)
Given how high the stakes are perceived to be, getting China to yield might require the US to take a truly dominant lead. Such a dominant lead is far from assured, even if the US believes it could ultimately outrace China.
Both nations would benefit from lowering the stakes of the competition - like “hardening” the world so it’s less vulnerable to the risks of powerful AI, and cooperating on international safety standards.
~~~~
Continues here;
Twitter post: https://x.com/sjgadler/status/1925372613721038910
Thank you to Justis Mills of LessWrong’s feedback service, among others in the Acknowledgements