I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.
I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.
There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.
The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.
Per realist dictate:
1. Nation states live in an anarchic system without a central authority
2. Within this anarchic system they continuously face existential threats from other states
3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.
The logic of realism argues the USA is existentially threatened by states such as China or Russia. In other words, the USA and China both hold a knife on each others throat. With both of them constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight.
Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.
I think you have unipolar era intuitions, and I think the world has changed. I think few people had such intuitions in between 1900-1945, or even between 1945 - 1991.
Imagine you lived during the time period where the USA developed nuclear weapons but the soviet union did not have any, and then the soviet union built nuclear weapons and you now have live in the nuclear arms race potential armageddon era.
It's pretty clear that if you were living in the cold war a big part of you would have been wondering "why the f' did we not annihilate the soviet union and then threatened to annihilate everyone that even dared to think about nuclear weapons so that we could maintain unipolar nuclear hegemony and usher in an era of perpetual peace. Now the USA and the USSR are going to blow up the whole world!". I bet von neumann was thinking that. In my view that would have been a really really valid argument to make.
Things have 'worked out' (maybe, we don't really know). But did they work out because we made the right decisions, or got lucky?
Just think about what it means for China to be encircled by the USA with missiles aimed at them and with their navy contained, as it currently is. Do you think the Chinese think the USA has good intentions? "We merely have a gun aimed at your head because we love you?". I think they think "We must become so powerful, the USA can not threaten us anymore".
The intuition I am trying to communicate here is that we are in a room with each other where everyone has guns aimed at each other while we are building this ASI. And that's the world we have to navigate.
I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.
I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.
There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.
The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.
Per realist dictate:
1. Nation states live in an anarchic system without a central authority
2. Within this anarchic system they continuously face existential threats from other states
3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.
The logic of realism argues the USA is existentially threatened by states such as China or Russia. In other words, the USA and China both hold a knife on each others throat. With both of them constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight.
Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.