I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.
I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.
There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.
The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.
Per realist dictate:
1. Nation states live in an anarchic system without a central authority
2. Within this anarchic system they continuously face existential threats from other states
3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.
The logic of realism argues the USA is existentially threatened by states such as China or Russia. In other words, the USA and China both hold a knife on each others throat. With both of them constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight.
Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.
It only looks like the compassionate action because we live in the world where it so far has worked out.
It's also not clear or obvious to these states that building ASI is certain death and what the risk calculations are. There's an important assymetry in information here vis a vis the risk calculations with respect to attempt to build aligned-ASI or not, versus dieing as a state due to the anarchic nature of the international system.
The Chinese regime knows it can die and have their country be plunged into centuries of humiliation due to other statesas this has happened in the past. The risk of ASI are merely speculative high-minded theoretical arguments, whereas the benefits to e.g., miltech of AI are already clear. This means that on the margins, any state will locally want to keep developing AI to improve their miltech at the very minimum, and never at any point would they want to stop making these local incremental improvements. Where the optimization gradient points is very clear.
The task here, of convincing states based on theoretical arguments that this theoretical thing is more dangerous than the person in the room with a knife on your throat is not an easy one.
I can make predictions based on this worldview as well:
I personally strongly believe that China will not listen to the USA and rein in their AI capabilities if the USA wants to stop; at least not until they have surpassed the USA in capabilities themselves. There is in my worldview, absolutely zero odds of an international treaty for AI-safety until China has reached equal or greater capabilities in AI as the USA.
Furthermore, I believe it is fundamentally impossible for the USA to rein in its own AI capabilities: it will never do so. Which means a serious international treaty will never happen. Although for a while many people will be trying, and even many documents will be signed.
I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.
I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.
There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.
The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.
Per realist dictate:
1. Nation states live in an anarchic system without a central authority
2. Within this anarchic system they continuously face existential threats from other states
3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.
The logic of realism argues the USA is existentially threatened by states such as China or Russia. In other words, the USA and China both hold a knife on each others throat. With both of them constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight.
Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.