> Yes. And I think it's important to understand and think about this. And: killing everyone by accident is against ones' self-interest. How much risk of doom do you think nations would accept, for how chance of dominating the future/avoiding being dominated? 10%? 20%? More? The nuclear example suggests that we'll accept a very high risk, particularly if we wander into it.
I think they would take on near infinite risk, sadly. In my view way to look at the anarchic situation we find ourselves in and the USA/China relationship is that either of them has a knife on each others throat. Because both of them have a knife on each others throat, they probably don't kill each other. But both of them are constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight. This is in my view just the nature of anarchic systems. I believe people whom largely lived through the unipolar era don't realize how much the world has changed with the rise of China.
This was the situation pre-nuclear weapons as well, and so the nukes got built. Once the nukes got built there could be some or a lot of coordination around nuclear matters. But the hand was forced in some sense. You have a knife on your throat from your opponent, you can wrestle this knife out of his hands by building nukes. So you build the nukes. Then in reality the other person also build nukes a while later because you ignored John Von Neumanns advice to just obliterate your opponent with your decisive strategic edge. And what happened here is that both sides stopped holding a knife on each others throat, and instead they placed one big bomb over each others heads that either can trigger at will. This feels more comfortable. It's very comfortable not to have a knife on your throat (in the analogy this means we now have less conventional war), but the existential risk for either party is the same or greater.
I mean this knife analogy quite exactly. China is surrounded by tens of thousands of missiles aimed directly at them. This is the view from China (AI-generated infographic):
And the USA understands that if China breaks out of the 1st and 2nd island chain, the knife of China will move closer and closer to the neck of the USA.
> Another related thought: Nuclear disarmament failed, and is a failure of coordination. But nuclear nonproliferation largely succeeded. So if slowdown and nonproliferation is all we need for success, we can get that even with strong governmental self-interest.
I believe this succeeded because the great powers already had the decisive strategic advantage that they needed vis a vis the rest of the world. It makes sense for them to now coordinate to prevent proliferation in the rest of the world. And because they are the great powers, and hence have all the power, they largely succeeded at this with a variety of carrots and sticks.
I'm really pointing here at the dynamics between the great powers, not the great powers and the rest of the world. Rest of the world is largely irrelevant for this analysis in my view.
In my view of the world both China and the USA are basically forced to attempt to build aligned-ASI. If one of them succeeds first, they would use this power to then prevent other ASI's anywhere else from being built.
> Here's my main hope: Just like the US and Russia cooperated to prevent proliferation of nuclear weapons, even at the height of their mutual enemnity and distrust, the US and China cooperate to go slowly enough for a good chance at aligned ASI, and to cooperate to prevent others' from going faster.
I think this and everything else you say after is wishful thinking. I think the USA and China have no choice but to race with each other. I do think we are structurally doomed here. If one of them reaches aligned-ASI first, they would use this immediately to disarm the entire rest of the world. If both of them reach aligned-ASI around the same time, they would be incentivized to coordinate on not blowing each other up with it. But the race to aligned-ASI seems structurally impossible to stop from my point of view.
I think we have to overcome the deep structure of this anarchic reality with human ingenuity, empathy and wisdom to a degree we've never succeeded at before. It's not hopeless, where there's life there's hope, but the odds are slim.
I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.
I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.
There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.
The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.
Per realist dictate:
1. Nation states live in an anarchic system without a central authority
2. Within this anarchic system they continuously face existential threats from other states
3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.
The logic of realism argues the USA is existentially threatened by states such as China or Russia. In other words, the USA and China both hold a knife on each others throat. With both of them constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight.
Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.