1) I think the problem with this position is that it strips specific individuals of their agency, downplaying the role of individual people in history and giving priority to some kind of collective intelligence that governs countries.
But in reality, I think that’s simply not how things work. The decision not to destroy the USSR, just like the decision to drop nuclear bombs on Hiroshima and Nagasaki, was made by a VERY small group of people, not by “the US as a country.” Change the specific decisions made by specific individuals, and history could have turned out the complete opposite.
2) The same goes for this case. Suppose the US president were someone like Yudkowsky, and the General Secretary of the Chinese communist party were someone like Nate Soares. Imagine the creators of our simulation simply decided to see what would happen and swapped Trump’s and Xi’s brains for those of Yudkowsky and Soares. What kind of Moloch would stop them from striking a deal?
3) Let’s imagine that tomorrow, Xi reads AI 2027 and becomes convinced that AI is going to kill us all. Just one person in the world comes to believe this. And he wakes up one morning thinking that he has to stop AI and that this is his purpose in life, just as the head of SoftBank has declared that his cosmic purpose is to create ASI.
What happens next? Well, he could publicly announce that China will withdraw from the AI race. He could call Trump every day and beg him to stop. He could even fall to his knees before him, if necessary. He could, I don’t know, offer to pay the US compensation amounting to a percentage of China’s GDP in exchange for sparing humanity. He could voluntarily agree to let the US control all of China’s computing resources.
What, specifically, would stop Xi, if he became as convinced as Yudkowsky that stopping the race to ASI is his life’s purpose, from substantially improving the odds of international control and a halt to the race?
I am not arguing against anything, i'm more just explaining a view of the world! Yes, I agree that it's a structuralist view and that human agency matters a great great deal! I do believe human agency is improved when it comes from a place of realism.
I think its worthwhile pointing out there's a difference between chemical weapons and nuclear weapons. Or treaties on biological weapons and aligned-ASI. And the reason for this is as I outlined in the post. Furthermore, I also wanted to illustrate that I personally view the world in a way that the treaties o... (read more)
I'm not sure if it's game theoretically possible for a nation state to give up a decisive strategic edge and hence I am not confident the development of ASI can realistically ever be stopped as I believe governments will attempt to develop aligned-ASI in secret if they commercially ban it.
I have been listening to Nate Soares interviews and he will often point to international cooperation on nuclear weapons. But to me nuclear weapons treaties look like failures of international cooperation. We still have enough nuclear weapons on earth to destroy basically all of modern civilization. We built enough to destroy all of earth, and then we said: "let's not build more than we need to destroy all of earth" and then we patted ourselves on the back "oh boy, good that we stopped building more of those nuclear bombs, imagine if we destroyed all of civilization twice over!". And by the way China is still building nuclear weapons because they rightly assess they can't destroy as much of the earth as Russia and the USA can and they want to reach parity on destroying all of modern civilization.
There have been instances of internationally cooperating to forego some strategic edge, such as chemical weapons. However, I do not think chemical weapons ever were a decisive strategic edge in the way that nuclear weapons are, or that aligned ASI would be.
The reason for states never being able to give up nukes, aligned-ASI, or any other decisive strategic edge I believe to be simple.
Per realist dictate:
1. Nation states live in an anarchic system without a central authority
2. Within this anarchic system they continuously face existential threats from other states
3. To optimally survive in such a world, you have to compete for power. Which is a function of economic wealth (w/ input technology) and population size.
The logic of realism argues the USA is existentially threatened by states such as China or Russia. In other words, the USA and China both hold a knife on each others throat. With both of them constantly armwrestling trying to get a decisive edge such that they can finally wrestle the knife off their throat and gain a decisive edge in the knife fight.
Therefore, the USA is in a bind. It has to secure a decisive strategic edge somehow to existentially secure itself in this anarchic system. Technology in history has often, if not always, offered this decisive strategic edge, And because other states are a proven existential threat, whereas the threat of ASI is unproven, the USA and China may feel compelled to seriously attempt to build aligned-ASI to gain a decisive strategic edge.