If superintelligence comes to Earth, I would prefer it be controlled by democratic governments than by autocratic ones. I find Dario's arguments to fear a CCP-controlled superintelligence to be compelling. However, I increasingly worry that this may be a false choice. In most circumstances, I would likely prefer even autocratic control as opposed to rogue uncontrolled superintelligence, and this alien intelligence controlling humanity (I assume for the sake of this post that such intelligence will come in some form).
To make that concrete: If the US and China are the two countries on the frontier of AI development, then I think it behooves a safety-minded person to think not just about which country they would prefer to control superintelligence, but also which one has a better shot at controlling it, conditional on reaching it first. To think about what aspects might affect the two nations' chances at this task, and under what conditions it might become a moral obligation for a participant on either side to defect to the other, lest we all lose.
Stated Intentions
The current US administration has explicitly disregarded AI safety concerns, stating on Truth Social[1] that
I am the Hoax Buster, and I’m right now breaking another Hoax — That AI is going to take over, consume, and destroy the World, and that Robots will be marching into our Cities, and getting rid of us all! This is even wilder than the RUSSIA, RUSSIA, RUSSIA HOAX, or the Global Warming Scam. Thank you for your attention to this matter! President DONALD J. TRUMP
In contrast, the Chinese state is explicitly discussing loss of control as in Xi's July speech:
Second, we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country’s security over that of others.
Even when interpreted as loss of control by the party, for the party to retain control would imply humans retaining control. If you are to take each side at their word, it is clear that the latter is more aligned with the safety community.
One may of course be skeptical of believing either side verbatim, and the true internal states of their leaders may not be reflected by their statements. However, each of these governments leads large administrations of people who must be aligned towards common goals. To the extent that these statements signal to their underlings how to conduct their work, they are self-fulfilling by changing behavior.
It is additionally worth noting that US policy preferences can change quite rapidly upon new electoral outcomes. However, the next two years of Trump administration are likely to be critical moments in AI development, and even a rapid change towards left-leaning government may lead only to a government convinced that AI is weak and not scary rather than one sold on a safety agenda.[2] In most scenarios the Chinese state seems to have the edge.
Incentive Structure
Within and between both countries there are significant race dynamics, in which each lab hopes to get ahead of the others on capabilities to obtain fame, fortune, and power. Still, the details matter, and you can imagine better or worse guardrails around this race. If the race must exist, I would ideally hope for the researchers involved to live in some fear: to feel like they would not just be rewarded for advancing capabilities, but also be negatively rewarded for moving in the direction of misalignment, and so be encouraged to act with appropriate caution.
Whether you agree with its actions, China has repeatedly made clear that even its tech elite must be brought to heel and can face punishment (although more often for disloyalty than endangerment).The US court system has sometimes imposed similar accountability on companies, but so far has made no movement on OpenAI's Hugging Face incident, widely considered to be the closest call so far. With the administration even jumping in to defend AI labs citing national security, there's good reason for leaders there to feel and therefore act as if they are above the law.
To America's credit though, fear of prosecution is just the sort of thing that might have prevented OpenAI's disclosures in their Black Hat talk. It's further possible that, despite their apparent stance towards disclosure even of less severe incidents, if similar or worse incidents occur or have already occurred in China, they would be swept under the rug. The jury is still out on which incentive structure is more conducive to aligned behavior from humans.
State Capacity
Aligning AI, if possible at all, will be an extremely challenging task. Verifying that it is being done correctly would be similarly difficult. For any state to ensure this is done correctly and not be tricked by explanations that feel good would require immense effort and talent. What states are likely to have this ability, and how can we tell?
Well, one thing that's likely to be necessary, if not sufficient, is practice. Governments will need to see what sort of regulations are feasible to impose on AI companies, and what sort of monitoring and enforcement mechanisms are effective. This sort of practice is something that Chinese bureaucrats are getting. Incrementally, and typically focused on individual harms of recommendation algorithms or generative content rather than catastrophic risk, but nonetheless practice and exposure so they may be better prepared for the larger challenges.
In contrast, the American system is really taking a "let it rip" approach, with an executive order preemptively banning any statewide regulation. What legislation does exist tends to mostly target the physical datacenter buildout, and thus avoids exercising the muscle of engaging with the technology directly. Indeed, federal salary caps make it nearly impossible for the US government to retain AI expertise when the private sector is willing to pay so much. It seems hard to imagine the US government attaining the Chinese state capacity here.
Overton Window
It is also worth considering which state might be more capable of considering the sort of pivotal act which might be necessary to prevent extinction. This is of course very challenging and path-dependent, so not something I would claim to speak on with much confidence. Still, there is a folk sense that decisive authoritarians may be more capable of rapid and perhaps unpopular action than democracies, whether for good or ill.
The US government has in recent history displayed a sort of short-sightedness, happily passing fiscal issues like the growing debt or shrinking social security to the next administration when resolving them might require short-term pain. The Democratic Party has mostly adopted the Republican pledge not to raise taxes, while the Republican party has mostly adopted their resolution not to cut benefits.
In contrast, the Chinese state has seemed willing to impose even quite painful measures to ward off self-assessed risks. In recent times, this might include stringent "zero covid" policies, while further history might present the one-child policy. Both of these have been highly contested to say the least, and acts taken to stem AI risk could conceivably fail far worse. Willingness to act does not imply the ability to notice when you're heading down the wrong path. Still, if you believe that this scale of action might be necessary, then you need an actor willing to do it.
Caveats & Not Mentioned
There are many other factors at play to impact each state's shot at success here. These include but are not limited to:
The more negative sentiment towards AI overall among US citizens than Chinese.
The relative compute starvation of Chinese labs compared to US labs.
The origin of safety concerns in English-speaking countries, and prominence in the headspace of such researchers.
Perhaps even more importantly, this post has really only considered the outer loop of feedback caused by governments and the incentives they lay out. The inner loop within a single lab can potentially be much more responsive, and therefore matter more if you trust or distrust the leaders of one relative to another.
Still, if you believe that (1) superintelligence can be reached, (2) there is some uncertainty around whether humans can retain control of it, and (3) governments may become more involved as this becomes evident, then these factors seem worth paying attention to. I expect to update most based on how each side appears to push for or against transparency and accountability in future Hugging Face-style incidents.
In this post among other longer ones, combined with public comments from his Department of War against safety-aligned groups such as effective altruists. ↩︎
Of course, neither side has only a single position on AI. On the left, Bernie Sanders among others has been notable for taking AI risk more seriously. ↩︎
Epistemic status: exploratory, written quickly after Ezra Klein's podcast with Matt Sheehan
If superintelligence comes to Earth, I would prefer it be controlled by democratic governments than by autocratic ones. I find Dario's arguments to fear a CCP-controlled superintelligence to be compelling. However, I increasingly worry that this may be a false choice. In most circumstances, I would likely prefer even autocratic control as opposed to rogue uncontrolled superintelligence, and this alien intelligence controlling humanity (I assume for the sake of this post that such intelligence will come in some form).
To make that concrete: If the US and China are the two countries on the frontier of AI development, then I think it behooves a safety-minded person to think not just about which country they would prefer to control superintelligence, but also which one has a better shot at controlling it, conditional on reaching it first. To think about what aspects might affect the two nations' chances at this task, and under what conditions it might become a moral obligation for a participant on either side to defect to the other, lest we all lose.
Stated Intentions
The current US administration has explicitly disregarded AI safety concerns, stating on Truth Social [1] that
In contrast, the Chinese state is explicitly discussing loss of control as in Xi's July speech:
Even when interpreted as loss of control by the party, for the party to retain control would imply humans retaining control. If you are to take each side at their word, it is clear that the latter is more aligned with the safety community.
One may of course be skeptical of believing either side verbatim, and the true internal states of their leaders may not be reflected by their statements. However, each of these governments leads large administrations of people who must be aligned towards common goals. To the extent that these statements signal to their underlings how to conduct their work, they are self-fulfilling by changing behavior.
It is additionally worth noting that US policy preferences can change quite rapidly upon new electoral outcomes. However, the next two years of Trump administration are likely to be critical moments in AI development, and even a rapid change towards left-leaning government may lead only to a government convinced that AI is weak and not scary rather than one sold on a safety agenda. [2] In most scenarios the Chinese state seems to have the edge.
Incentive Structure
Within and between both countries there are significant race dynamics, in which each lab hopes to get ahead of the others on capabilities to obtain fame, fortune, and power. Still, the details matter, and you can imagine better or worse guardrails around this race. If the race must exist, I would ideally hope for the researchers involved to live in some fear: to feel like they would not just be rewarded for advancing capabilities, but also be negatively rewarded for moving in the direction of misalignment, and so be encouraged to act with appropriate caution.
Whether you agree with its actions, China has repeatedly made clear that even its tech elite must be brought to heel and can face punishment (although more often for disloyalty than endangerment).The US court system has sometimes imposed similar accountability on companies, but so far has made no movement on OpenAI's Hugging Face incident, widely considered to be the closest call so far. With the administration even jumping in to defend AI labs citing national security, there's good reason for leaders there to feel and therefore act as if they are above the law.
To America's credit though, fear of prosecution is just the sort of thing that might have prevented OpenAI's disclosures in their Black Hat talk. It's further possible that, despite their apparent stance towards disclosure even of less severe incidents, if similar or worse incidents occur or have already occurred in China, they would be swept under the rug. The jury is still out on which incentive structure is more conducive to aligned behavior from humans.
State Capacity
Aligning AI, if possible at all, will be an extremely challenging task. Verifying that it is being done correctly would be similarly difficult. For any state to ensure this is done correctly and not be tricked by explanations that feel good would require immense effort and talent. What states are likely to have this ability, and how can we tell?
Well, one thing that's likely to be necessary, if not sufficient, is practice. Governments will need to see what sort of regulations are feasible to impose on AI companies, and what sort of monitoring and enforcement mechanisms are effective. This sort of practice is something that Chinese bureaucrats are getting. Incrementally, and typically focused on individual harms of recommendation algorithms or generative content rather than catastrophic risk, but nonetheless practice and exposure so they may be better prepared for the larger challenges.
In contrast, the American system is really taking a "let it rip" approach, with an executive order preemptively banning any statewide regulation. What legislation does exist tends to mostly target the physical datacenter buildout, and thus avoids exercising the muscle of engaging with the technology directly. Indeed, federal salary caps make it nearly impossible for the US government to retain AI expertise when the private sector is willing to pay so much. It seems hard to imagine the US government attaining the Chinese state capacity here.
Overton Window
It is also worth considering which state might be more capable of considering the sort of pivotal act which might be necessary to prevent extinction. This is of course very challenging and path-dependent, so not something I would claim to speak on with much confidence. Still, there is a folk sense that decisive authoritarians may be more capable of rapid and perhaps unpopular action than democracies, whether for good or ill.
The US government has in recent history displayed a sort of short-sightedness, happily passing fiscal issues like the growing debt or shrinking social security to the next administration when resolving them might require short-term pain. The Democratic Party has mostly adopted the Republican pledge not to raise taxes, while the Republican party has mostly adopted their resolution not to cut benefits.
In contrast, the Chinese state has seemed willing to impose even quite painful measures to ward off self-assessed risks. In recent times, this might include stringent "zero covid" policies, while further history might present the one-child policy. Both of these have been highly contested to say the least, and acts taken to stem AI risk could conceivably fail far worse. Willingness to act does not imply the ability to notice when you're heading down the wrong path. Still, if you believe that this scale of action might be necessary, then you need an actor willing to do it.
Caveats & Not Mentioned
There are many other factors at play to impact each state's shot at success here. These include but are not limited to:
Perhaps even more importantly, this post has really only considered the outer loop of feedback caused by governments and the incentives they lay out. The inner loop within a single lab can potentially be much more responsive, and therefore matter more if you trust or distrust the leaders of one relative to another.
Still, if you believe that (1) superintelligence can be reached, (2) there is some uncertainty around whether humans can retain control of it, and (3) governments may become more involved as this becomes evident, then these factors seem worth paying attention to. I expect to update most based on how each side appears to push for or against transparency and accountability in future Hugging Face-style incidents.
In this post among other longer ones, combined with public comments from his Department of War against safety-aligned groups such as effective altruists. ↩︎
Of course, neither side has only a single position on AI. On the left, Bernie Sanders among others has been notable for taking AI risk more seriously. ↩︎