I noticed I'm confused about the "race with China" aspect of AGI, and I can see people I respect who care about extinction risks taking it seriously.
"If China beats us" obviously implies that somehow if the Chinese state gets AGI first, then they can use it to permanently disempower the American state, or something like enforce their authority worldwide with no hope of ever challenging them again.
They can't possibly be arguing that China find a cancer cure 6 months earlier than the USA, because then who cares about being first? We get the cure anyway, and USA cures heart disease 6 months after that..Everyone wins, so it can't be that. While I understand it's nice to have the best social media industry for example, no one would claim we can't unilaterally ban social media, because China create more engaging apps (assuming there were harms we're trying to avoid of course)
So implicit in this is a frame of some kind of permanent disempowerment / hegemonic ability that is purely the capability of this single technology. This level of AI hasn't appeared yet, but the posters all believe it's possibly fairly soon.
Similarly, if the American state somehow gets to control this, I similarly assume they're claiming it would use it to disempower the CCP or permanently stamp an even more powerful US hegemony or bring democracy worldwide or something like that.
Ok fine, I'm following so far...But I mean this seems like an obvious naive fiction that doesn't track how the world would actually work. If you are a lab who is staring at a technology with this power, then why would you then go ahead and hand it to your government? You can use this to make FrontierLab LLC become world government instead of giving it to the US president? Like obviously, if you could implement your preferred political system worldwide, you would do so? You already claim the CCP leaders or US state leaders would do this. But somehow AI lab leaders (both Chinese and US ones) are immune from this desire? You can take this one step further and say the team that is close to launching this would do it themselves? If you are the one who gets to first ask your genie what to do, are you going to ask it to implement your preferred world, or Dear Leader's (either your country or your lab's)?
Now things may be different if the labs were nationalized and the state had a system of intense scrutiny on the exact current capability level and checks that ensured the president or congress calls the shots when this level of AI first appears. But none of these "beat China" posters are calling for the labs to be nationalized, or CCP-style state-embedding into these companies where they review all decisions.
The other option is AGI is agentic on it's own. Maybe they're claiming the data it's trained on will change what the agent ends up wanting? Like somehow, we RL the AI to saying it loves the US constitution and it ends up truly wanting that? And if the Chinese RL it into saying the CCP is great, then it somehow get's aligned to the CCP? I mean, this seems like a unserious view today, but if that's what they actually believe, I'd like them to state that upfront so I can argue that instead.
It probably does matter what specific rules our global overlord has, and it does seem like US hegemony has been pretty good for the world in general (vs. most alternatives I can think of). But yeah CCP hegemony > extinction, assuming they solve alignment of course. The Chinese seem to be living fairly decent lives too.
I'm personally a fan of "let the many flowers bloom" philosophy of states and think we should have 10x the number of countries we have today. Having a personal (human) group have indefinite power seems bad.