An easy coordination problem?

KatjaGrace

Common wisdom says that it is incredibly hard to coordinate to not build more dangerous AI. This sounds believable in the abstract: international geopolitics arms race game theory something something.

But pragmatically, what exactly is the difficulty?

I agree there would seem to be obstacles for the average person. But four of the people apparently succumbing to the overpowering arms race forces while saying AI poses a huge imminent risk to humanity are Sam Altman, Elon Musk, Demis Hassabis and Dario Amodei. Shouldn’t this be fairly tractable for them? What exactly is the difficulty?

Like, if they discussed together and decided they wanted to mutually pause, do you think that wouldn’t happen? Do you think they couldn’t get cooperation from other necessary people? Do you think they couldn’t figure out the verification and policing details?

It’s true that one of the necessary people is the leader of China, but what exactly is the problem there? None of the CEOs have his phone number? He won’t talk to them? He is beyond reason or incentives? He is intent on building AI regardless of how dangerous it is to his own country because he is fundamentally bad? They have nothing he wants?

Like, these people are not only incredibly powerful and wealthy and smart, but they include a Diplomacy world team champion, the acknowledged king of making complex things happen more efficiently than was believed possible, and one of the most gifted social maneuverers in the world. I don’t feel like they are bringing their A game to this.

Picture: Zhongnanhai, photo by 維基小霸王 (Wiki Little Overlord)

As far as I understand, this is a bias similar to the one which has historically caused conventional wars. Unlike Agent-5/Safer-4 and DeepCent-2 from the AI-2027 scenario, who came up with a peace treaty and need only to have the humans accept the treaty's visible part, real humans are biased towards overestimating the probability of their success and/or towards warfare or competition. Or they may have an utility function with convex parts.

Returning to the example with the AI race, mankind would need to unmantle all of these mechanisms.

First of all, the Anthropic Consensus mocked by Kokotajlo and Greenblatt is that alignment is likely easy for Anthropic-like methods. If this is actually the case, then the AI race between those who care about alignment is just a zero-sum game where each company has to take over as big share of power as possible while avoiding bankrupcy, which in turn requires releasing increasingly impressive results and products (or, in China's case, releasing home-made products close to the leaders' capabilities as a defense measure; if DeepCent's AI was aligned, then the AI-2027 forecast wouldn't have ended with China being sold out or genocided)
If Anthropic and OpenAI co-lock in 50% of the world's resources each, then they might implicitly view it as a worse result than having a 49% chance each to take over the world and a 2% chance to destroy the world. Alternatively, coexistence might be implicitly viewed as genuinely impossible.
A special mention goes to the case where Anthropic believes that p(ASI is misaligned|xAI creates it) is close to 100%. Then xAI HAS to be destroyed, put under thorough control to ensure that it doesn't dare to release a misaligned model or at least outcompeted, even if this means that p(Anthropic's ASI is misaligned) reaches 50%.

Returning to the example with the AI race, mankind would need to unmantle all of these mechanisms.

First of all, the Anthropic Consensus mocked by Kokotajlo and Greenblatt is that alignment is likely easy for Anthropic-like methods. If this is actually the case, then the AI race between those who care about alignment is just a zero-sum game where each company has to take over as big share of power as possible while avoiding bankrupcy, which in turn requires releasing increasingly impressive results and products (or, in China's case, releasing home-made products close to the leaders' capabilities as a defense measure; if DeepCent's AI was aligned, then the AI-2027 forecast wouldn't have ended with China being sold out or genocided)
If Anthropic and OpenAI co-lock in 50% of the world's resources each, then they might implicitly view it as a worse result than having a 49% chance each to take over the world and a 2% chance to destroy the world. Alternatively, coexistence might be implicitly viewed as genuinely impossible.
A special mention goes to the case where Anthropic believes that p(ASI is misaligned|xAI creates it) is close to 100%. Then xAI HAS to be destroyed, put under thorough control to ensure that it doesn't dare to release a misaligned model or at least outcompeted, even if this means that p(Anthropic's ASI is misaligned) reaches 50%.

LESSWRONG
LW

LESSWRONG
LW

24

An easy coordination problem?

24

24

24