Let's talk about the AI coordination problem

KatjaGrace

Let's talk about the AI coordination problem — LessWrong

161 Let's talk about the AI coordination problem

by KatjaGrace

4th Sep 2026

2 min read

161

Yesterday I asked if this ‘coordinate not to build dangerous AI’ problem was actually easy.

Why would I think that, contrary to so much belief?

Well, I don’t feel like I’ve actually heard much about the detail of it. In my experience people don’t talk about it like it’s a real practical problem with details, like the negotiation to end a war.

They also don’t talk about it like it’s a serious problem of global geopolitical import, like the negotiation to end a war.

It’s more like a topic for obscure intellectuals, sophomores and trolls to discuss for as long as it takes for one to mention it and another to assuredly dismiss it.

If we treated negotiation to end a war similarly, state leaders would never attempt it, and if you suggested it on social media, the conversation would mostly be strangers appearing to tell you you’re an idiot because you obviously can’t coordinate thousands of people not to kill each other. (Also, do you not realize there are big financial incentives? And if you somehow stopped Country A from killing people from Country B, Country A is just going to pay someone else to do it!)

That is, saying things that sound plausible on a five-second analysis from some reasonable heuristics, with zero curiosity about whether their assessment is missing anything key, or if there might be a way to a solution.

This would be a huge mistake! Pragmatically, you can negotiate to end wars. And given this, it is very important to notice this and do so, because otherwise lots of people die.

The stakes of AI racing are just as real, though our vision of them is hazier at present. But with AI racing, everyone seems to just nod along with this level of argument.

This is not enough to give me the confidence to give up! I do not trust this kind of thinking.

And thinking about it in a ‘real-world pragmatic problem that we want to solve’ mental mode, rather than a ‘abstract philosophical chit-chat’ mental mode, I notice confusion at finding such coordination super hard, let alone unimaginably hard beyond worth trying, and especially so for the specific people involved. If you can coordinate to continue as CEO of a company that you were just fired from and for the board to leave instead, I think there’s some chance you can sort out a more run-of-the-mill ‘if you all don’t, we won’t’ deal.

What do I think should happen differently?

I. We should talk seriously about AI coordination

What would need to happen for coordination to avoid building potentially catastrophic AI? If it’s so hard, what steps are difficult? What would a road map look like? Who would need to do what? If there are hard bits, are there alternatives? We should have a detailed picture, and discuss it like a real problem that we care about solving.

II. AI leaders should talk seriously about AI coordination

If we were taking this situation seriously and expecting these people to do a real job navigating the risk, would we settle for assuming they’ve sufficiently considered and pursued the options for coordinating, without discussing it at all in public?

If they are so sure it can’t be done, why is that? Shouldn’t we demand to know?

AI ControlAI GovernanceAI

Frontpage

161

Let's talk about the AI coordination problem

New Comment

13 comments, sorted by

top scoring

Click to highlight new comments since: Today at 7:05 AM

[-]Buck1mo1411

I agree that it would be a mistake to assume that this coordination is impossible, but I think your "ending wars" example is less rosy than you make it out to be. In particular, wars famously sometimes happen for a while before they end, even though at every moment the war is usually worse than a negotiated settlement. There are various explanations for this (my favorites are Fearon, and Schelling "Strategy of Conflict" and some of "Arms and Influence"). I think that the a priori position should be that mutually damaging failures to coordinate sometimes happen and sometimes don't.

[-]Mateusz Bagiński1mo71

Quoting gugu

Dario rejects doomerism re:misalignment, fair enough.
But what of doomerism re:slowdown?
Furthermore, the last few years should make clear that the idea of stopping or even substantially slowing the technology is fundamentally untenable. The formula for building powerful AI systems is incredibly simple, so much so that it can almost be said to emerge spontaneously from the right combination of data and raw computation.
[...]

If all companies in democratic countries stopped or slowed development, by mutual agreement or regulatory decree, then authoritarian countries would simply keep going. Given the incredible economic and military value of the technology, together with the lack of any meaningful enforcement mechanism, I don’t see how we could possibly convince them to stop.
Predicting the difficulty of cooperating around / enforcing a "substantial" slow down of AI development seems similarly difficult as predicting the difficulty of avoiding misalignment? Perhaps it is true that this would be historically unprecedented, but as Dario notes, the whole possibility of a country of geniuses in a datacenter is historically unprecedented.
[...]

[-]Harjas1mo1613

If all companies in democratic countries stopped or slowed development, by mutual agreement or regulatory decree, then authoritarian countries would simply keep going. Given the incredible economic and military value of the technology, together with the lack of any meaningful enforcement mechanism, I don’t see how we could possibly convince them to stop.

I really don't understand this point. Authoritarian countries have agreed to nuclear treaties before, right? Is the issue really just the lack of enforcement mechanism?

[-]Mateusz Bagiński1mo82

The issue is the un(der)questioned even deeper atheism on the AI macro-strategic scale that implies that you should grab power because if you don't, then the other less good guy does. As we all know, this has never ever backfired.

[-]Harjas1mo10

(I want to read whatever you intended to link, but right now it just directs me back to the Less Wrong homepage.)

[-]Mateusz Bagiński1mo20

((Ooops, fixed))

[-]Haiku1mo32

The issue is a willful lack of curiosity about verification mechanisms, which are viable in the very near term.

[-]eleweek1mo30

The analogy with the war works in the opposite direction you intend it to!

If we treated negotiation to end a war similarly

We... already do treat it this way? Have you never seen discussion how the Russia/Ukraine war is about incentives?

the conversation would mostly be strangers appearing to tell you you’re an idiot because you obviously can’t coordinate thousands of people not to kill each other.

It sort of is this way! Putin can't stop the war machine easily without getting some kind of a win, at least taking the Donbas. Zelensky can't stop the war machine because losing the Donbas would make the Ukrainian people view him as weak. Additionally, he would have to run a Presidential election. Not too mention his previous corruption would be investigated.

And if you somehow stopped Country A from killing people from Country B, Country A is just going to pay someone else to do it!

Again sort of, in the form: “If Putin got the Donbas, he'd just restart the war”.

state leaders would never attempt it

But yes, neither Putin nor Zelensky want a negotiation. The logic checks out.

And thinking about it in a ‘real-world pragmatic problem that we want to solve’ mental mode, rather than a ‘abstract philosophical chit-chat’ mental mode

What would this look like? What would be the difference between the former and the latter, for a given person whose impact on the world is comparatively small? The talk I hear about ending Russia/Ukraine war is a lot like “abstract philosophical chit-chat” precisely because it's hard for a single person (not counting Putin) to make iterative progress on this.

If anything, the safety/alignment conversations I hear are a lot more practical.

[-]xpym1mo10

Putin can’t stop the war machine easily without getting some kind of a win

I'm pretty sure he can. The Wagner/Prigozhin mutiny embarrassment was easily glossed over, and I'd expect that a low-intensity conflict similar to what was pre-Feb. 2022 could in principle be arranged without much fuss. Of course, he's uninterested in doing this, because all of the reasons for the outright invasion are not only still present, but magnified.

[-]Cat McGee1mo32

This reminds me of a tweet from James Miller:

If aliens arrived on Earth and told us to stop developing superhuman AI or they'll blow up our solar system, we would stop. This thought example shows that we could coordinate to stop if we believed that AI was sufficiently dangerous.

I don't think the issue is coordination. It's about convincing the powers that be of something dangerous that is much more abstract than a war.

[-]Thao Pham1mo20

AI coordination sounds nice in theory (mech design, incentive compatibility, etc.) and somewhat horrible in practice (messy preferences, imperfect approximation, etc.). It's far more under-motivating compared to other social progress problems, and I also feel like AI leaders are deviating from AI coordination because they actually don't want any coordination in the first place.

[-]Charlie Sanders1mo10

It’s worth mentioning that the principal reason given for why we cannot coordinate on AI is because the American AI industry currently understands itself to be representing a version of the future where transformative AI is developed in a free and democratic society instead of a repressive authoritarian dictatorship. Everything is downstream of that conclusion.

Incidentally, the Trump-Xi summit is in two weeks And AI is expected to feature heavily in discussions.

[-]CRISPY1mo1-3

AI coordination is hostile to financial gain, which is what AI is all about. Maybe a handful of people had high minded ideals about what AI can do for humanity, but those people have little control over AI now. The‘ve had to sell that control to command the investment required to make all this happen.

So much money had been borrowed for AI scaling that is has dried up capital markets globally. That kind of spend requires a corresponding risk management strategy and that means banks run the AI companies, regardless of how AI executives act.

The financiers will not permit meaningful coordination or throttling of frontier AI development. At least until something awful happens. Until then, everyone has their own investments to protect and the exposure is preposterously high. Accidentally giving someone an advantage is an unthinkable proposition.

Coordination will not be voluntary. It will require regulation which, again, demands some kind of spectacular disaster to have occurred. The best bet at this time is to put effort into defining what the post disaster regulatory environment will look like. A group needs to be already in place and waiting to fill that role when the time comes.

Moderation Log

More from KatjaGrace

Curated and popular this week

13Comments