Cross-posted from my website.

Prior discussion: niplav's shortform (2025); Planning for Extreme AI Risks (2025) by Joshua Clymer

A frontier AI company (any one, I don't care which) should close shop and make an announcement along the lines of:

Powerful AI could end the human race. We are too worried that we don't know how to make this technology safe. We have decided to shut down because we don't want to be responsible for building the thing that kills us all.

A common refrain among safety-conscious AI developers: "it doesn't matter if we stop building dangerous AI, because someone else will just build it instead." Is that really true, though? If a multi-hundred-billion-dollar company comes out and says "We've concluded that our product is horribly dangerous, nobody knows how to make it safe, and there's too high a risk that it leads to human extinction", this won't raise any eyebrows? This has no chance of spurring policy-makers into action?

Shutting down would make people say, holy shit, they are serious about this extinction risk thing. Shutting down sends a strong signal to governments that they should pay serious attention to AI x-risk.

It also encourages other companies to take safety more seriously. Right now, at least three AI companies have said something like, "maybe we'd prefer to slow down and pay more attention to safety, but then the other companies will plow ahead recklessly." If one company decides not to plow ahead recklessly, and actually stops building existentially dangerous technology, that sends a hard-to-ignore message that coordination might be possible.

If a frontier AI company shuts down, will that work? Will companies work together to slow down? Will we get sane AI regulations as a direct result of the shutdown? Probably not. It won't singlehandedly solve all the coordination problems. But it's still a better idea than the current strategy of "race ahead while doing a dash of safety research on the side", which is even less likely to work. By AI companies' own admission, competitive pressures don't allow them to slow down. Why would things change in the future? How are they going to align AI if they have to move at maximum speed? Even if they slow down somewhat, what if alignment is hard ^[1] , and they can't slow down by enough to properly solve the problem?

Counterpoint: If the most safety-conscious company shuts down, then ...

Bruno Muchilwa

Bruno Muchilwa