My position (condensed into a few paragraphs, though of course I believe much more on this topic):
The moral thing to do for a company is to spend a large fraction of their resources on slowing down everyone equally, while also shouting desperately from the rooftops about the risks and the need for a pause (substantially above the minimum floor of what Anthropic did in the DoD dispute, and what Uber did in various municipalities to use its platform to lobby for laws that enable ride-sharing, which are inadequate but reasonable illustration what trying at all might look like).
The right commitment level for something like a pause is tricky to choose, and different commitments generalize differently. Simpler commitments are easier to verify, and irreversible commitments produce more stable equilibria. Of course more complicated commitments maintain more flexibility and are cheaper, and so are more likely to be followed by other actors if they don't agree on the importance of the thing the commitment is trying to achieve.
It's easier to verify that your company is genuinely out of the race than whether your company is pausing but in a way where it could restart any time. The default assumption in the latter situation will be that you are not pausing and are just continuing research covertly or in a way that is conceptually confusing and maybe in the letter but not the spirit of the norm you are trying to propagate.[1] This means setting a norm to irrevocably stop is probably more likely to generalize by being much easier to enforce (though of course less likely on base-rates). I personally don't really see much of any difference between OpenAI having a 6 month lead or Anthropic having a 6 month lead, so the expected likelihood of any current actors at the frontier actually successfully stopping at the brink of superintelligence and solving alignment in a mad dash is the same, and so from an impartial norm perspective, it seems better for any actor to just leave the race and take the wins of easier verification and a bit more time for other things to catch up.[2]
Even better than stopping unilaterally is to try to take as much of the industry down with you. The best way to go out is to spend all your influence and cash and talent on actions that slow down everyone. Ideally by lobbying for regulation that just slows down everyone, but there are also other ways of e.g. trying to hire away key staff and broadly increasing anti-AI sentiment with candid communication about the risk, etc. This also generalizes better since you need a smaller fraction of the industry to do the same in order to get a complete pause (simplified, if any company could spend its resources to take down one other company the same size, you only need half of the companies to join your efforts, and in reality the balance here seems more favored towards offense than 1:1).
From a nation-state perspective, the situation seems a bit clearer to me. The U.S. could halt global AI progress unilaterally, and basically nothing in its enforcement apparatus relies on being ahead in AI. The U.S. should unilaterally stop, then use conventional methods to force other countries to do the same.[3]
If that somehow fails, which seems unlikely[4] given how extremely compute-scaling driven progress has been and how easy global compute input regulation is, but not impossible, then the U.S. should probably restart AI development domestically. If you're really backed into a corner and left with no other choice, be loud, tell everyone that they are fucking insane and going to kill us all, then pick up the gun again and try to somehow flip the game board by being ahead.
But again, verification is tricky and path-dependence is important. The U.S. smashing its GPUs and smashing the global semiconductor supply chain is a much more verifiable signal that it's actually no longer racing than anything else. Given that the probability of successfully slowing down the rest of the world via conventional escalation seems quite high to me, the benefits of choosing an easier-to-verify and less fragile equilibrium are great. So probably the U.S. should have chosen a path of a hard-to-reverse pause (probably not literally smashing the GPUs, but more just a bill which clearly binds the executive's hands and which can only be changed via a highly transparent process like congressional debates and a replacement bill).
And then, the fact that nation-state coordination seems better-placed to handle this situation than inter-company coordination, due to the offense dominance of current conventional techniques for retarding AI progress, makes marginal slowdown on the company level much more valuable. The U.S. does not need to be leading the race to stop other actors from building powerful AI systems. Every week the world is waking up more to the threat of AI, which increases the likelihood of appropriate action here, and a U.S. that is behind as a result of domestic regulation and companies choosing to go slower is if anything more likely to turn into an actor that will try to slow down global AI progress, because it is no longer tempted by the (largely false) fruits of potentially "winning". This means the cost of the U.S. falling behind globally is smaller because the "race" is not denominated in AI progress, it's instead substantially determined by which of the few existing superpowers is willing to sabotage the others with conventional escalation, with extremely fragile global semiconductor supply chains which are vulnerable to sabotage by a large number of different actors.
My current best guess is that the world would be better off if the U.S. was behind in the global race due to dynamics like this, since it would spur the U.S. to action to drive a global slowdown[5], for which it is by far the best positioned actor in the world (and it would be very marginally, but almost imperceptibly better if Anthropic was leading in the U.S. over other companies in a way that would make that consideration largely irrelevant for plans here).
And then also, at the end of the day, you should not build doomsday devices. Even if in the end, all things considered you will choose to race towards ASI again, you should do so while accepting great censure and shame, and be ready to go to jail and stand trial for your crimes if we do somehow survive. You should be shown leniency, and possibly even redemption in the history books, but ultimately still face some substantial punishment (while maybe still netting in the green). Not threatening everyone's lives is important, and liability law and reckless endangerment do not provide an escape hatch of "but like 5 other companies were going to recklessly endangering people if I wasn't going to and I was recklessly endangering people for the greater good", and neither do common-sense principles of justice.
So if you do it, and it's the right thing to do, you should still be shunned for it as you cross the line, and in the meantime you can wipe your tears with the billions of dollars you will make along the way.
I like Zvi's coverage of this in his response to you. The point of deontological prohibitions of this kind, in mine and his book, is not to create lines you can never cross. It's to create lines whose price you have to pay no matter your reason. There is a finite, if extremely large, social cost to building a doomsday device, which needs to be paid as you cross it no matter your reason. And that finite cost can be overcome, and a sinner of such acts could be redeemed if the history books play out the way you are predicting they will, but they shouldn't be absolved in advance.
- ^
The OpenAI "pause" you keep citing seems like the perfect example of this. I don't believe OpenAI's actions meaningfully set back its capabilities at all, and I have no way to verify this "supposed" pause. Of course if OpenAI had actually given up its GPU contracts or credibly stopped some other way, then I could verify that, but I can't and now people like you keep saying that OpenAI "paused". I don't believe it and of course none of the other actors believe it. Short pauses are hard to verify and less stable!
- ^
There exist potential organizations that have demonstrated enough competence and understanding of the alignment problem that I would see substantial value in them having a lead, but this seems very hard to achieve given current pressures on frontier companies, and any such company, in order to pursue a plan where it tries to pull ahead, would need to get much wider buy-in than from just me, to get my support in racing ahead and the associated costs to more generalizable norms
- ^
Similarly, other countries including middle powers should stop and then threaten escalation using the tools they have to cause things to slow down (e.g. the Netherlands should use their leverage over ASML to slow things down, and the European union should threaten sabotage of foreign datacenters, and god forbid, if there is truly no other choice and all other options have been exhausted, Europe should invoke the threat of nuclear weapons)
- ^
At least right now. As we delay actions like this we could end up in a more software-progress dominated world where such actions become harder. This mostly just makes all plans harder and is a reason for why the right time to pause is now, and why marginal time bought right now via unilateral slowdowns is extremely valuable, as the U.S. government seems to become much more likely to pursue such strategies with every passing month.
- ^
I give an example of such a scenario here: https://www.lesswrong.com/posts/xikGZ8fuhdG6DApAo/against-sura?commentId=eytaDPdqvixir5mjF
2
1
1
1
1
Summary: Scott Alexander’s Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott’s post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on.
The good version of unilateral pausing is good
Here is the good version of unilateral pausing:
There are multiple places in the post where Scott makes it clear he wants the reader to be thinking of much worse plans than this. Emphasis added:
Again, the correct version of a unilateral pause is a company pivoting to safety + security + product, not “disbanding their entire company immediately”.
This is again arguing against a strawman: OpenAI would not be “out of the industry” if they decided to unilaterally pause. Sam Altman loves to build products! He would very much still be in the industry.
It is possible that Scott will correctly point out that some people do advocate for the extreme version here, but I don’t think he should get credit for this. The title of the post is “Against Stop Unilaterally and Raise Awareness”. I think it is perfectly reasonable for someone to say that SURA-the-phrase applies to my better version. His post is very strongly being interpreted as arguing against unilateral stopping in general. It is being interpreted that way because of his title and framing.
Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticism
Scott appears to agree that the good version of unilateral pausing is good, but attempts to claim that only one group is advocating for this good version, the AI Futures Project.
This should not be a parenthetical. If the goal is clarification for people who haven’t thought about the details much, and Scott believes as I do that there is a good version of unilateral pausing, it should have been in the opening section. It is not. The reader should not be expected to review every section before realizing that the title of the post is wrong.
Oversimplified game theory is bad, and you should feel bad
The core argument behind unilateral pausing is that the game theory is complicated, and it is better to go via deontology and other non-consequentialist analyses if you can’t model the game theory well. Scott reasonably points out that one shouldn’t just say this: he wants details. Unfortunately, in the absence of others providing details, he provides his own game theory and makes basic mistakes, providing evidence for the original why-deontology point.
In particular (emphasis added):
Agreeing to pause conditional on others pausing is not strictly easier than unilaterally stopping. There are three reasons for this: (1) “some definition” hides dragons, (2) promises for future actions can be harder than actions, even if the immediate actions are stronger, (3) the boundary between frontier lab and not is ambiguous.
“Some definition” hides dragons
There are several versions Scott could have meant by “some definition”:
It should be clear that getting a company to agree to the general version is out of the question: it is too wide a promise. Set that aside.
The internal definition version is useless without additional rounds of negotiation. If all companies come up with their own definitions and agree to act on them if others act on their own internally-generated definitions, the immediate outcome is no action. Additional rounds of negotiation then clearly falsify “strictly easier”.
The external definition version is also not strictly easier. It is in practice much harder for a company to agree on an external definition than to take action based on their own internal definition. This is a combination of not-invented-here syndrome and different definitions affecting different companies in different ways due to different team structures and research programs.
I am not guessing here. I have spoken directly to multiple people at all of Anthropic, OpenAI, and DeepMind about unilateral pauses, and they are all sketched out by this question of the precise definition of a pause. One of those people was the CEO of one of those labs. Hiding this complexity behind “some definition” is oversimplifying the game theory, and that results in bad analysis and bad conclusions.
Now of course, unilateral pausing also means definition mismatch and multiple rounds of negotiation. It might be the case that one company pauses according to definition D1, and another company refuses to follow suit unless they strengthen to definition D2. But accepting that we live in a multiround negotiation world is one of the core arguments behind unilateral pauses: the goal is to start a process that can then ratchet up through negotiation.
Promises can be harder than actions
It is not the case that promising to do X in the future based on condition Y is strictly easier than doing X now. This is because
The above are some abstract first principles. But they conform to how AI company executives speak about the issue, and how I’ve seen negotiations break down in the past: Executives are very uncomfortable making conditional promises.
As one example, DeepMind’s attempt to become a company limited by guarantee (CLG) failed in part for similar “reality fails to conform to simplistic game theory” reasons. DeepMind leadership was attempting to guard against “AI capabilities are crazy” worlds where risks are high, and the Google Board was attempting to guard against “AI fizzles and DeepMind wastes a bunch of money” worlds. Simplistically, there was an arbitrage to be made. In practice, negotiations fell through. One should be very careful when assuming that simple theorems of simple game theory apply.
How many frontier labs are there?
Another reason it is not strictly easier to do a Scott-version conditional pause is that it is not clear how many frontier labs there are. In the U.S., the number is somewhere between 2 and 6. There are clearly worlds in which getting 2 companies to do a unilateral and then bilateral action is easier than to get 6 parties to do even a much weaker action, and I claim we are in one of those worlds.
Definitional fairness is less important than survival
As mentioned, when I talk to people at AI companies, they are very concerned that if they stop, other people will stop slightly less, and it will be unfair. We can contrast the following two risks from AI:
The second risk is more important!
Now, one can argue that the first risk is important too, and indeed significant work should go into minimizing the unfairness, since that unfairness does impact pause effectiveness. Too much unfairness means we don’t slow down AI capability progress enough and don’t buy enough time. But I think that fairness is to be achieved over time, after multiple rounds of action, and should not be a precondition for taking the first step.
It may be much easier for an AI company to act on their own definition than by one imposed on them externally. So shifting the conversation to the relative unimportance of the definition used for the first action may be critical to actually causing that action to happen. This applies not just to unilateral or bilateral pauses but also to the grand international coordination that Scott prefers. In that context, it is notable that Scott’s only engagement with definitions is using the phrase “some definition”, handwaving a bit, and adding a footnote about forecaster wording.
Once we will fill in the subtleties around definitions and the details of the game theory, the correct action is to stop.