In my quote I am giving a summary of what seem to me the crucial shared attributes between Dario, Elon and Sam Altman. I am giving an exaggerated version for rhetorical affect, but I am not actually saying anything false. Scott literally says that the plan he wants Anthropic to pursue is to stop right at the brink, and does not provide any mechanism of verification for how we can know in advance that they will stop at the brink.
I think the part of my summary that would be fine to criticize is the sentence "please don't try to stop me". Scott is saying that we should try to stop Dario, Elon and Sam, but only if we can stop all three, and also China at the same time, not if we can only slow down the leading actor (who in Scott's plan is Dario). I think it would be OK to criticize that part as potentially misleading, though I think there is a simple clarification that would make it solidly correct.
in some sense this is to the credit of the hostile paraphrase paradigm - it quickly gets to the point of disagreement.
I mean, this paradigm only works if you tell the other party if they get it wrong. Which I am doing here. Are you saying Scott should be allowed to give hostile paraphrases but then I need to be polite when I point out how they are inaccurate, because it was to Scott's credit that he used hostile paraphrases? Scott of course should point out how my exaggerated paraphrase is wrong. I stand behind it as written, and am open to arguments against it. Scott's seems wrong by the same standard, and I am open to arguments against that too. I disagree with your footnote how mine is wrong, and now we can argue about it.[1]
Summarizing at least mine, Eliezer's or Robby's position as "you can't use game theory in any way" is just really not anywhere close to what is true. Mine, Eliezer's and Robby's position are, to the best of my knowledge, all downstream of us thinking quite a bit about the game theory, and we've all written many long pieces about how we think game theory should be applied to this situation. It's as if I was exaggeratedly summarizing Scott's piece as "and Scott's plan is founded on the belief that none of the labs are allowed to talk to each other". You could make some extremely thin argument that this position is implied by plan C being the result of coordination breaking down, but it would be immediately falsified by reading Scott's summary of plan C which involves a lot of labs talking to each other and forming coalitions of some kind.
To be clear, this is not an enormously important point, though it's representative of something like 15-20 places in Scott's post where he makes similarly clearly false summaries. Like when he summarizes that (cut together from two sections) "the community consensus from the beginning of the community was to believe in Plan C, which is to use generic capabilities without a deep understanding of alignment to build a system just on the brink of superintelligence and then use that to solve alignment in a mad 6-12 month dash, until 2023 when ChatGPT caused basically everyone to abandon this plan and memory hole that it was the default thing everyone was trying to do since the beginning of the community". That summary is worse, but there are just a lot of places where Scott says straightforwardly false things about what the people he is arguing against believe.
I don't think Scott wants "risk-awareness" as the primary attribute. Sam Altman is very aware of the risks, and Scott repeatedly emphasizes how important an Anthropic lead over OpenAI is. Scott's plan of course requires the people in the lead to use the 6-12 months of potential lead time they can burn at the end for the kind of alignment research that Scott thinks has any chance of building aligned Superintelligence. He doesn't go into a ton of detail on what that kind of research would be, but the primary term he uses is "alignment-pilled".
Summary: Scott Alexander’s Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott’s post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on.
The good version of unilateral pausing is good
Here is the good version of unilateral pausing:
There are multiple places in the post where Scott makes it clear he wants the reader to be thinking of much worse plans than this. Emphasis added:
Again, the correct version of a unilateral pause is a company pivoting to safety + security + product, not “disbanding their entire company immediately”.
This is again arguing against a strawman: OpenAI would not be “out of the industry” if they decided to unilaterally pause. Sam Altman loves to build products! He would very much still be in the industry.
It is possible that Scott will correctly point out that some people do advocate for the extreme version here, but I don’t think he should get credit for this. The title of the post is “Against Stop Unilaterally and Raise Awareness”. I think it is perfectly reasonable for someone to say that SURA-the-phrase applies to my better version. His post is very strongly being interpreted as arguing against unilateral stopping in general. It is being interpreted that way because of his title and framing.
Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticism
Scott appears to agree that the good version of unilateral pausing is good, but attempts to claim that only one group is advocating for this good version, the AI Futures Project.
This should not be a parenthetical. If the goal is clarification for people who haven’t thought about the details much, and Scott believes as I do that there is a good version of unilateral pausing, it should have been in the opening section. It is not. The reader should not be expected to review every section before realizing that the title of the post is wrong.
Oversimplified game theory is bad, and you should feel bad
The core argument behind unilateral pausing is that the game theory is complicated, and it is better to go via deontology and other non-consequentialist analyses if you can’t model the game theory well. Scott reasonably points out that one shouldn’t just say this: he wants details. Unfortunately, in the absence of others providing details, he provides his own game theory and makes basic mistakes, providing evidence for the original why-deontology point.
In particular (emphasis added):
Agreeing to pause conditional on others pausing is not strictly easier than unilaterally stopping. There are three reasons for this: (1) “some definition” hides dragons, (2) promises for future actions can be harder than actions, even if the immediate actions are stronger, (3) the boundary between frontier lab and not is ambiguous.
“Some definition” hides dragons
There are several versions Scott could have meant by “some definition”:
It should be clear that getting a company to agree to the general version is out of the question: it is too wide a promise. Set that aside.
The internal definition version is useless without additional rounds of negotiation. If all companies come up with their own definitions and agree to act on them if others act on their own internally-generated definitions, the immediate outcome is no action. Additional rounds of negotiation then clearly falsify “strictly easier”.
The external definition version is also not strictly easier. It is in practice much harder for a company to agree on an external definition than to take action based on their own internal definition. This is a combination of not-invented-here syndrome and different definitions affecting different companies in different ways due to different team structures and research programs.
I am not guessing here. I have spoken directly to multiple people at all of Anthropic, OpenAI, and DeepMind about unilateral pauses, and they are all sketched out by this question of the precise definition of a pause. One of those people was the CEO of one of those labs. Hiding this complexity behind “some definition” is oversimplifying the game theory, and that results in bad analysis and bad conclusions.
Now of course, unilateral pausing also means definition mismatch and multiple rounds of negotiation. It might be the case that one company pauses according to definition D1, and another company refuses to follow suit unless they strengthen to definition D2. But accepting that we live in a multiround negotiation world is one of the core arguments behind unilateral pauses: the goal is to start a process that can then ratchet up through negotiation.
Promises can be harder than actions
It is not the case that promising to do X in the future based on condition Y is strictly easier than doing X now. This is because
The above are some abstract first principles. But they conform to how AI company executives speak about the issue, and how I’ve seen negotiations break down in the past: Executives are very uncomfortable making conditional promises.
As one example, DeepMind’s attempt to become a company limited by guarantee (CLG) failed in part for similar “reality fails to conform to simplistic game theory” reasons. DeepMind leadership was attempting to guard against “AI capabilities are crazy” worlds where risks are high, and the Google Board was attempting to guard against “AI fizzles and DeepMind wastes a bunch of money” worlds. Simplistically, there was an arbitrage to be made. In practice, negotiations fell through. One should be very careful when assuming that simple theorems of simple game theory apply.
How many frontier labs are there?
Another reason it is not strictly easier to do a Scott-version conditional pause is that it is not clear how many frontier labs there are. In the U.S., the number is somewhere between 2 and 6. There are clearly worlds in which getting 2 companies to do a unilateral and then bilateral action is easier than to get 6 parties to do even a much weaker action, and I claim we are in one of those worlds.
Definitional fairness is less important than survival
As mentioned, when I talk to people at AI companies, they are very concerned that if they stop, other people will stop slightly less, and it will be unfair. We can contrast the following two risks from AI:
The second risk is more important!
Now, one can argue that the first risk is important too, and indeed significant work should go into minimizing the unfairness, since that unfairness does impact pause effectiveness. Too much unfairness means we don’t slow down AI capability progress enough and don’t buy enough time. But I think that fairness is to be achieved over time, after multiple rounds of action, and should not be a precondition for taking the first step.
It may be much easier for an AI company to act on their own definition than by one imposed on them externally. So shifting the conversation to the relative unimportance of the definition used for the first action may be critical to actually causing that action to happen. This applies not just to unilateral or bilateral pauses but also to the grand international coordination that Scott prefers. In that context, it is notable that Scott’s only engagement with definitions is using the phrase “some definition”, handwaving a bit, and adding a footnote about forecaster wording.
Once we will fill in the subtleties around definitions and the details of the game theory, the correct action is to stop.