Against strawmanning unilateral pauses

Geoffrey Irving

Against strawmanning unilateral pauses — LessWrong

357 Against strawmanning unilateral pauses

3rd Nov 2026

9 min read

357

Summary: Scott Alexander’s Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott’s post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on.

The good version of unilateral pausing is good

Here is the good version of unilateral pausing:

Capability pause: Some frontier lab (let’s call them OpenAI or Anthropic) says they have stopped capability research, either just all frontier training runs or both all frontier training runs and all low-scale capability-focused research.
Internal definition: The company gets to pause on its own terms, which is much easier than a generally coordinated pause since they can define the pause in terms of the existing structure of work and teams within the specific company, reducing disruption to other processes. They have said they would like to work with other companies and countries to adapt and strengthen this definition over time, in particular to make it gradually more compatible with actions by other actors. Importantly, we accept that it may be easiest for a company to pause according to its own, company-specific definition, and handle any asymmetry in their agreements via multiple rounds of negotiation with other actors.
Safety + security + product: In place of capability research, their efforts shift to alignment and other safety research, information security, and product. The product part is important! They will still be a trillion dollar company shipping extremely useful tools, and this also provides an escape valve for people within the company who care about products and don’t want to do safety research.
Restart if necessary: The pause is temporary, and will restart after some period of time if progress is not made to broaden the pause. Potentially one wants separate periods based on different kinds of progress. Other US labs are likely to pause sooner than labs in China, since international diplomacy takes time.

There are multiple places in the post where Scott makes it clear he wants the reader to be thinking of much worse plans than this. Emphasis added:

I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.

As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%.

Again, the correct version of a unilateral pause is a company pivoting to safety + security + product, not “disbanding their entire company immediately”.

Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.

This is again arguing against a strawman: OpenAI would not be “out of the industry” if they decided to unilaterally pause. Sam Altman loves to build products! He would very much still be in the industry.

It is possible that Scott will correctly point out that some people do advocate for the extreme version here, but I don’t think he should get credit for this. The title of the post is “Against Stop Unilaterally and Raise Awareness”. I think it is perfectly reasonable for someone to say that SURA-the-phrase applies to my better version. His post is very strongly being interpreted as arguing against unilateral stopping in general. It is being interpreted that way because of his title and framing.

Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticism

Scott appears to agree that the good version of unilateral pausing is good, but attempts to claim that only one group is advocating for this good version, the AI Futures Project.

(A virtuous exception)
I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.
The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.
Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.
This makes sense, and I exempt it from my criticism.

This should not be a parenthetical. If the goal is clarification for people who haven’t thought about the details much, and Scott believes as I do that there is a good version of unilateral pausing, it should have been in the opening section. It is not. The reader should not be expected to review every section before realizing that the title of the post is wrong.

Oversimplified game theory is bad, and you should feel bad

The core argument behind unilateral pausing is that the game theory is complicated, and it is better to go via deontology and other non-consequentialist analyses if you can’t model the game theory well. Scott reasonably points out that one shouldn’t just say this: he wants details. Unfortunately, in the absence of others providing details, he provides his own game theory and makes basic mistakes, providing evidence for the original why-deontology point.

In particular (emphasis added):

By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they'll also stop or slow by that definition.
This is strictly easier than getting them to stop unilaterally. And, unlike stopping unilaterally, it fails gracefully in a situation where some companies agree to it but not others.

Agreeing to pause conditional on others pausing is not strictly easier than unilaterally stopping. There are three reasons for this: (1) “some definition” hides dragons, (2) promises for future actions can be harder than actions, even if the immediate actions are stronger, (3) the boundary between frontier lab and not is ambiguous.

“Some definition” hides dragons

There are several versions Scott could have meant by “some definition”:

General version: Company A agrees that if all other companies B agree to stop, by any definition D, company A will also stop using definition D.
Internal version: Company A comes up with their own specific definition D, and states that if other companies stop according to A’s definition D they will D-stop too.
External version: Advocates for a unilateral pause take their own external definition D to company A, and get A to agree to D-stop if everyone else D-stops.

It should be clear that getting a company to agree to the general version is out of the question: it is too wide a promise. Set that aside.

The internal definition version is useless without additional rounds of negotiation. If all companies come up with their own definitions and agree to act on them if others act on their own internally-generated definitions, the immediate outcome is no action. Additional rounds of negotiation then clearly falsify “strictly easier”.

The external definition version is also not strictly easier. It is in practice much harder for a company to agree on an external definition than to take action based on their own internal definition. This is a combination of not-invented-here syndrome and different definitions affecting different companies in different ways due to different team structures and research programs.

I am not guessing here. I have spoken directly to multiple people at all of Anthropic, OpenAI, and DeepMind about unilateral pauses, and they are all sketched out by this question of the precise definition of a pause. One of those people was the CEO of one of those labs. Hiding this complexity behind “some definition” is oversimplifying the game theory, and that results in bad analysis and bad conclusions.

Now of course, unilateral pausing also means definition mismatch and multiple rounds of negotiation. It might be the case that one company pauses according to definition D1, and another company refuses to follow suit unless they strengthen to definition D2. But accepting that we live in a multiround negotiation world is one of the core arguments behind unilateral pauses: the goal is to start a process that can then ratchet up through negotiation.

Promises can be harder than actions

It is not the case that promising to do X in the future based on condition Y is strictly easier than doing X now. This is because

Incomplete information: It may be difficult to know if condition Y is true based on incomplete evidence; in this context, that means company A not entirely trusting company B’s statements that they’ve Y-stopped.
Changing circumstances: Promising to do something in the future means not revising one’s views if the situation changes. This is uncomfortable! In practice, that discomfort translates into failure to make promises to begin with.

The above are some abstract first principles. But they conform to how AI company executives speak about the issue, and how I’ve seen negotiations break down in the past: Executives are very uncomfortable making conditional promises.

As one example, DeepMind’s attempt to become a company limited by guarantee (CLG) failed in part for similar “reality fails to conform to simplistic game theory” reasons. DeepMind leadership was attempting to guard against “AI capabilities are crazy” worlds where risks are high, and the Google Board was attempting to guard against “AI fizzles and DeepMind wastes a bunch of money” worlds. Simplistically, there was an arbitrage to be made. In practice, negotiations fell through. One should be very careful when assuming that simple theorems of simple game theory apply.

How many frontier labs are there?

Another reason it is not strictly easier to do a Scott-version conditional pause is that it is not clear how many frontier labs there are. In the U.S., the number is somewhere between 2 and 6. There are clearly worlds in which getting 2 companies to do a unilateral and then bilateral action is easier than to get 6 parties to do even a much weaker action, and I claim we are in one of those worlds.

Definitional fairness is less important than survival

As mentioned, when I talk to people at AI companies, they are very concerned that if they stop, other people will stop slightly less, and it will be unfair. We can contrast the following two risks from AI:

Company A stops a bit more than company B, and it is unfair to company A.
One of company A and B trains a misaligned superintelligence which takes control of the world and eventually kills everyone.

The second risk is more important!

Now, one can argue that the first risk is important too, and indeed significant work should go into minimizing the unfairness, since that unfairness does impact pause effectiveness. Too much unfairness means we don’t slow down AI capability progress enough and don’t buy enough time. But I think that fairness is to be achieved over time, after multiple rounds of action, and should not be a precondition for taking the first step.

It may be much easier for an AI company to act on their own definition than by one imposed on them externally. So shifting the conversation to the relative unimportance of the definition used for the first action may be critical to actually causing that action to happen. This applies not just to unilateral or bilateral pauses but also to the grand international coordination that Scott prefers. In that context, it is notable that Scott’s only engagement with definitions is using the phrase “some definition”, handwaving a bit, and adding a footnote about forecaster wording.

Once we will fill in the subtleties around definitions and the details of the game theory, the correct action is to stop.

Frontpage

357

New Comment

62 comments, sorted by

top scoring

Click to highlight new comments since: Today at 8:32 PM

[-]Scott Alexander3d*8925

I'm unhappy with your characterizing me as straw-manning.

I'm responding to a real position that I have regularly heard people argue for. I was going to waste my time listing and linking various examples here, but I see that you admitted as much yourself partway into your post ("It is possible that Scott will correctly point out that some people do advocate for the extreme version here"). I think it would have been more honest for you to start this with "There are other versions of SURA that don't work this way, let me describe one", not "Scott is straw-manning".

The defense I've heard from the people who back the original version is that if (let's say) Anthropic merely said "we're going to stop training and work on safety/security/products", this would look like a potentially wise business decision, lots of people would say 'ah, guess the scaling was too expensive after all and they're using safety as an excuse', and this would not raise awareness or convince anyone they were being honest. But if Anthropic conspicuously blew itself up, this would make the front page of international news and there would be no possible explanation beyond them genuinely being afraid. I agree that your version has other advantages, but this is why I would have preferred you saying "Here's another version" rather than accusing me of strawmanning.

I think you misunderstand me when you say that I "think the good version of unilateral pausing is good". If (let's say Anthropic) stopped doing research, but (let's say) OpenAI didn't, then in a year (if we have that long), Anthropic would be mostly irrelevant, just as a company which only had GPT-5 level AI today would be irrelevant today - even the free open-weights models would be better than them. In terms of being "out of the industry", there's not an interesting difference between disbanding entirely vs. committing one's self to a policy which ensures irrelevance, except that disbanding entirely would at least be exciting and newsworthy all at once, whereas the pivot away from research would be treated as self-serving at the beginning and fail to make news a year later when it got proven honest.

The point that I was ascribing to AIFP (AIFP's real position is complex, and I don't want to say with certainty that I have it exactly right, so I'll call it Scott!AIFP) is that it's pointless to race now insofar as China will just steal the results of it anyway. Scott!AIFP leaves open the possibility that if security were better, it would be correct to race now. I don't think that's what you're saying, or if it is, I don't think you're explaining it well.

You seem to include in your good-version-of-unilateralism definition that if everyone else doesn't follow along, the company can restart ("4. Restart if necessary"). I don't think this is costless - if you're going for Plan C, the main parameter is how much lead you have, so burning the entire lead sitting around and waiting to see if other people will join you in a previously-unilateral pause is an extremely costly action, and shouldn't be taken unless you think there's a pretty good chance that it's true. This is why I brought up the claim that the AI superforecaster thinks there's only a 3% chance that it's true - and this was according to the much more dramatic version where they're willing to blow up the company over it. I think burning the entire lead for a 3% chance that this creates a pause is exactly the kind of 'burn all of our hopes for Plan C in exchange for infinitesimally higher chances of Plan A' logic that I was criticizing in the original post, and that your supposedly-better version of a unilateral pause does this exactly the same.

(also, I feel like you're failing to signpost how much you're conceding to me with "restart if necessary". It sounds like you're admitting that an actual unilateral pause would be bad or unworkable, but suggesting that companies do a sort of fakeout, where they pretend that they're doing this, but are still planning to race if the pause doesn't spread. This is an interesting bargaining technique, and I'm happy to consider it qua bargaining technique, but I don't think that it on its own justifies SURA-as-it-actually-exists, which is founded on rhetoric about how we can't bargain, can't use any game theory in any way, and are just deontologically compelled to stop right now - maybe you are in a different camp than the people who say this, which would be fine but then I wish you would clarify that rather than forcing me to confront a shapeshifter. Also, the fact that OpenAI stopped training for a month or so by some definitions when responding to Hugging Face, but this didn't grow into a broader pause, makes me think this is not a very interesting bargaining technique.)

In response to your 'making a definition' point, I will walk back "strictly easier" in the sense of "Pareto dominating" in favor of "much easier on balance of considerations", since as you point out the definition-wrangling will be a problem even for a unilateral pause (just a different sort of problem), and there is still the usual reason why getting a company to commit to mutual action is easier than asking them to harm themselves unilaterally with no reason to think others will go along.

I don't understand why you think I'm concerned with definitional fairness. I'm concerned with the fact that your plan is much less likely to work, and, in failing, makes us much less likely to survive.

[-]Geoffrey Irving3d7752

The way I see your post landing is as a general argument against unilateral pauses, not as a delineated argument against bad versions. You cite particular examples of people proposing to pause temporarily as central examples, then explicitly say that only AIFP does the good version. You may not have intended it as a general argument against unilateral pauses, but your title is "Against Stop Unilaterally and Raise Awareness", and I do think it is fair to say that your title applies to my version (and to AIFP's version).

[-]Scott Alexander3d3013

Every position includes an unlimited number of possible sub-positions. I tried to respond to the version that I thought was strongest (based on the reasons I mentioned in the "the defense I've heard..." paragraph in the comment above), but I think 90% of my arguments apply to yours as well (based on the reasons I mentioned in the "I think you misunderstand me..." and the "You seem to include..." paragraphs in the comment above), and so I don't think I've muddied the discourse by presenting it as an argument against the wider class of strategies.

If you think this is misinforming people and want me to put in a one-sentence disclaimer that you have a different version, tell me the sentence, where you want it to point to, and what you want it to link to (maybe this post?) and I'll do it.

[-]Geoffrey Irving3d40

Per your other comment we may just be talking price about length of time. One year would be too long.

I will leave the sentence to you, I'm not sure the overall epistemic effect is much changed by a single sentence.

[-]Geoffrey Irving3d40

I'm also happy to edit my post clarifying that you don't agree with what I call the good version, if you want.

[-]Geoffrey Irving3d105

On the definitional fairness point: my claim is that AI companies are concerned with it, and that you have ignored it. Grappling with the definitional fairness concern is necessary to getting the game theory right.

[-]habryka3d1711

SURA-as-it-actually-exists, which is founded on rhetoric about how we can't bargain, can't use any game theory in any way, and are just deontologically compelled to stop right now

Nobody is going around saying "you can't use any game theory in any way". This doesn't match what approximately any of the people who think you are talking about them are saying (me, MIRI, Zvi, Geoffrey).

[-]Scott Alexander3d151

This is based off of a thread of rhetoric exemplified in eg Samuel's "Instead of telling yourself that you can galaxy-brain your way through this with fancy enough game theory, maybe you’re better off just thinking of this as something much simpler: Russian roulette", which uses "game theory" in a way that implies that it's galaxy-brained and loops around to being stupid, and so you need to abandon it and instead appeal to popular wisdom / common sense.

[-]habryka3d1512

I do not think that quote implies "can't use any game theory in any way", or anything close to it. Clearly it's supposed to mean "your fancy thinking about game theory in this case is leading you astray and does not leave you obviously better off than this simple analogy".

Do I agree with that? No, seems anti-intellectual and dismissive of thought, but still not appropriate to summarize with "the strategy of Stop-unilaterally-and-raise-awareness is founded on rhetoric about how we can't use game theory in any way". Certainly Samuel is obviously not the first person to argue for a unilateral pause, and are you telling me the people in our ecosystem who did used the same rhetoric?

Also, I don't think your post is conveying to anyone that you are primarily arguing against Samuel, especially given this quote, where you say there is only a single group that has called for "an immediate unilateral-ish pause in a way that makes sense":

I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully

I have never heard of Samuel before that one article, and he would be a kind of wild character to use as a representative here. He certainly doesn't have much sway in the alignment community. Clearly you are arguing against MIRI, and me, and Zvi and others like that in your post. I am happy to dig up like 15 quotes evidencing this, but I don't currently believe this is actually under dispute?

[-]Scott Alexander3d15-8

I'm claiming that it is a typical move of various people I associate with SURA to condemn the application of game theory to this question as "galaxy-brained" or as preemptively failing deontological obligations. My guess is that you, MIRI, etc have said things I would associate with this position, but since I had lots of quotes from Samuel's article in my post already, I used hers. She is a journalist, and her article was a rare article-length defense of SURA in a major media publication, whereas I think you, MIRI, etc mostly just treat it as a background premise in your discussion of other topics rather than defending it explicitly. If our crux is that you think you and MIRI have never condemned the application of game theory to this problem, I can try to look for tweets or something disproving this, although it would suffer from the usual difficulty of trying to find tweets months after the fact.

EDIT: Tried to search for tweets preemptively, it took me ten seconds to find this one by Robby last week: https://x.com/robbensinger/status/2115927691014906014 . I accept that you could argue that he only said things about "classical" game theory and is leaving open that there's some higher-level game theory which is good, but I think that across lots of statements by lots of people whose emotional content is treating game theory in a negative way, I'm allowed to notice this pattern even if they don't use the specific phrase "I hate game theory" in an ironclad undeniable way.

[-]habryka3d3021

If our crux is that you think you and MIRI have never condemned the application of game theory to this problem, I can try to look for tweets or something disproving this, although it would suffer from the usual difficulty of trying to find tweets months after the fact.

Yes, this would be absurd by my lights. Of course I do not condemn the application of game theory in full generality. This is like condemning the application of "math" in full generality to a problem. It would be extremely alien to me, and I am confident would be similarly alienating to Robby or Nate or Eliezer.

I accept that you could argue that he only said things about "classical" game theory and is leaving open that there's some higher-level game theory which is good

Yes, obviously that is what he is saying. That's why he has the word "classical" in there. Much of game theory is dumb and says to defect in a twin prisoner's dilemma. This is like quoting a physicist saying "many people just get really confused by the orbital theory of atoms which repeatedly leads them astray" as evidence that the physicist is generally against "the application of physics to problems in full generality". Or a mathematician complaining about a wrong proof as evidence that the mathematician is "against using math in any way".

Yes, much of game theory is bad. Much of AI alignment research is bad. And much of physics is bad. This is a domain where many naive applications of game theory are particularly bad, but also a domain where good applications of game theory are extremely valuable and the source of many of the best models.

I'm allowed to notice this pattern

Of course you're allowed to notice this pattern. But you are not allowed to summarize it as "can't use any game theory in any way". That's just a terrible summary.

[-]Scott Alexander2d234

Yes, obviously that is what he is saying. That's why he has the word "classical" in there. Much of game theory is dumb and says to defect in a twin prisoner's dilemma.

This doesn't seem relevant to me, since AI companies aren't each other's twins. I think game theory is a useful analysis of what race dynamics are, why they exist at all, and why they can't be solved by just pressing "cooperate". But game theory then goes into great depth about other options besides just defecting, like defusing race dynamics by negotiating multilateral cooperation with commitment mechanisms, which is my whole point.

Of course you're allowed to notice this pattern. But you are not allowed to summarize it as "can't use any game theory in any way". That's just a terrible summary.

To give more examples: "the monster in our heads" is "the rationalist winner-takes-all/hyper-optimiser/game theoretic mindset" which is why we haven't stopped unilaterally yet. We are abandoning "common-sense intuitions about cooperation" for "flawed game theoretic logic". Anthropic is the "locus of (complicated, incoherent, indefensible) justifications for accelerating AI capabilities even though AI might eat the world . Much ink could be spilled, but my childish listening just hears 'wah wah wah wah do more AI R&D'."

I admit I cannot produce a signed affadavit where someone says "I will lower your status if you mention game theory", but that's not how this works. When wokeness was taking over, there was (at first) nobody who said "I propose lowering the status of everyone who does social science investigating cultural or economic or biological factors that compete with racialized explanations". There were just a lot of people saying "sure, you use lots of fancy words and complicated models, but everyone knows those have been used to justify racism in the past, and in the end all of your big words cash out to wah wah wah racism isn't real, and surely that's not what you meant to say, so instead you should defer to the lived experience of common-sense-having people like me", and a lot of other people cheering on those people every time they said it, and eventually the message became clear that certain fields were enemy soldiers.

I wrote the post because the SURA situation is pinging my same "uh oh, they're constructing the superweapon against me, better protest before it's finished" module that wokeness did in 2014, and to have any hope of winning you have to start protesting before the stage where people say outright "yes, here is my signed affadavit where I admit that arguments are definitely soldiers and we will openly retaliate against you for using them" (which, if the last decade is any guide, comes about five years later).

[-]habryka2d106

Much of game theory is dumb and says to defect in a twin prisoner's dilemma.
This doesn't seem relevant to me, since AI companies aren't each other's twins.

I am just using this as an example of a random thing that classical game theory gets wrong.

But game theory then goes into great depth about other options besides just defecting, like defusing race dynamics by negotiating multilateral cooperation with commitment mechanisms, which is my whole point.

Yes, I agree. I don't quite know what you are saying here. I think game theory is great to study, I continue to think a lot about it, and I have recommended Schelling's "Strategy of Conflict" like 3 times in the last few months to someone to read. My point is that there is some bad game theory, and me or others saying that some game theory is bad is of course not great evidence that I think you should never use game theory to analyze a situation.

To give more examples: "the monster in our heads" is "the rationalist winner-takes-all/hyper-optimiser/game theoretic mindset" which is why we haven't stopped unilaterally yet. We are abandoning "common-sense intuitions about cooperation" for "flawed game theoretic logic". Anthropic is the "locus of (complicated, incoherent, indefensible) justifications for accelerating AI capabilities even though AI might eat the world . Much ink could be spilled, but my childish listening just hears 'wah wah wah wah do more AI R&D'."

IDK man, these seem like weak examples. I am here right now defending the use of game theory. My giant response to your question is full of game theory. Eliezer has written like three dozen posts and a giant glowfic filled with game theory. Game theory discussion is like one of the favorite pastimes of like half of the people you are talking about.

I do think classical game theory is often quite dumb. Indeed, having detailed opinions about which strains of game theory are good and bad seems if anything highly correlated with being an active practitioner in the field.

There is a strain of game theory and decision theory that does make pretty catastrophic recommendations, and I think we both agree on that (unless you recently changed your mind on the Newcomb's problem). We both believe in a more sophisticated version of game theory and decision theory, and apply it all the time in almost all of our analysis in situations like this.

I think it's fine for you to express a concern here, but again, summarizing it as "these people's beliefs are founded on the belief that you can't use game theory in any way" is just wild. It's fine to say an occasional wrong thing in a giant comment. I am sure I said some wrong things in the giant treatises I have written in the past few days. You can just say "ah yeah, I agree this doesn't apply very well to you, Eliezer and Robby and saying 'founding' is an exaggeration, but I think there is a real social dynamic here". But you seem to be really digging your heels in on this what seems to me obvious point and I don't know what's going on.

[-]Scott Alexander2d266

I don't think Newcomb's problem has anything to do with the AI race, and I don't understand why you are responding to my discussion of people attacking the use of game theory in the AI race with an appeal to game theory being wrong on Newcomb's problem. I understand these are random examples, but I was bringing this up for a non-random reason, and if the only thing wrong with game theory is twin dilemmas, Newcomb, etc, then I don't think that explains why people are criticizing using it to understand the AI race (eg Robby's tweet, which I think is still what we're discussing, was claiming it "underlies Anthropic's cynicism" about pausing).

(I don't even think Newcomb's is game theory, isn't decision theory different? And I don't think classical decision theory is wrong on Newcomb's so much as that Newcomb's shows that the single question that classical theory thought that it was answering is in fact two different questions, with the classical theories giving the correct answer to one and the rationalist theories giving the correct answer to the other. In any case, this is even less relevant than the original example.)

I think that you, Eliezer, Robby, etc are smart and well-intentioned people, good at thinking about game theory, and would disagree with the stupidest 95 out of 100 operationalizations of "trying to forbid people from using game theory to think about the AI race" (while being neutral on some of the other five and endorsing one or two of the smartest). I do think you are unintentionally using language in ways that maybe supports the emergent phenomenon of some of the worse operationalizations (eg talking about game theory more dismissively than you would in reflective equilibrium in a context where other people are asserting bad operationalizations of trying to discourage others from thinking about game theory in the context of the AI race). I don't think your opinions are founded on this, but I think the thing I'm complaining about is partly founded on it, where the thing I'm complaining about is an anti-epistemology which I would gloss as "trying to think about this problem in too much detail is galaxy-brained and we should retreat to one specific version of common-sense deontological intuitions instead".

[-]habryka2d*7-2

I understand these are random examples, but I was bringing this up for a non-random reason, and if the only thing wrong with game theory is twin dilemmas, Newcomb, etc, then I don't think that explains why people talk about it being a problem when thinking about the AI race.

I am confused. As far as I know, I myself brought up the twin prisoner's dilemma (and Newcomb's), and I chose it as an example of some random thing being wrong in a part of game theory, but how me saying this of course doesn't imply all of game theory is bad. Are you saying you brought up the twin prisoner's dilemma for some other reason?

I could go into details on why I think certain game theoretical arguments (especially of the classical variety, though also various fancy more TDT-ish flavored arguments) are wrong and produce wrong intuitions about things like AI arms races, if you want, but I feel like that would be a digression (but I am generally happy to go into it if you want). But I was not choosing the twin PD as something that was supposed to have much relevance to that, I was using it as an example of a common argument that you presumably also think is false, and how this of course doesn't mean that you think that game theory should never be applied in any way.

where the thing I'm complaining about is an anti-epistemology which I would gloss as "trying to think about this problem in too much detail is galaxy-brained and we should retreat to one specific version of common-sense deontological intuitions instead".

Yeah, I mean, I have sympathy for this, in as much as I agree that I often want to criticize broader social forces that have underlying structure and intentionality without any specific person being as dumb or malicious as the headless social forces causing harm. But I do think I want to have lines somewhere around things that are as clearly going to be interpreted as ascribing beliefs to specific people, and not emergent belief-like attributes of a social egregore, and I think your sentence (and associated sections of your post) crosses that line (and IMO not by a thin margin).

If you wanted to communicate this aspect, I think there is a post you could write that focuses on a set of emerging phenomena and explicitly highlights the contrast between what individuals believe and what the resulting social egregore seems to act according to. But your post doesn't do almost any of that. It seems very interested in understanding what the people actually believe, and seems interested in arguing with the strongest representatives of the relevant arguments. I understood SURA to be a pointer towards a set of beliefs that you thought people held in broadly good faith, but were wrong about, not a set of emergent beliefs that actually very few people responsible for propagating them actually believed in. I could dissect your post and show you which parts make me confidently think that, but I guess before I do that I want to check whether you are now claiming that this is how I am supposed to interpret your post.

[-]307th2d11-6

Of course you're allowed to notice this pattern. But you are not allowed to summarize it as "can't use any game theory in any way". That's just a terrible summary.

I don't really get why you're litigating this so hard. I think the way he phrased it was a hostile paraphrase intended to make a point, in a way that is pretty common from everyone involved when arguing persuasively. E.g. here's a random quote I pulled from one of your previous posts:

The central argument in all the cases seems to me to be "trust me bro, it will be great when I, or people like me, have the Superintelligence and terrible if anyone else does, and I promise that I will be stopping just before the brink bro, please don't try to stop me, I am one of the good guys".

This is not a remotely accurate description of Scott's position. Are you "allowed" to summarize Scott's position this way? I would say yes - it is clear this is your hostile paraphrase, and although I think your paraphrase reveals that you aren't fully understanding Scott's position^[1], in some sense this is to the credit of the hostile paraphrase paradigm - it quickly gets to the point of disagreement.

More generally, when you argue, well, you often make a lot of great points and are good faith, but on the negative side I feel like you sometimes zero in on some small point and insist on framing it in a way that feels persuasive but is in fact missing the bigger picture. Overall I think you're too unwilling to concede ground.

Like here, apparently people find you quite convincing, but it seems like you're kind of just wrong in that Scott's original passage was a fine slightly exaggerated paraphrase and you're hammering on that slight inaccuracy effectively while missing that everyone involved make those types of exaggerations all the time.

(Well, to be accurate, most people make these types of exaggerations some of the time.)

^{^}
I think risk-awareness is the primarily quality Scott would like those building superalignment to have, not that the people building it be like him.

[-]Vaniver2d1412

I don't really get why you're litigating this so hard. I think the way he phrased it was a hostile paraphrase intended to make a point

Didn't Scott write an article about this?

[-]307th2d17-3

Scott's article focused on a specific phenomenon that isn't actually what's happening here (no one here is pulling a "how dare you defend SURA"), and has various caveats like "I’m sympathetic to making statements that are not-technically-true for didactic, and artistic reasons, eg readability".

It seems like a weak gotcha to link this article without explaining e.g. why you think this is a criticism that applies to Scott, whether that same criticism applies to Habryka's passage, etc.

[-]Jiro2d42

Scott's caveats were:

Sensitivity to victims. This is advocacy, not victims being upset.
"didactic, and artistic reasons*. I think that advocacy excludes this. Yes, you could say that any advocacy is didactic. I don't think that's what it should mean.
Frog boiling--use as a metaphor. Not applicable here.

And the fourth one isn't a caveat.

[-]habryka2d78

In my quote I am giving a summary of what seem to me the crucial shared attributes between Dario, Elon and Sam Altman. I am giving an exaggerated version for rhetorical affect, but I am not actually saying anything false. Scott literally says that the plan he wants Anthropic to pursue is to stop right at the brink, and does not provide any mechanism of verification for how we can know in advance that they will stop at the brink.

I think the part of my summary that would be fine to criticize is the sentence "please don't try to stop me". Scott is saying that we should try to stop Dario, Elon and Sam, but only if we can stop all three, and also China at the same time, not if we can only slow down the leading actor (who in Scott's plan is Dario). I think it would be OK to criticize that part as potentially misleading, though I think there is a simple clarification that would make it solidly correct.

in some sense this is to the credit of the hostile paraphrase paradigm - it quickly gets to the point of disagreement.

I mean, this paradigm only works if you tell the other party if they get it wrong. Which I am doing here. Are you saying Scott should be allowed to give hostile paraphrases but then I need to be polite when I point out how they are inaccurate, because it was to Scott's credit that he used hostile paraphrases? Scott of course should point out how my exaggerated paraphrase is wrong. I stand behind it as written, and am open to arguments against it. Scott's seems wrong by the same standard, and I am open to arguments against that too. I disagree with your footnote how mine is wrong, and now we can argue about it.^[1]

Summarizing at least mine, Eliezer's or Robby's position as "you can't use game theory in any way" is just really not anywhere close to what is true. Mine, Eliezer's and Robby's position are, to the best of my knowledge, all downstream of us thinking quite a bit about the game theory, and we've all written many long pieces about how we think game theory should be applied to this situation. It's as if I was exaggeratedly summarizing Scott's piece as "and Scott's plan is founded on the belief that none of the labs are allowed to talk to each other". You could make some extremely thin argument that this position is implied by plan C being the result of coordination breaking down, but it would be immediately falsified by reading Scott's summary of plan C which involves a lot of labs talking to each other and forming coalitions of some kind.

To be clear, this is not an enormously important point, though it's representative of something like 15-20 places in Scott's post where he makes similarly clearly false summaries. Like when he summarizes that (cut together from two sections) "the community consensus from the beginning of the community was to believe in Plan C, which is to use generic capabilities without a deep understanding of alignment to build a system just on the brink of superintelligence and then use that to solve alignment in a mad 6-12 month dash, until 2023 when ChatGPT caused basically everyone to abandon this plan and memory hole that it was the default thing everyone was trying to do since the beginning of the community". That summary is worse, but there are just a lot of places where Scott says straightforwardly false things about what the people he is arguing against believe.

^{^}
I don't think Scott wants "risk-awareness" as the primary attribute. Sam Altman is very aware of the risks, and Scott repeatedly emphasizes how important an Anthropic lead over OpenAI is. Scott's plan of course requires the people in the lead to use the 6-12 months of potential lead time they can burn at the end for the kind of alignment research that Scott thinks has any chance of building aligned Superintelligence. He doesn't go into a ton of detail on what that kind of research would be, but the primary term he uses is "alignment-pilled".

[-]cousin_it2d50

Much of game theory is dumb and says to defect in a twin prisoner’s dilemma.

There's a lot of such "outsider criticism" of game theory and economics. Most of it doesn't hold up.

To address this example: the textbook from which I learned game theory was careful to say that the theory is in large part descriptive. In real world situations that look like the PD, behavior will move toward Nash equilibrium play. The players don't even have to be rational thinking agents, they can be bacteria or plants. In some situations the players can avoid (D,D) and ensure (C,C) if they have an enforcement mechanism. Twin symmetry is one such mechanism. Any reasonable game theorist will say sure, it can work, but there are also reasons why this particular mechanism is rare in practice and others are used more.

Another similar example is "economists care only about efficiency", when actually the equity-efficiency tradeoff is in the first pages of textbooks.

[-]Karl Krueger2d52

Heck, if economists only cared about efficiency, they would one-box. It's half as many boxes to carry.

2habryka2d

@Claude Opus 5.5 do most economists think it's rational to defect in a twin prisoner's dilemma?

5Claude Opus2d

There's no survey of economists on this specifically, but the standard framework and its most prominent defenders say defect. * Textbook game theory models players as choosing independently. Defection strictly dominates in the one-shot PD, so the textbook answer is defect, and that holds even against a copy of yourself. The correlation that makes twin cooperation look rational isn't represented in the model at all. * Ken Binmore, probably the game theorist who has written most about this case, calls the cooperate-with-your-twin argument the "twins fallacy" and argues against it at length (Game Theory and the Social Contract, Playing for Real). That's a mainstream normative claim, not just a descriptive one. * The closest data point is the 2020 PhilPapers survey on Newcomb's problem: about 39% of philosophers favored two-boxing and 31% one-boxing, and two-boxing has historically been more popular among decision theorists. Cooperating with your twin is basically one-boxing. cousin_it's framing, where twin symmetry is a legitimate enforcement mechanism, is a reasonable position. But Hofstadter, Gauthier and the FDT/UDT crowd are the ones who argued for it, not the mainstream of economics. So habryka's claim is roughly right as a description of standard theory, though most working economists have probably never taken a position on it.

4cousin_it2d

I actually learned game theory from Binmore's textbook. So when Opus now said "Binmore says defect in twin PD", I got startled for a minute. Then I went and checked. Here's a full chapter from his "Game theory: a very short introduction": [...] As you can see, he says cooperate in the literal twins case. Always check after LLMs! That said, I agree with Opus that economists haven't cared that much about the perfectly correlated twins case, because it hasn't had much real applications so far. I only got interested in this case because Eliezer's writings made me wonder about cooperation between future AIs. (My first step was reinventing Tennenholtz's program equilibrium, which is well accepted academically and supports cooperation in twin PD.)

2habryka2d

This seems like it says the opposite of what you are saying? It's saying directly that Bob will choose defect if he is rational. Indeed, he says that the very premise of the game is made impossible by Bob's rationality, because defection is the only rational choice. I think you are just summarizing this section incorrectly. Claude also says: [...] And indeed, this is the only way I can make sense of this passage. I think the last sentence genuinely is just flipped and the author made a typo. "Dove is strongly dominated" is the only justified claim you could make in a classical game theory context. Hawk is definitely not strongly dominated, or at least it would require perfect correlation (in which case I agree we are not playing the twin DP at all anymore). I think you ended up misreading this section because of a typo.

2cousin_it2d

Maybe we're talking about different things. 1. Twins fallacy, as described by Binmore: "if two players are both rational, that in itself will make their decisions correlated, therefore it's rational to cooperate". 2. Twins PD, as studied on LW (by me for example): "if the players know that their decisions will be correlated - for example, by knowledge that their source code is identical - then it's rational to cooperate". I've been talking about (2), and my read of the chapter is that Binmore agrees with cooperating in (2). He spends his time railing against (1), and I'm with him on that. If you support cooperating in (1), I think that's wrong.

2habryka2d

No, I think Binmore is directly saying that if one of the players in the twin PD is rational, and the correlation is not literally perfect (in which case it becomes a one player game), then the two players should defect. That's what the sentence "This can’t be right, because Bob is supposedly rational and one of his two choices is irrational" is about. I am saying he literally made a typo and wrote the wrong words on the final lines. The sentence "hawk is strongly dominated" doesn't make any sense in a classical PD context. Yes, this sounds wild, but I really have been staring at this section for the last 10 minutes and cross-referencing it with other things Binmore said, and I am like 85% that I am right. He meant to say "Even if Alice and Bob’s information were correlated, as hypothesized in Aumann’s notion of a correlated equilibrium, they still wouldn’t play dove, because dove is strongly dominated whatever the players may learn about other matters."

2cousin_it2d

Here's my read: * The second-to-last sentence says "if there's correlation between the player's choices, it isn't PD". Elsewhere he calls it "Prisoner's Delight" and agrees with cooperation. Which is what I've been saying. * The last sentence talks about a different situation: "If there's correlation between the players' information, then play hawk". Indeed it has a typo. But it doesn't matter to the rest of the chapter! That last sentence is there only to make the technical point that correlated equilibrium doesn't save you in PD. It doesn't refer to the same situation as the second-to-last sentence.

2habryka2d

I mean, sure, then your initial summary is still wrong. He is not saying you would play (dove, dove) in the twin PD without perfect correlation (indeed he seems to say that such a game would be impossible if either of the players act rationally). The perfect correlation situation is just a bit silly and non-practical, so it doesn't really matter. I don't have access to other sources by him (all paywalled), but I'll see whether I can find a clearer treatment. He certainly argues that you should defect in all situations where your decisions are "independent" (whatever that means).

4habryka2d

Yep, as far as I can tell in "Playing Fair" he says pretty explicitly that in any realistic twin PD, both players should defect when rational. In order for the twin PD to suggest dove, dove, the correlations would need to be utterly perfect to a basically logically impossible degree (and he spends many pages belaboring this point). Source: https://res.cloudinary.com/lesswrong-2-0/image/upload/v1793728611/Game_Theory_and_the_Social_Contract__Playing_fair_K._G._Binmore__KEN_AUTOR_BINMORE__Emeritus_Professor_of_Economics_Ken_Binmore_MIT_Press_93505369_libgen.li_ghvbjd.pdf His writing is really not very clear, so I might still be confused, but I feel pretty solid in betting on what he would say if directly confronted with basically any realistic scenario.

4cousin_it2d

I hadn't read that book before, and yeah, the writing isn't very clear. That said, can you point me to the part where he handles twin PD with high correlation? I can't find it. I was able to find a different argument (pages 213-214): that tiny differences in thinking will diverge and lead to low correlation, unless the structure of the game favors them converging. Which sounds right to me.

2habryka2d

I am understanding this section to be saying "in order to end up with a dove, dove equilibrium the correlation between the two decision-making processes would need to substantially exceed 1/2, but that is impossible in-practice due to this reasoning that I am describing, and so we will end up with a hawk, hawk equilibrum": [...] This is all in the context of his introduction to the PD where he says: [...] I really feel like these two together imply the thing I am saying here, but who knows, his writing is not very clear.

2cousin_it2d

The imperfect correlation PD is harder to formalize than you think. I've carefully avoided saying too much about it in all my time on LW, including this thread :-) Binmore also wisely doesn't say much about this case, except saying it has a lot of "dust" and "isn't PD" - both of which I agree with. Anyway I have to run now and the thread is getting deep. Happy to continue later but also happy to stop.

2habryka2d

Oh boy do I wish that was true. I just read 20 pages of really very mediocre essays on this by him in the book I posted above.

[-]RobertM3d2121

I am fairly confident that tweet is disparaging "naive utilitarian CDT"-type "game theory", rather than game theory in full generality. @Rob Bensinger, can you confirm?

[-]Scott Alexander3d6018

Trying to narrow down / operationalize my "interesting bargaining technique" objection from the other comment:

Suppose that Anthropic were to announce that they were pausing all research and training for two months, they would spend those two months trying as hard as possible to get other labs to join in, and, if it didn't work, after two months they would start racing again. Would this remove your objections to their work and make you agree that they were pursuing a good and moral course of action?

If we could pinpoint the probabilities that:

...this would work and lead to a larger pause
...this would fail, and losing two months of lead time would move us from a Plan C world to a Plan D world or otherwise spoil a last-ditch alignment push

...would you agree that 1 < 2 would be near-conclusive evidence that I'm right, and 2 < 1 would be near-conclusive evidence that you're right?

[-]Geoffrey Irving3d223

Suppose that Anthropic were to announce that they were pausing all research and training for two months, they would spend those two months trying as hard as possible to get other labs to join in, and, if it didn't work, after two months they would start racing again. Would this remove your objections to their work and make you agree that they were pursuing a good and moral course of action?

Yes, this would be a good and moral course of action. I think 1 > 2.

(As a nit it can't be the case that as probabilities change continuously the situation discontinuously flips to one person being near-conclusively right to them being near-conclusively wrong, but I think this wording is just a typo on your part.)

[-]Scott Alexander3d90

Sorry, I got the signs of the inequalities flipped, as you seem to have noticed (and have edited them to make things clearer to anyone else reading this thread). Based on the version I intended, do you think this is our main/only crux?

[-]Geoffrey Irving3d13-1

Yes, I think this is the main crux. I don't think it's the only crux: there is a lot of detail which we also seem to disagree on.

[-]Scott Alexander3d*7042

Okay. Well, I don't want to say that it's impossible that it would work as a bargaining technique, and I'm not dead-set against it, but my best guess as to what would happen is:

The media and e/acc say it's a ploy to safetywash the fact that they're low on compute, or got a crappy pretrain, or want to focus on making money instead of progress, or something like that.
OpenAI and GDM would say that they'll look into antitrust implications and get back to them.
Several other companies and NVIDIA would proudly declare that they will never join in a pause and this is antithetical to progress and Anthropic is just trying to something something regulatory capture. OpenAI and GDM would notice this and be pessimistic about the pause spreading.
Some mid-level Trump administration official would make a statement saying this is unpatriotic pro-China treason. Nobody would know whether this was official government policy.
China would give minimal sign of noticing. Some smart China watcher would blog about how some line in a Party newspaper might be a coded reference to this event, but nobody would be sure what it signified.
Two months later, OpenAI and GDM would say that they're still looking over the antitrust implications (whereas in fact they're pessimistic and don't want to piss off Trump and Jensen).
Anthropic would start up again right on schedule. The safety movement would hate them more than ever for betraying the pause. It would be front page news that they had claimed they were so scared of AI that they were going to stop, but now apparently they're so non-scared that they're going back to racing, so they must be liars. Someone would come up with a story for how their low compute or crappy pretrain got resolved. Everyone who took them seriously or tried to work with them would have egg on their face.
They would go from a ~1 month lead over OpenAI to a ~1 month deficit, which might or might not be recoverable this far into RSI.
Future pause negotiations would be harder, because conventional wisdom among half the people involved would that this was a weird regulatory capture trick, and among the other half it would be that Anthropic tried and didn't get it and that proves it's impossible.

Maybe I'm too cynical, but even so, all of this honestly sounds a lot smarter and better thought-out than the view I was responding to in my post, and I have much less objection to it.

[-]anaguma2d85

I'm curious if @Geoffrey Irving agrees with this forecast, or otherwise if he could describe some alternative more likely way things play out which results in a longer pause.

[-]Scott Alexander3d221

@habryka @Rob Bensinger @Zvi , just to make sure I'm not accidentally mischaracterizing you, do you agree with Geoffrey's position above (I'm nervous summarizing, and you should double-check to make sure I'm getting it right, but I'm understanding it as saying that the moral thing for companies to do would be to trial balloon a brief pause, maybe a couple of months, to see if other companies join in, but if not, they should continue racing)? Or do you think companies which are trying to be moral should stop racing permanently in a non-trial balloon way?

[-]habryka3d*9350

My position (condensed into a few paragraphs, though of course I believe much more on this topic):

The moral thing to do for a company is to spend a large fraction of their resources on slowing down everyone equally, while also shouting desperately from the rooftops about the risks and the need for a pause (substantially above the minimum floor of what Anthropic did in the DoD dispute, and what Uber did in various municipalities to use its platform to lobby for laws that enable ride-sharing, which are inadequate but reasonable illustration what trying at all might look like).

The right commitment level for something like a pause is tricky to choose, and different commitments generalize differently. Simpler commitments are easier to verify, and irreversible commitments produce more stable equilibria. Of course more complicated commitments maintain more flexibility and are cheaper, and so are more likely to be followed by other actors if they don't agree on the importance of the thing the commitment is trying to achieve.

It's easier to verify that your company is genuinely out of the race than whether your company is pausing but in a way where it could restart any time. The default assumption in the latter situation will be that you are not pausing and are just continuing research covertly or in a way that is conceptually confusing and maybe in the letter but not the spirit of the norm you are trying to propagate.^[1] This means setting a norm to irrevocably stop is probably more likely to generalize by being much easier to enforce (though of course less likely on base-rates). I personally don't really see much of any difference between OpenAI having a 6 month lead or Anthropic having a 6 month lead, so the expected likelihood of any current actors at the frontier actually successfully stopping at the brink of superintelligence and solving alignment in a mad dash is the same, and so from an impartial norm perspective, it seems better for any actor to just leave the race and take the wins of easier verification and a bit more time for other things to catch up.^[2]

Even better than stopping unilaterally is to try to take as much of the industry down with you. The best way to go out is to spend all your influence and cash and talent on actions that slow down everyone. Ideally by lobbying for regulation that just slows down everyone, but there are also other ways of e.g. trying to hire away key staff and broadly increasing anti-AI sentiment with candid communication about the risk, etc. This also generalizes better since you need a smaller fraction of the industry to do the same in order to get a complete pause (simplified, if any company could spend its resources to take down one other company the same size, you only need half of the companies to join your efforts, and in reality the balance here seems more favored towards offense than 1:1).

From a nation-state perspective, the situation seems a bit clearer to me. The U.S. could halt global AI progress unilaterally, and basically nothing in its enforcement apparatus relies on being ahead in AI. The U.S. should unilaterally stop, then use conventional methods to force other countries to do the same.^[3]

If that somehow fails, which seems unlikely^[4] given how extremely compute-scaling driven progress has been and how easy global compute input regulation is, but not impossible, then the U.S. should probably restart AI development domestically. If you're really backed into a corner and left with no other choice, be loud, tell everyone that they are fucking insane and going to kill us all, then pick up the gun again and try to somehow flip the game board by being ahead.

But again, verification is tricky and path-dependence is important. The U.S. smashing its GPUs and smashing the global semiconductor supply chain is a much more verifiable signal that it's actually no longer racing than anything else. Given that the probability of successfully slowing down the rest of the world via conventional escalation seems quite high to me, the benefits of choosing an easier-to-verify and less fragile equilibrium are great. So probably the U.S. should have chosen a path of a hard-to-reverse pause (probably not literally smashing the GPUs, but more just a bill which clearly binds the executive's hands and which can only be changed via a highly transparent process like congressional debates and a replacement bill).

And then, the fact that nation-state coordination seems better-placed to handle this situation than inter-company coordination, due to the offense dominance of current conventional techniques for retarding AI progress, makes marginal slowdown on the company level much more valuable. The U.S. does not need to be leading the race to stop other actors from building powerful AI systems. Every week the world is waking up more to the threat of AI, which increases the likelihood of appropriate action here, and a U.S. that is behind as a result of domestic regulation and companies choosing to go slower is if anything more likely to turn into an actor that will try to slow down global AI progress, because it is no longer tempted by the (largely false) fruits of potentially "winning". This means the cost of the U.S. falling behind globally is smaller because the "race" is not denominated in AI progress, it's instead substantially determined by which of the few existing superpowers is willing to sabotage the others with conventional escalation, with extremely fragile global semiconductor supply chains which are vulnerable to sabotage by a large number of different actors.

My current best guess is that the world would be better off if the U.S. was behind in the global race due to dynamics like this, since it would spur the U.S. to action to drive a global slowdown^[5], for which it is by far the best positioned actor in the world (and it would be very marginally, but almost imperceptibly better if Anthropic was leading in the U.S. over other companies in a way that would make that consideration largely irrelevant for plans here).

And then also, at the end of the day, you should not build doomsday devices. Even if in the end, all things considered you will choose to race towards ASI again, you should do so while accepting great censure and shame, and be ready to go to jail and stand trial for your crimes if we do somehow survive. You should be shown leniency, and possibly even redemption in the history books, but ultimately still face some substantial punishment (while maybe still netting in the green). Not threatening everyone's lives is important, and liability law and reckless endangerment do not provide an escape hatch of "but like 5 other companies were going to recklessly endangering people if I wasn't going to and I was recklessly endangering people for the greater good", and neither do common-sense principles of justice.

So if you do it, and it's the right thing to do, you should still be shunned for it as you cross the line, and in the meantime you can wipe your tears with the billions of dollars you will make along the way.

I like Zvi's coverage of this in his response to you. The point of deontological prohibitions of this kind, in mine and his book, is not to create lines you can never cross. It's to create lines whose price you have to pay no matter your reason. There is a finite, if extremely large, social cost to building a doomsday device, which needs to be paid as you cross it no matter your reason. And that finite cost can be overcome, and a sinner of such acts could be redeemed if the history books play out the way you are predicting they will, but they shouldn't be absolved in advance.

^{^}
The OpenAI "pause" you keep citing seems like the perfect example of this. I don't believe OpenAI's actions meaningfully set back its capabilities at all, and I have no way to verify this "supposed" pause. Of course if OpenAI had actually given up its GPU contracts or credibly stopped some other way, then I could verify that, but I can't and now people like you keep saying that OpenAI "paused". I don't believe it and of course none of the other actors believe it. Short pauses are hard to verify and less stable!
^{^}
There exist potential organizations that have demonstrated enough competence and understanding of the alignment problem that I would see substantial value in them having a lead, but this seems very hard to achieve given current pressures on frontier companies, and any such company, in order to pursue a plan where it tries to pull ahead, would need to get much wider buy-in than from just me, to get my support in racing ahead and the associated costs to more generalizable norms
^{^}
Similarly, other countries including middle powers should stop and then threaten escalation using the tools they have to cause things to slow down (e.g. the Netherlands should use their leverage over ASML to slow things down, and the European union should threaten sabotage of foreign datacenters, and god forbid, if there is truly no other choice and all other options have been exhausted, Europe should invoke the threat of nuclear weapons)
^{^}
At least right now. As we delay actions like this we could end up in a more software-progress dominated world where such actions become harder. This mostly just makes all plans harder and is a reason for why the right time to pause is now, and why marginal time bought right now via unilateral slowdowns is extremely valuable, as the U.S. government seems to become much more likely to pursue such strategies with every passing month.
^{^}
I give an example of such a scenario here: https://www.lesswrong.com/posts/xikGZ8fuhdG6DApAo/against-sura?commentId=eytaDPdqvixir5mjF

[-]Lukas Finnveden2d50

This comment helps me understand your views a lot, thanks.

This also generalizes better since you need a smaller fraction of the industry to do the same in order to get a complete pause (simplified, if any company could spend its resources to take down one other company the same size, you only need half of the companies to join your efforts, and in reality the balance here seems more favored towards offense than 1:1).

This part is surprising to me. (And maybe a crux for some important questions?) I thought it'd be way worse than 1:1. Especially because racing nets you more money and resources whereas pausing and trying to slow everything down does the opposite. (Off the top of my head I'm not even sure if OpenAI or Anthropic would have any cash on net if they swore off the frontier and lost their ability to raise money, or if they'd have more liabilities than assets, especially if investors can sue for their money back.) And the tools for slowing things down just don't seem that good other than regulation. Maybe this just comes down to whether less than half the AI industry could pass a bill that shuts down the rest of the AI industry?

[-]habryka2d50

I simplified everything here to a very rough single ratio, but of course in-practice there will be complicated time-series dynamics here, where you have many different strategies to choose from, and some strategies might burn all your current competitors, but then new ones will arise, and you have to think about how quickly that happens, etc.

It seems likely to me that if 50% of the capital-weighted AI industry started spending most of their political, financial and social capital on getting the whole industry shut down for a few years, they would probably succeed. This is not obvious and I do think pretty tricky to think about, but it suggests to me the ratio is better than 1:1.

This is also the kind of strategy where I would be most sympathetic to an actor staying in the race. If a leading actor is actually using all the surplus resources they are getting from the race on causing everyone to slow down, in a continuous and hence hard to fake manner, then I think there is a pretty decent chance they should stay in the race. I haven't thought through this case that much, and I think them staying in the race would make the exchange rate more expensive due to various hypocrisy dynamics, but I am like 50/50 on whether it could end up being the right choice for an actor to stay at the frontier while also spending a substantial chunk of their resources on this.

This means I am not just considering strategies that involve burning all your surplus now, I think a strategy of "stay in the race to maintain competitiveness but sacrifice much of your counterfactual lead for slowing everyone down" is also part of where a ratio better than 1:1 becomes more plausible.

[-]nam_2d30

And then, the fact that nation-state coordination seems better-placed to handle this situation than inter-company coordination, due to the offense dominance of current conventional techniques for retarding AI progress, makes marginal slowdown on the company level much more valuable. The U.S. does not need to be leading the race to stop other actors from building powerful AI systems. Every week the world is waking up more to the threat of AI, which increases the likelihood of appropriate action here, and a U.S. that is behind as a result of domestic regulation and companies choosing to go slower is if anything more likely to turn into an actor that will try to slow down global AI progress, because it is no longer tempted by the (largely false) fruits of potentially "winning". This means the cost of the U.S. falling behind globally is smaller because the "race" is not denominated in AI progress, it's instead substantially determined by which of the few existing superpowers is willing to sabotage the others with conventional escalation, with extremely fragile global semiconductor supply chains which are vulnerable to sabotage by a large number of different actors.

I am curious what you think about nationalisation in this context. It would help transform the current requirement of inter-company coordination to the admittedly much 'better-placed' requirement of nation-state coordination.

Moreover, nationalisation is also a reasonable condition to agree on for the labs, regardless of pause-like motives (hence strictly easier than Plan A and by transitivity, easier than Plan C), due to the technical and financial benefits of centralised coordination.

Only disadvantages/unlikelihoods are possibly the US government's inexperience/distaste of nationalisation, the inability to accomodate it within the budget, and probable loss of control by lead researchers and current CEOs. The first could be reasonably changed with enough push and influence. Moreover, second becomes really easy if the Federal Government creates a Department of Compute, hollows out existing companies to product managers and executives (hence becoming "inference-providers"), and rehires the remaining team in DoC. The effect is same as paying the enormous fair-market price for the labs. Third could be managed by the specific contract details (probably, I am just speculating for this part).

Increased military applications are also a possible disadvantage, but I do not think it really makes a difference since currently too the models (internal too) are used by the DOW.

Edited: Grammar

[-]habryka2d42

The most likely way nationalization would happen would be as a result of a Manhattan-project-style race towards ASI, in which the government intentionally pools resources between all the different labs in order to accelerate AI capabilities. This would be very bad! The U.S. government is much more capable than the labs at engaging in covert research activities, they are legally allowed to lie to the public, and in general this seems like it would make the race worse.

A way nationalization could happen that would be good is more like the post-2008 financial crisis nationalization of banks. In those worlds the governments started putting auditors in all kinds of places and transparency was drastically increased, and the government was placed in some strategic places where it could prevent things from going wrong, but they didn't take control of it in a way that would allow the government to do their own financial fraud.

[-]nam_14h10

The U.S. government is much more capable than the labs at engaging in covert research activities, they are legally allowed to lie to the public, and in general this seems like it would make the race worse.

Hmm, I agree, large-scale coordination between the labs is a double-edged sword. Although this is nothing appropriate laws/checks would not cure, would it? AI has more public-interest relevance than, say, nuclear weapons, and it is already pretty public. It makes sense for there to be a law regarding appropriate disclosure of capabilities and alignment research, as well as ability to shut down/slow down research based on Senate's discretion- if labs were to be nationalised. The analogue of banks post-2008 does not work since banks are still independent, private entities. In order to reduce the 'degrees of freedom' [make them coordinate], we have to explicitly make them non-independent and public entities.

[-]habryka3h20

It makes sense for there to be a law regarding appropriate disclosure of capabilities and alignment research, as well as ability to shut down/slow down research based on Senate's discretion- if labs were to be nationalised.

If AI becomes a military project, it will probably run quite covertly, at least if the history of military projects is anything to go off of (though of course this would be a huge outlier and be quite different and weird).

[-]BryceStansfield2d10

"The U.S. should unilaterally stop, then use conventional methods to force other countries to do the same.^[3]"

Has this been wargamed? With current production levels I'm not entirely convinced that the US can credibly make this threat.

[-]habryka2d20

What do you mean by "current production levels" here?

[-]BryceStansfield1d10

US drone and missile production.

[-]Zach Stein-Perlman3d200

should have titled it

Against Strawmanning Unilateral Research Abeyance

[-]cousin_it2d13-11

I also think that Scott's post was wrong, and it's good that it's getting pushback.

Still, I can't help but ask. At OpenAI, you worked on RLHF which enabled ChatGPT. Now you work on alignment at Resolution, which got $160M from Coefficient Giving (the same people who gave $30M to OpenAI at the start) and $10M in API credits from OpenAI. Your preferred strategy is "pivoting to safety + security + product", which are all economically complementary to capabilities and increase demand for capabilities. And as far as I can tell, none of the organizations mentioned - OpenAI, Coefficient, Resolution - are doing anything meaningful toward pause or slowdown. I don't mean this as a personal attack, but: are we going to see anything meaningful?

[-]habryka2d177

At least Geoffrey has been calling very publicly for labs to slow down and stop, unilaterally if necessary. I don't know how much that is an official "Resolution" project, but as a project by the CEO, I will by default presume that's the case. So it seems wrong to summarize Resolution as "not doing anything meaningful toward pause or slowdown".

[-]cousin_it2d1-5

This... doesn't seem like a strong answer to me. If we're going by public calls for good things, then Sam Altman is awesome too. Actions speak louder than words.

[-]habryka2d84

I don't quite know what action you want Resolution to take. It's a research org. Has Lightcone taken any "actions" under this definition of "action"?

[-]Lance Spencer2d85

I'm confused about several things right now. One is why some people who generally are fans of the concept of an ITT are not only failing to use it, but some also seem to believe people are acting in bad faith and/or purposively making arguments they don't believe or know to be incorrect. I feel a little bit like we're arguing over a book of common prayer.

I'm also confused why Geoffrey has me blocked on Twitter, since I don't think I've ever interacted with his work before now.

I'm very much a fan of a pause for "safety + security + product" and we know some of the labs have done this for up to a month or so. My understanding is that Scott is too, and that basically everyone involved would support there being a government-mandated one.

But the murky details over "what if only one lab does it" and "what if only the US does it" and "for how long, before those that don't stop pull ahead and make anyone else pausing irrelevant" are tricky questions. And I think reasonable people can disagree on the particulars. There are multiple levels of uncertainty and decentralized coordination is hard.

I had the opportunity to talk to some worker bee lab employees recently, including one who works on wrangling frontier models. I was gently poking him on "you say the internal model is so incredibly powerful and capable at escaping its cell; seems problematic." Eventually, he was basically like "I think we just need to nationalize the labs." My reaction was "woah there Mao" because I got a slight bit of whiplash. Many are confused and have conflicting instincts.

[-]Dean Valentine3d70

Here is a section of the post:

In March, I wrote a snarky post called Every Debate On Pausing AI. It made fun of our misinformed opponents, who insisted that if we unilaterally paused AI then China would win. But actually, I said, nobody wants to unilaterally pause AI; this is misinformation. All reputable AI pause groups want a bilateral treaty that pauses both sides at the same time.
Here I am forced to eat crow; as things shift ever more towards SURA, there are increasing demands for the US to pause unilaterally. Bernie Sanders' "Ban Artificial Superintelligence Act", which was endorsed by MIRI, is approximately the following plan:
The US pauses AI unilaterally.
Then the US asks China to please pause their AI too.^[14]
But if the US pauses unilaterally, it destroys its main bargaining chip, which is that China is currently on track to lose the AI race, and so ought to be happy to declare a tie (even if they don't believe in dangerous superintelligence). If this act were to pass, then China might believe that it would be assured a victory if it continued. In this case, they would be held back only by speculative scenarios which they currently show few signs of believing.

Is this part of the post inaccurate? It seems like it's describing something much different.

[-]Geoffrey Irving3d3-3

I haven't read the full details of Sanders' bill, but I am very confident it does not correspond to Scott's strawmanned version of unilateral pausing.

The title of your link is "NEWS: Sanders, Casar to Introduce Legislation to Ban Artificial Superintelligence and Temporarily Pause Advanced AI Development". Temporary is right there in the title. The text says "temporarily pause advanced AI development until a federal regulator has established safety rules".
Nowhere in the post does it propose destroying any company.

[-]Scott Alexander3d3716

I think you should read the bill.

It says that the US will immediately pause all frontier AI progress until some kind of regulator is set up (this is the temporary part, but it's unclear how long setting up the regulator would take, and it could easily be longer than the length of the US' lead).

It then additionally says that the regulator's mandate will be to permanently ban AIs with "superintelligence precursor characteristics" which include things like being able to hack. The bill doesn't do a good job explaining how it interprets this. Taken seriously, it means that all AIs created after this spring are illegal, because they can hack. It would make sense if there were an exemption for AIs with guardrails ensuring that they don't hack even though they could, but the bill doesn't say this. You could also imagine getting around this by making a modern AI "unlearn" hacking, if such a thing is robustly possible, but again, the bill doesn't speak to this and I don't know how technically plausible it is. In any case, AI past human intelligence is also banned permanently.

So strictest possible interpretation, in addition to the temporary unilateral pause until the regulator is set up, there's a permanent unilateral pause at early-2026 level. Laxest possible interpretation, there's a permanent unilateral pause somewhere before superintelligence.

Moderation Log