I believe that the disagreement is mostly about what happens before we build powerful AGI. I think that weaker AI systems will already have radically transformed the world, while I believe fast takeoff proponents think there are factors that makes weak AI systems radically less useful. This is strategically relevant because I'm imagining AGI strategies playing out in a world where everything is already going crazy, while other people are imagining AGI strategies playing out in a world that looks kind of like 2018 except that someone is about to get a decisive strategic advantage.

Huh, I think this is most useful framing of the Slow/Fast takeoff paradigm I've found. Normally when I hear about the slow takeoff strategy I think it comes with a vague sense of "and therefore, it's not worth worrying out because our usual institutions will be able to handle it."

I'm not sure if I'm convinced about the continuous takeoff, but it at least seems plausible. It still seems to me that recursive self improvement should be qualitatively different - not necessarily resulting in a "discontinuitous jump" but changing the shape of the curve.

The curve below is a bit sharper than I meant it to come out, I had some trouble getting photoshop to cooperate.

More thoughts to come. I agree with Qiaochu that this is a good "actually think for N minutes" kinda post.

Thanks for writing this! Your point about chimps vs. humans was new to me and I'll chew on it.

Because I think this post is important enough I'm gonna really go above and beyond here: I'm gonna set a timer and think for 5 minutes, with paper, before writing the rest of my comment.

Okay, that was fun. What I got out of reading your post twice (once before writing this and once during the 5-minute timer) was basically the following. Let me know if this is an accurate summary of your position:

AI researchers will try to optimize AI to be as useful as possible. There are many tradeoffs that need to be navigated in order to do this, e.g. between universality and speed, and the Pareto boundary describing all of these tradeoffs is likely to grow slowly / continuously. So we probably won't see fast / discontinuous Pareto improvements in available AIs.

I find this reasonably persuasive, but I'm not convinced the positions you've described as slow takeoff vs. fast takeoff are what other people mean when they talk about slow vs. fast takeoff.

I'd be interested to see you elaborate on this:

There is another reason I’m skeptical about hard takeoff from universality secret sauce: I think we already could make universal AIs if we tried (that would, given enough time, learn on their own and converge to arbitrarily high capability levels), and the reason we don’t is because it’s just not important to performance and the resulting systems would be really slow. This inside view argument is too complicated to make here and I don’t think my case rests on it, but it is relevant to understanding my view.

(Also, meta: I notice that setting the 5-minute timer had the effect of shifting me out of "look for something to disagree with" and into "try to understand what is even being claimed in the first place." Food for thought!)

Since there are some aspects of your version of "slow" takeoff that have been traditionally associated with the "fast" view, (e.g. rapid transformation of society), I'm tempted to try to tease apart what we care about in the slow vs fast discussion into multiple questions:

1) How much of a lead will one team need to have over others in order to have a decisive strategic advantage...

a) in wall clock time?

b) in economic doublings time?

2) How much time do we have to solve hard alignment problems...

a) in wall clock time?

b) in economic doublings time?

It seems that one could plausibly hold a "slow" view for some of these questions and a "fast" view for others.

For example, maybe Alice thinks that GDP growth will accelerate, and that there are gains to scale/centralization for AI projects, but that it will be relatively easy for human operators to keep AI systems under control. We could characterize her view as "fast" on 1a and 2a (because everything is happening so fast in wall clock time), "fast" on 1b (because of first-mover advantage), and "slow" on 2b (because we'll have lots of time to figure things out).

In contrast, maybe Bob thinks GDP growth won't accelerate that much until the time that the first AI system crosses some universality threshold, at which point it will rapidly spiral out of control, and that this all will take place many decades from now. We might call his view "fast" on 1a and 1b, and "slow" on 2a and 2b.

Curious to hear thoughts on this framework and whether there's a better one.

Just want to mention that this post caused me to update towards a continuous take-off. The humans and chimps section the most, as I hadn't noticed how much chimps weren't being optimised for general intelligence. And then the point that slow take-off means more change, sooner, was also key. I had been rounding slow take-off to a position which didn't really appreciate the true potential for capability gain, but your post showed me how to square that circle.

Fast take-off seems to me at the very least like a simplifying assumption, because I feel pretty unequipped to predict what will happen in the space before 'strongly superintelligent AGI' but after 'radical change due to AGI'. Would be interested to hear anyone's suggestions for reasoning about what that world will look like.

This is tangential, but I wanted to note that, a slow takeoff is, at least in some ways, a worse scenario than fast takeoff in terms of AI risk.

Designing an aligned AI (probably) involves trade-offs between safety and capability. In a fast takeoff scenario, the FAI "only" has to be capable enough to prevent UFAI from ever coming into being, in order for AI risk to be successfully mitigated. The most powerful potentially adversarial agents in its environment would be mere humans (and, yes, humans should be counted as somewhat adversarial since they are the source of the UFAI risk). In a (sufficiently) slow takeoff scenario, there will be a number of competing superintelligences, and it seems likely that at least some of them will not be aligned. This means that the FAIs have to be capable enough to effectively compete with UFAIs that are already deployed (while remaining safe).

I imagine the "secret sauce" line of thinking as "we are solving certain problems in the wrong complexity class". Changing complexity class of an algorithm introduces a discontinuity; when near a take-off, then this discontinuity can get amplified into a fast take-off. The take-off can be especially fast if the compute hardware is already sufficient at the time of the break-through.

In other words: In order to expect a fast take-off, you only need to assume that the last crucial sub-problem for recursive self-improvement / explosion is done in the wrong complexity class prior to the discovery of a good algorithm.

For strong historical precedents, I would look for algorithmic advances that improved empirical average complexity class, and at the same time got a speed-up of e.g. 100 x on problem instances that were typical prior to the algorithmic discovery (so Strassen matrix-multiply is out).

For weaker historical precedent, I would look for advances that single-handedly made the entire field viable -- that is, prior to the advance one had a tiny community caring about the problem; post the advance, the field (e.g. type of data analysis) became viable at all (hence, very limited commercial / academic interest in the subfield prior to its breakthrough). I think that this is meaningful precedent because people optimize for expected pay-off, and it is sometimes surprising that some small-but-crucial-if-possible subproblem can be solved at all (reasonable quickly)!

And I do believe that there are many parts of modern ML that are in the wrong complexity class (this does not mean that I could do better, nor that I necessarily expect an improvement or even discontinuous jump in usefulness).

There seems to be a stronger version of the (pro-fast-takeoff) chimp vs human argument, namely average-IQ humans vs von Neumann: "IQ-100 humans have brains roughly the same size as von Neumann, but are much worse at making technology." Does your counterargument work against that version too? (I'm not sure I fully understand it.)

I like this post, but I think it's somewhat misleading to call your scenario a "slow takeoff". To my mind, a "slow takeoff" evokes an image from non-singularitarian science fiction where you have human-level robots running around and they've been running around for decades if not centuries: that is, a very slow and gradual development that gives society and institutions plenty of time to adapt. But your version is clearly not this, since you are talking on the timescale of a few years, and note yourself that time will be of the essence even with a "slow" takeoff:

If takeoff is slow: it will become quite obvious that AI is going to transform the world well before we kill ourselves, we will have some time to experiment with different approaches to safety, policy-makers will have time to understand and respond to AI, etc. But this process will take place over only a few years, and the world will be changing very quickly, so we could easily drop the ball unless we prepare in advance.

You also note that much of the safety community seems to believe in a fast takeoff, in disagreement with you. I don't know whether you're including me there, but I've previously talked about a takeoff on the scale of a few years being a fast one, since to me a "fast takeoff" is one where there's little time for existing institutions to prepare and respond adequately, and a few years still seems short enough to meet that criteria.

I'd prefer to use some term like "moderate takeoff" for the scenario that you're talking about.

[edit: sorry if I seem like I'm piling on with the terminology, I first wrote this comment and only then read the other comments and saw that they ~all brought up the same thing.]

I first interpreted your operationalization of slow take off to mean something that is true by definition (assuming the economy is strictly increasing).

I assume how you wanted me to interpret it is that the first 4 year doubling interval is disjoint from the first 1 year doubling interval. (the 4 year one ends before the 1 year one starts.)

Updated me quite strongly towards continuous takeoff (from a position of ignorance). (I would also nominate the AI Impacts post, but I don't think it ever got cross-posted.)

Here's an argument why (at least somewhat) sudden takeoff is (at least somewhat) plausible.

Supposing:

(1) At some point P, AI will be as good as humans at AI programming (grandprogramming, great-grandprogramming, ...) by some reasonable standard, and less than a month later, a superintelligence will exist.

(2) Getting to point P requires AI R&D effort roughly comparable to total past AI R&D effort.

(3) In an economy growing quickly because of AI, AI R&D effort increases by at least the same factor as general economic growth.

Then:

(4) Based on (3), if there's a four year period during which economic growth is ten times normal because of AI (roughly corresponding to a four year doubling period), then AI R&D effort during that period is also at least ten times normal.

(5) Because 4*10=40 and because of additional R&D effort between now and the start of the four year period, total AI R&D effort between now and the end of such a period would be at least roughly comparable to total AI R&D effort until now.

(6) Therefore, based on (2) and (1), at most a month after the end of the first four year doubling period, a superintelligence will exist.

I think (1) is probable and (2) is plausible (but plausibly false). I'm more confused about (3), but it doesn't seem wrong.

There's a lot of room to doubt as well as sharpen this argument, but I hope the intuition is clear. Something like this comes out if I introspect on why it feels easier to coherently imagine a sudden than a gradual takeoff.

If there's a hard takeoff claim I'm 90% sure of, though, the claim is more like (1) than (6); more like "superintelligence comes soon after an AI is a human-level programmer/researcher" than like "superintelligence comes soon after AI (or some other technology) causes drastic change". So as has been said, the difference of opinion isn't as big as it might at first seem.