This is a linkpost for https://epoch.ai/gradient-updates/the-case-for-multi-decade-ai-timelines
The case for multi-decade AI timelines [Linkpost]
20ryan_greenblatt
4Lukas Finnveden
5ryan_greenblatt
5Lukas Finnveden
4ryan_greenblatt
10elifland
2Noosphere89
10Vladimir_Nesov
6Noosphere89
7Vladimir_Nesov
3faul_sname
2Vladimir_Nesov
3faul_sname
2Vladimir_Nesov
2faul_sname
3Vladimir_Nesov
2faul_sname
2Noosphere89
8ryan_greenblatt
8elifland
2Asta7k
2Noosphere89
New Comment
Another potential crux[1] is that Ege's world view seemingly doesn't depend at all on AIs which are much faster and smarter than any human. As far as I can tell, it doesn't enter into his modeling of takeoff (or timelines to full automation of remote work which partially depends on something more like takeoff).
On my views this makes a huge difference because a large number of domains would go much faster with much more (serial and smarter) intelligence. My sense is that a civilization where the smartest human was today's median human and also everyone's brain operated 50x slower[2] would in fact make technological progress much slower. Similarly, if AIs were as much smarter than the smartest humans as the smartest human is smarter than the median human and also ran 50x faster than humans (and operated at greater scale than the smartest humans with hundreds of thousands of copies all at 50x speed for over 10 million parallel worker equivalents putting aside the advantages of serial work and intelligence), then we'd see lots of sectors go much faster.
My sense is that Ege bullet bites on this and thinks that slowing everyone down wouldn't make a big difference, but I find this surprising. Or maybe his views are that parallelism is nearly as good as speed and intelligence and sectors naturally scale up parallel worker equivalents to match up with other inputs, so we're bottlenecking on some other inputs in the important cases.
FWIW, that's not the impression I get from the post / I would bet that Ege doesn't "bite the bullet" on those claims. (If I'm understanding the claims right, it seems like it'd be super crazy to bit the bullet? If you don't think human speed impacts the rate of technological progress, then what does? Literal calendar time? What would be the mechanism for that?)
The post does refer to how much compute AIs need to match human workers, in several places. If AIs were way smarter or faster, I think that would translate into better compute efficiency. So the impression I get from the post is just that Ege doesn't expect AIs to be much smarter or faster than humans at the time when they first automate remote work. (And the post doesn't talk much about what happens afterwards.)
Example claims from the post:
My expectation is that these systems will initially either be on par with or worse than the human brain at turning compute into economic value at scale, and I also don’t expect them to be much faster than humans at performing most relevant work tasks.
...
Given that AI models still remain less sample efficient than humans, these two points lead me to believe that for AI models to automate all remote work, they will initially need at least as much inference compute as the humans who currently do these remote work tasks are using.
...
These are certainly reasons to expect AI workers to become more productive than humans per FLOP spent in the long run, perhaps after most of the economy has already been automated. However, in the short run the picture looks quite different: while these advantages already exist today, they are not resulting in AI systems being far more productive than humans on a revenue generated per FLOP spent basis.
If I'm understanding the claims right, it seems like it'd be super crazy to bit the bullet? If you don't think human speed impacts the rate of technological progress, then what does? Literal calendar time? What would be the mechanism for that?
Physical bottlenecks, compute bottlenecks, etc.
The claim that you can only speed up algorithmic progress (given a fixed amount of compute) by a moderate amount even with an arbitrarily fast and smart superinteligence reduces to something like this.
So the impression I get from the post is just that Ege doesn't expect AIs to be much smarter or faster than humans at the time when they first automate remote work. (And the post doesn't talk much about what happens afterwards.)
Yes, but if you can (e.g.) spend extra compute to massively accelerate AI R&D or a smaller number of other key sectors which might be bottlenecked on fast labor, then doing this might be much more useful than very broadly automating remote work. I think it's somewhat hard to end up with a view where generally automating remote work across the whole economy using a form factor pretty similar to a human worker (in speed and smarts) is plausible unless you also don't think there are huge returns to accelerated speed given that things are so likely to funge and be variable.
(E.g., right now it is possible to spend more money to run AIs >10x faster. So, given limited fab ability, it will probably be possible to run AIs 10x faster than otherwise at substanitially higher cost at a point before when you would otherwise been able to automate all remote work. This implies that if there are high returns to speed, then you'd deploy these fast AIs for these tasks.)
You can interpret my argument here as claiming that in some important sectors/tasks AIs will be vastly more productive than typical humans per flop spent due to higher smarts (even if the AIs aren't superhuman, the best humans are very scarce) and serial speed. By the time you've gotten around to automating everything, quite likely the AIs are very superhuman because you drilled down on narrower parts of the economy. (Then there is the question of whether these parts of the economy bottleneck without growing everything in parallel which is a generalization of the software only singularity question.)
Separately:
while these advantages already exist today, they are not resulting in AI systems being far more productive than humans on a revenue generated per FLOP spent basis.
I must misunderstand what Ege means by this because isn't this trivially false on a task by task basis? If you tried to use a human in cursor it would be much less useful in many respects due to insufficient serial speed.
Maybe Ege means "the current marginal revenue from using 1e15 FLOP / s isn't much higher than the revenue from a reasonably capable human", but isn't this just an extremely trivial implication of there being a market for compute and the cost of compute being below the cost of labor. (A human in the US costs $7-100 / hour while human equivalent flop (1e15 FLOP / s) costs around $2 / hour.) I think this can't possibly be right because this claim was trivially false 30 years ago when chips were worse. I certainly agree that compute prices are likely to rise once AIs are more capable.
Physical bottlenecks, compute bottlenecks, etc.
Compute would also be reduced within a couple of years, though, as workers at TSMC, NVIDIA, ASML and their suppliers all became much slower and less effective. (Ege does in fact think that explosive growth is likely once AIs are broadly automating human work! So he does think that more, smarter, faster labor can eventually speed up tech progress; and presumably would also expect slower humans to slow down tech progress.)
So I think the counterfactual you want to consider is one where only people doing AI R&D in particular are slowed down & made dumber. That gets at the disagreement about the importance of AI R&D, specifically, and how much labor vs. compute is contributing there.
For that question, I'm less confident about what Ege and the other mechanize people would think.
(They might say something like: "We're only asserting that labor and compute are complementary. That means it's totally possible that slowing down humans would slow progress a lot, but that speeding up humans wouldn't increase the speed by a lot." But that just raises the question of why we should think our current labor<>compute ratio is so close to the edge of where further labor speed-ups stop helping. Maybe the answer there is that they think parallel work is really good, so in the world where people were 50x slower, the AI companies would just hire 100x more people and not be too much worse off. Though I think that would massively blow up their spending on labor relative to capital, and so maybe it'd make it a weird coincidence that their current spending on labor and capital is so close to 50/50.)
Re your response to "Ege doesn't expect AIs to be much smarter or faster than humans": I'm mostly sympathetic. I see various places where I could speculate about what Ege's objections might be. But I'm not sure how productive it is for me to try to speculate about his exact views when I don't really buy them myself. I guess I just think that the argument you presented in this comment is somewhat complex, and I'd predict higher probability that people object (or haven't thought about) some part of this argument then that they bite the crazy "universal human slow-down wouldn't matter" bullet.
1Yeah, I agree with this and doesn't seem that productive to speculate about people's views when I don't fully understand them.
They might say something like: "We're only asserting that labor and compute are complementary. That means it's totally possible that slowing down humans would slow progress a lot, but that speeding up humans wouldn't increase the speed by a lot." But that just raises the question of why we should think our current labor<>compute ratio is so close to the edge of where further labor speed-ups stop helping.
I discuss this sort of thing in this comment and in a draft post I've DM'd you.
I still think full automation of remote work in 10 years is plausible, because it’s what we would predict if we straightforwardly extrapolate current rates of revenue growth and assume no slowdown. However, I would only give this outcome around 30% chance.
In an important sense I feel like Ege and I are not actually far off here. I'm at more like 65-70% on this. I think this overall recommends quite similar actions. Perhaps we have a more important disagreement regarding something like P(AGI within 3 years), for which I'm at approx. 25-30% and Ege might be very low (my probability mass is somewhat concentrated in the next 3 years due to an expectation that compute and algorithmic effort scaling will slow down around 2029 if AGI or close isn't achieved).
My guess is that this disagreement is less important to make progress on than disagreements regarding takeoff speeds/dynamics and alignment difficulty.
I do think the difference between an AGI timeline median of 5 years and one of 20 years does matter, because politics starts affecting whether we get AGI way more if we have to wait 20 years instead of 5, and serial alignment agendas make more sense if we assume a timeline of 20 years is a reasonable median.
Also, he argues against very fast takeoffs/software only singularity in the case for multi-decade timelines post.
I think the main crux with AI 2027 is the very possibility of software-only singularity, rather than details of human economy, or details of software-only singularity. For people feeling skeptical about it, it's hard to find legible arguments that it can happen, and so they tend to lose interest in discussing its details, instead discussing other systems such as the datacenter-relevant parts of the economy, which doesn't chip away at this crux of the disagreement. I talk more about this in another comment in response to Ege Erdil's post.
A well-known example where a system different from human industry matters more for the dynamics of AI takeoff is nanotech, which makes the inputs derived from human industry irrelevant for scaling compute.
Many people are skeptical of nanotech, but alternatively there's macroscopic biotech, designing animal robots assembled with metamorphosis from small mobile things analogous to fruit flies, which can double biomass every 1-3 days (an impossible level of growth compared to the human economy). This only depends on procuring power and raw materials (rather than human technology) and provides manipulation in the physical world to assemble mines and factories and such (possibly even directly growing biological compute). There is much more of a proof of concept for this kind of thing than for nanotech.
1Basically agree with this, but the caveat here is fruit flies are pretty much pure instinct, and a lot of the nanotech that is imagined is more universalist than that.
But yeah, fruit flies are an interesting case where biotech has managed to get pretty insane doubling times, and if we can pack large amounts of effective compute into very small spaces, this would hugely support something like a software only singularity.
Though I did highlight the possibility of a software-only singularity as the main crux in my post.
Many people are skeptical of nanotech
The best (in my opinion) nanotech skeptical cases are from @Muireall and @bhauth below:
https://muireall.space/nanosystems/
https://muireall.space/pdf/considerations.pdf#page=17
https://www.lesswrong.com/posts/FijbeqdovkgAusGgz/grey-goo-is-unlikely
Macroscopic biotech is not software-only singularity (and doesn't depend on it being possible), it's a separate example of a system that is not human industry, that seems sufficient to create a rate of scaling completely implausible for the human economy (illustrating the crux with Ege Erdil's position, which system is relevant for timelines). Software-only singularity operates within an unchanging amount of raw compute, while macroscopic biotech takeoff manufactures new compute much faster than human industry could be instructed to.
The "fruit flies" are the material substrate, packed with DNA that can specialize the cells to do their part in a larger organism reassembled from these fast-growing and mobile packages of cells, and so precise individual control over the "fruit flies" is unnecessary, only the assembled large biorobots will need to be controllable. The main issue is that macroscopic biotech probably won't be able to create efficient compute or fusion plants directly, but it can create integrated factories that manufacture these things (or non-biological robots) in more standard ways by reproducing the whole necessary supply chains starting with raw materials (or whatever is most convenient to transport) on site, plausibly extremely quickly. With enough robots, city-sized factories could be assembled overnight, and then reassembled again and again wholesale, as feedback from trying it out in the physical world reveals flaws in design.
2- How long does you expect it would take to assemble a "biorobot"?
- How many serial attempts before the kinks in the process are worked out? Keep in mind that with biological components, things will try to eat and parasitize the things you build, so you have a moving target.
- When do you expect the first attempt?
- Does this timeline work for a parallel human-economy-level system spun up within the next 10 years?
The "fruit flies" are the source of growth, so the relevant anchor is how long it takes to manufacture a lot of them. Let's say there are 1000 "flies" 1 mg each to start, doubling in number every 2 days, and we want to produce 10 billion 100 kg robots (approximately the total mass of all humans and cattle), which is 1e15x more mass and will take 100 days to produce. Anchoring to the animal kingdom, metamorphosis takes a few days to weeks, which doesn't significantly affect the total time.
things will try to eat and parasitize the things you build
I'm assuming the baseline of existing animals such as cattle that are doing OK, not completely novel design.
When do you expect the first attempt?
I'm not assuming a software-only singularity (substantially increasing compute efficiency and intelligence of AIs), only ~100x faster-than-human automated R&D that's necessary for it (but possibly not sufficient). The AIs instead develop better biology modeling software (and all the theory that requires), to the point where only months' worth of compute and actual experiments would be necessary to fix important discrepancies with ground truth, making engineering of a wide variety of functional biological robots feasible.
So the overall prediction is that 1-2 years from hitting automated R&D, even without a software-only singularity, there could be a ~humanity-sized workforce that is instantly customizable for any physical manipulation purpose and could subsequently double every 2 days, constrained only by raw materials and ability to convert them into power plants and factories to sustain themselves.
What does it mean that the fruit flies are a source of growth? Is the idea to use them as raw biomass?
Because if the goal is "get a billion metric tons of dry biomass", I expect already straightforward to use agricultural waste. Global rice straw (the stuff left in the fields after the rice is harvested) production alone is already about a billion metric tons per year. At $165 / ton, it would be a bit pricy - twice the $100B OpenAI is immediately deploying for their Stargate data center - but very much manageable for a big company if the expected payoff was there.
I don't think raw biomass is a meaningful bottleneck. If your timelines had a couple year period "time it takes for the AI to establish control over a billion tons of biomass" I think you should remove that period from your timeline.
The AIs instead develop better biology modeling software (and all the theory that requires), to the point where only months' worth of compute and actual experiments would be necessary to fix important discrepancies with ground truth, making engineering of a wide variety of functional biological robots feasible.
I think this is where the crux is. In software, you can take a system with a bug, determine where the bug is, fix it, deploy the fix, and have the fixed version running minutes to hours after you identified the bug.
With engineered biological systems, some "bugs" don't manifest until the system has been running for weeks or months. Your cycle time, then, is not the generational time of your "fruit flies", but the time it takes between when you start assembling a biorobot and when that biorobot starts doing useful work.
Maybe the crux is that you expect that it is feasible to construct biological simulations which perform well for long-term modeling, even when modeling something where that simulation has not been tuned to match observational data from that domain, and I expect that not to be a thing that is available at near-future levels of compute.
there could be a ~humanity-sized workforce that is instantly customizable for any physical manipulation purpose and could subsequently double every 2 days, constrained only by raw materials and ability to convert them into power plants and factories to sustain themselves
I mean if you have a billion tons of biomass sitting around and the ability to "program" that biomass into biorobots, I don't think it particularly makes sense to talk about the "doubling time" of biorobots - biorobots aren't a meaningful bottleneck to the production of more biorobots, so once you have one you can go straight to having a billion. I think the difficult part is the bit where you get one biorobot that functions how you want it to.
The manufacturing process starts with AI-designed DNA, which is used to produce a few "flies" using scarce biotech equipment, and then those flies are used to manufacture a billion tons of cells with AI-designed DNA within ~100 days, using only low-tech inputs like cheap feed. The cells form the bodies of the "flies", and the "flies" can assemble into large robots using something like metamorphosis, repurposing the cells. So by flies being the source of growth I mean the growth of the amount of high tech capital that is the cells with AI-programmed DNA capable of assembling into functional large robots.
I don't think raw biomass is a meaningful bottleneck.
The whole point of going with biological robots and then "fruit flies" is that this removes bottlenecks on volume production, once the cells have been designed and the first few "flies" can be manufactured. And once there is a billion tons of robots, they can rapidly set up more infrastructure than the human industry would be able to, and continue the process.
biorobots aren't a meaningful bottleneck to the production of more biorobots
That's why you need the "flies". Since megafauna can't double their numbers every 2 days, without the "flies" the number of biorobots would be a bottleneck to production.
With engineered biological systems, some "bugs" don't manifest until the system has been running for weeks or months.
The point of developing simulation software/models is to become able to run accurate high-speed simulations of biological organisms, in order to fix bugs in cell/fly/robot designs faster than the experiments on them could be performed in the physical world. Feedback from the physical experiments that will still be performed is probably more useful for fixing the errors in the simulation software/models, rather than for directly fixing the errors in cell/fly/robot designs.
Maybe the crux is that you expect that it is feasible to construct biological simulations which perform well for long-term modeling, even when modeling something where that simulation has not been tuned to match observational data from that domain, and I expect that not to be a thing that is available at near-future levels of compute.
Sure, this seems similar to intuitions about impossibility of software-only singularity. The point of the macroscopic biotech example is that it doesn't depend on either superintelligence or much higher compute efficiency in AI. But it does depend on high efficiency/accuracy long horizon simulations of large biological organisms based on engineered DNA being possible to develop in 100-200 years of human-equivalent theory/software progress with limited opportunity to run physical experiments.
Feedback from the physical experiments that will still be performed is probably more useful for fixing the errors in the simulation software/models, rather than for directly fixing the errors in cell/fly/robot designs.
This is something I want to poke at a bit, because it seems like a pretty core disagreement.
In a completely different domain, do you expect something like DGMR (DeepMind's precipitation nowcasting ML thingy., basically a GAN over weather radar maps) would work better than non-ML weather models to predict US weather after being trained only on UK weather? I expect not, and I don't expect the reason it wouldn't work is anything like "the ML engineers weren't clever enough".
The bio simulations need to be more clever than ML on bio data, they need to incorporate feedback from simulations of more basic/fundamental/general principles of chemistry and physics. Making this possible is what the 100-200 subjective years of R&D are for.
I'm not confident 100-200 subjective years of R&D help enough, for the same reason I don't think it would be enough for US weather forecasting to have 100-200 years to look at and build models of UK weather data in order to predict US weather well enough to make money in our crop futures markets. Training on UK data would definitely help more than zero at predicting US weather, but "more than zero" is not the bar.
Similarly, 200 years of improvements to biological simulations would help more than zero with predicting the behavior of engineered biosystems, but that's not the bar. The bar is "build a functional general purpose biorobot more quickly and cheaply than the boring robotics/integration with world economy path". I don't think human civilization minus AI is on track to be able to do that in the next 200 years.
Similarly, 200 years of improvements to biological simulations would help more than zero with predicting the behavior of engineered biosystems, but that's not the bar. The bar is "build a functional general purpose biorobot more quickly and cheaply than the boring robotics/integration with world economy path". I don't think human civilization minus AI is on track to be able to do that in the next 200 years.
I don't think it's on track to do so, but this is mostly because of the coming population decline meaning regression in tech is very likely.
If I instead assumed that the human population would expand in a similar manner to the AI population, and was willing to rewrite/ignore regulations, I'd put a 70-80% chance that we could build bio-robots more quickly and cheaply than the boring robotics path in 200 years, with the remaining 10-20% being on the possibility that biotech is just fundamentally way more limited than people think.
I don’t see the trends that one would extrapolate in order to arrive at very short timelines on the order of a few years. The obvious trend extrapolations for AI’s economic impact give timelines to full remote work automation of around a decade, and I expect these trends to slow down by default.
Actually, I think the most obvious extrapolation of revenue indicates around 4 years to full remote work automation: current annualized AI company revenue is around $20 billion, we see 3x revenue growth per year, AI companies might internalize 10% of the value, so for ~$20 trillion of value due to remote work automation, we might expect ~$2 trillion in revenue which would naively happen in a bit over 4 years (late 2029).
(I don't think revenue extrapolation is a great forecasting methodology on my views for multiple reasons, but still seems worth engaging with to some extent.)
I think there are plausible reasons for this trend to speed up or to slow down (speeding up seems plausible even without AI accelerated AI R&D making a difference), and more minimally we'll see big impacts prior to full automation. Slowing down seems more likely than speeding up or staying at the same rate, but doesn't seem overwhelmingly likely (maybe I'd say around 30% chance that the trend stays at or above exponential up to $2 trillion in global revenue).
I discuss this sort of extrapolation and related considerations in more detail here.
So, I think a key takeaway from revenue extrapolation is it makes <4 year timelines look quite plausible (though it doesn't suggest they are >50% likely)!
- It underrates the difficulty of automating the job of a researcher. Real world work environments are messy and contain lots of detail that are neglected in an abstraction purely focused on writing code and reasoning about the results of experiments. As a result, we shouldn’t expect automating AI R&D to be much easier than automating remote work in general.
I basically agree. The reason I expect AI R&D automation to happen before the rest of remote work isn't because I think it's fundamentally much easier, but because (a) companies will try to automate it before other remote work tasks, and relatedly (b) because companies have access to more data and expertise for AI R&D than other fields.
1I posted some of my thoughts on their website, might aswell share it on here:
What I don’t understand is why they would need as much inference compute as a human? Maybe future architecture will make it way more inference cheaply compared to the human brain. And I don’t know how you can compare the human amount of inference and the amount of inference ai needs in order to automate remote work. Also, sample efficiency doesn’t apply to inference since weights are not updated at this stage (yet, some research suggest we should do that) and maybe it’ll end up more sparse (like transformers), in the sense that we can reduce the amount of compute we need for that. I also think you exaggerate how compute bound we are. Suppose we invent a new architecture, new paradigm or just tweak the transformer which makes it so much more sample efficient and also cheaper compute wise, we can just use these automated researchers to make the next generation even more cheaper, and they could be used explicitly at OpenAI or other research labs to speed up ai research, there isn’t really broad deployment needed.
Curated and popular this week
So this post is an argument that multi-decade timelines are reasonable, and the key cruxes that Ege Erdil has with most AI safety people who believe in short timelines are due to the following set of beliefs:
This is a pretty important crux, because if this is true, a lot more serial research agendas like Infra-Bayes research, Natural Abstractions work, and human intelligence augmentation can work, and also it means that political modeling (like is the US economy going to be stable long-term) matter a great deal more than is recognized in the LW/EA community.
Here's a quote from the article: