Before our eyes, forecasts are contracting and RSI is becoming nearer with each day. That's why I decided to publish a wide list of the obstacles for AI progress. I've been collecting these ideas for a long time, trying to make the list as comprehensive as it can be.
Some of these obstacles may turn out to be almost irrelevant. Others might slow things down by months or even longer. Or they might not. It's difficult to estimate their impact, so for now my goal was just to briefly list them.
List will include more then just obvious things like market correction, hardware shortages, or diminishing returns. Those are here too, of course. But also there are a few things that don't get a lot of attention.
I will start with the ones I consider as less interesting (part I) and move on to the more unusual variants (parts II & III).
I. Physical and Economic Bottlenecks
1. Energy limits
Developers announce campuses in gigawatts, file for interconnection and publish target dates that rarely survive permitting, financing and power supply.
Actually, I don't personally consider this as a bottleneck. AI could learn from synthetic data or simulations without running into model collapse. We might also discover ways to train models more efficiently on the data we already have. Systems with continuous learning may emerge. After all, humans still generate thousands of petabytes of new data every day.
Additional:
There's also a tacit knowledge: knowledge that hasn't been formalized and is therefore difficult to transfer. But there's an obvious objection: AI can generalize beyond its training data. The real question is how well it can do that.
5. Political restrictions
Moratoriums on data center construction, regulations, export controls, and so on. The recent Democratic victory in the midterms makes me a bit more hopeful that this could become a meaningful obstacle.
6. Hardware performance ceiling
Even if an intelligence explosion happens, AI might still be limited by the performance of the hardware available at that moment. It would have to either come up with more energy-efficient algorithms or design new hardware and wait for it to be manufactured.
II. Algorithmic Constraints and Research Bottlenecks
7. Limits of distributed training
a) Data transfer within enormous clusters. Adding more compute eventually gives diminishing returns. Epoch has modeled potential data-movement bottlenecks.
b) Memory wall. Even if a processor is fast, memory bandwidth can still be a bottleneck. This problem is as old as time, or at least much older than AI.
8. Messy tasks
Training AI, especially with RL, is much harder when we're dealing with messy tasks where verifying the result is difficult, or where there's no single correct answer. It's much easier to train an AI to solve math or programming problems than to teach it something like investing or running a large company. I'd also put the famous problem of research taste here. It's not obvious how well something like that can be trained through RL.
9. Reliability on long-horizon tasks
Success rates depend on how long a task is, something METR has been tracking closely. Anyway, time horizons are steadily increasing. But the question is, will AI be able to handle tasks that take months or even years, especially if it has never encountered anything like them in its training data? This is related to the previous point.
There are also problems with long-term memory, context windows filling up, and so on.
10. Swarm scaling
A thousand researchers don't make research a thousand times faster. Five people won't make one cup of coffee five times faster than one person. For example, agents can duplicate each other's work.
Toby Ord estimated that increasing the number of agents by 10x produced gains equivalent to giving a single agent only 3-5x more reasoning tokens.
Additional:
a) Communication complexity. For some computational problems, there's a minimum amount of information that Alice and Bob have to exchange to solve the problem together, no matter how clever their protocol is.
b) Correlated failures. A lack of diversity: a million copies of the same model would stumble upon the same weaknesses.
11. Critical path
Sometimes you can't finish step C before getting the result of B, which itself depends on A, and so on. The related thing is Amdahl's law: the speedup you can get from parallelization is ultimately limited by the part of the work that can't be parallelized.
12. Sequential inference bottleneck
Transfomers generate text sequentially. Each new token depends on the tokens that came before it. Still, there is things like speculative decoding.
13. Experimental verification bottlenecks
Coming up with an idea is easier than verifying it. Experiments in the real world still take time, and in AI R&D there is a lot of experiments.
The same applies for new training runs, which can sometime take months.
14. Sample complexity
For some problems, there are mathematically proven lower bounds on the number of observations needed.
15. Diminishing returns themselves
And of course, there's the idea that finding each new useful idea requires more and more research effort.
On the other hand, a recent paper showed that automating research can create positive feedback loops strong enough to overcome diminishing returns.
III. Possible Fundamental Limits
16. Computational complexity
Some problems require massive amounts of computation in the worst case.
17. Computational irreducibility
For some processes, there may be no faster way to predict how they will evolve than to actually run through the process itself.
18. Chaos and limited predictability
Because of chaos, tiny uncertainties in initial conditions can grow very quickly. I'm not sure that even superintelligence could do a long-term predictions about certain physical systems.
19. Physical limits of computation
The Landauer limit, Bekenstein bound, Margolus–Levitin theorem, and the speed of light. These are all discussed in Seth Lloyd's classic paper.
Of course, if these are the only things left that limit superintelligence, our forecasts probably won't mean anything anymore. I'm including them as honourable mentions.
Before our eyes, forecasts are contracting and RSI is becoming nearer with each day. That's why I decided to publish a wide list of the obstacles for AI progress. I've been collecting these ideas for a long time, trying to make the list as comprehensive as it can be.
Some of these obstacles may turn out to be almost irrelevant. Others might slow things down by months or even longer. Or they might not. It's difficult to estimate their impact, so for now my goal was just to briefly list them.
List will include more then just obvious things like market correction, hardware shortages, or diminishing returns. Those are here too, of course. But also there are a few things that don't get a lot of attention.
I will start with the ones I consider as less interesting (part I) and move on to the more unusual variants (parts II & III).
I. Physical and Economic Bottlenecks
1. Energy limits
- Energy Aspects
2. Hardware limits
- IDC
3. Market correction
- IMF
4. Lack of training data
Actually, I don't personally consider this as a bottleneck. AI could learn from synthetic data or simulations without running into model collapse. We might also discover ways to train models more efficiently on the data we already have. Systems with continuous learning may emerge. After all, humans still generate thousands of petabytes of new data every day.
Additional:
There's also a tacit knowledge: knowledge that hasn't been formalized and is therefore difficult to transfer. But there's an obvious objection: AI can generalize beyond its training data. The real question is how well it can do that.
5. Political restrictions
Moratoriums on data center construction, regulations, export controls, and so on. The recent Democratic victory in the midterms makes me a bit more hopeful that this could become a meaningful obstacle.
6. Hardware performance ceiling
Even if an intelligence explosion happens, AI might still be limited by the performance of the hardware available at that moment. It would have to either come up with more energy-efficient algorithms or design new hardware and wait for it to be manufactured.
II. Algorithmic Constraints and Research Bottlenecks
7. Limits of distributed training
a) Data transfer within enormous clusters. Adding more compute eventually gives diminishing returns. Epoch has modeled potential data-movement bottlenecks.
b) Memory wall. Even if a processor is fast, memory bandwidth can still be a bottleneck. This problem is as old as time, or at least much older than AI.
8. Messy tasks
Training AI, especially with RL, is much harder when we're dealing with messy tasks where verifying the result is difficult, or where there's no single correct answer.
It's much easier to train an AI to solve math or programming problems than to teach it something like investing or running a large company.
I'd also put the famous problem of research taste here. It's not obvious how well something like that can be trained through RL.
9. Reliability on long-horizon tasks
Success rates depend on how long a task is, something METR has been tracking closely. Anyway, time horizons are steadily increasing. But the question is, will AI be able to handle tasks that take months or even years, especially if it has never encountered anything like them in its training data? This is related to the previous point.
There are also problems with long-term memory, context windows filling up, and so on.
10. Swarm scaling
A thousand researchers don't make research a thousand times faster. Five people won't make one cup of coffee five times faster than one person. For example, agents can duplicate each other's work.
Toby Ord estimated that increasing the number of agents by 10x produced gains equivalent to giving a single agent only 3-5x more reasoning tokens.
Additional:
a) Communication complexity. For some computational problems, there's a minimum amount of information that Alice and Bob have to exchange to solve the problem together, no matter how clever their protocol is.
b) Correlated failures. A lack of diversity: a million copies of the same model would stumble upon the same weaknesses.
11. Critical path
Sometimes you can't finish step C before getting the result of B, which itself depends on A, and so on. The related thing is Amdahl's law: the speedup you can get from parallelization is ultimately limited by the part of the work that can't be parallelized.
12. Sequential inference bottleneck
Transfomers generate text sequentially. Each new token depends on the tokens that came before it. Still, there is things like speculative decoding.
13. Experimental verification bottlenecks
Coming up with an idea is easier than verifying it. Experiments in the real world still take time, and in AI R&D there is a lot of experiments.
The same applies for new training runs, which can sometime take months.
14. Sample complexity
For some problems, there are mathematically proven lower bounds on the number of observations needed.
15. Diminishing returns themselves
And of course, there's the idea that finding each new useful idea requires more and more research effort.
On the other hand, a recent paper showed that automating research can create positive feedback loops strong enough to overcome diminishing returns.
III. Possible Fundamental Limits
16. Computational complexity
Some problems require massive amounts of computation in the worst case.
17. Computational irreducibility
For some processes, there may be no faster way to predict how they will evolve than to actually run through the process itself.
18. Chaos and limited predictability
Because of chaos, tiny uncertainties in initial conditions can grow very quickly. I'm not sure that even superintelligence could do a long-term predictions about certain physical systems.
19. Physical limits of computation
The Landauer limit, Bekenstein bound, Margolus–Levitin theorem, and the speed of light. These are all discussed in Seth Lloyd's classic paper.
Of course, if these are the only things left that limit superintelligence, our forecasts probably won't mean anything anymore. I'm including them as honourable mentions.