Today, I'll be linking both a Epoch blog post and a paper called The Bounded Parallelizability of R&D: Theory and Application to AI, about how standard models of AI R&D and standard models of the possibility of an intelligence explosion like Eth and Davidson 2025 neglect the possibility of bottlenecks because technological progress isn't perfectly parallelizable.

I'm mostly going to do a linkpost because I believe this isn't common knowledge in the community, but I will sprinkle in some of my own takes on parts of the paper and the blog post.

One important point to make is that the criticisms only apply to the technological singularity. An economic singularity where robots begin to be good enough such that they can replace people at doing physical tasks, enabling more resources and cheaper robots, thus enabling more robots ad infinitum until it reaches infinity in finite time is still possible ala our own history of superexponential growth (though in our reality, things would slow down at some point, and my hot take is that once we start to have our first dyson swarm complete, we can't grow superexponentially after the dyson swarm.)

(Tom Davidson does have good criticisms of the model, but the correct criticisms are about using the model to predict timelines, rather than takeoff.)

Without further ado, let's begin.

Paralleization technology

In a conventional model, technological progress depends on two forces: (1) ideas get harder to find as the technological frontier advances, but (2) society dedicates more resources to finding them. Technological progress is sustained when the second force offsets the first.

These models do allow R&D to exhibit diminishing returns to research inputs. Doubling the number of researchers need not double the rate of progress. Still, the models assume that by using enough R&D inputs at any given time, technology can advance arbitrarily quickly.

This is implausible. Imagine a robot factory that wants to increase its productivity: how quickly or cheaply it can produce each robot. No matter how many of today’s engineers, using today’s tools, surge onto the team, the factory’s productivity will not have doubled ten minutes later. Potential improvements must be identified, implemented, and tested, and it will likely take several rounds of this process before the improvements yield a more productive robot factory. Without technology to divide up that work — a sufficiently accurate simulator of the line, say, allowing parallel testing — no headcount can eliminate the time needed for these sequential steps.

This does not necessarily imply a hard physical limit on progress. Once better tools (including better robots) have already been developed, it might be possible to redesign and rebuild the factory much more quickly. But today’s tools place a ceiling on how much research effort can be absorbed productively today.

I therefore propose another input to technological progress: parallelization technology. “Effective research inputs” are limited by whichever is scarcer:

(2a) the raw research inputs; or

(2b) the “parallelization technology” needed to divide, execute, coordinate, and integrate their work.

For an example of parallelization technology, consider Anthropic’s development of agent teams. By coordinating the work of many AI agents, an effective agent team lets a given research project productively use more compute at once.

Historically, I think that parallelization technology has evolved quickly enough that a simple lack of research inputs has typically been the more important bottleneck to technological development. But if automating R&D allows research inputs in some domain to grow extremely quickly, the technology needed to parallelize them may matter much more.

Why this matters for an intelligence explosion

The simplest model of AI recursive self-improvement goes as follows. Once AI can fully automate AI research, better algorithms let us run more “AI researchers” on the same stock of compute. Those researchers produce still better algorithms, which create still more effective researchers, and so on.

In the conventional model, whether this feedback loop explodes can be inferred from historical growth rates. If intelligence — more precisely, “algorithm quality” in the relevant sense — has grown faster than the research inputs used to produce it, this suggests that the returns to additional research inputs are strong enough to overpower the increasing difficulty of innovation. Thus, once algorithm quality itself multiplies effective research inputs, algorithm quality will advance ever more quickly, and an intelligence explosion will be underway.

Bounded parallelizability introduces another parameter. Recursive self-improvement is now governed not only by

1. how quickly ideas get harder to find and
2. how strongly progress responds to more research inputs,

but also by

3. how quickly advances in technology raise the ceiling on how well research inputs can work in parallel.

There are three possible cases:

Case #1: If parallelization technology improves at least as quickly as effective research inputs, the conventional result survives.

Case #2: If it improves faster than the technological frontier but more slowly than research inputs, an explosion still occurs, but more slowly.

Case #3: If it improves only fast enough to sustain ordinary exponential growth, automating R&D may raise the growth rate without making it accelerate indefinitely.

It’s worth emphasizing that automating AI R&D could also accelerate research, including AI research, through channels other than increasing the number of “simulated researchers”. For example,

• Each instance of the AI system can get more intelligent over time — in terminology of the AI Futures Model, can develop better “research taste” — much more quickly than humans can.
• Unlike human researchers, AI models will presumably not be tempted to leave a frontier lab to start competitors. Some economic growth models suggest that this would facilitate technological development tremendously: see my earlier paper here, Section 4.2.
• A given number of AI model instances could be designed to exhibit much greater cognitive diversity than an equal-sized human research team. Today, to be sure, they exhibit much less, and this is a severe limitation. Even if every instance of Claude Fable 5 were in every sense as smart as Einstein, their biases are highly correlated across instances, and a hundred Einsteins might have taken longer to advance quantum mechanics than an Einstein and a Bohr. But since artificial neural networks are so much more malleable than human brains, investments in fine-tuning and continual learning should eventually allow AI teams to be more cognitively diverse than human teams.

Still, one important channel through which automating AI R&D will accelerate research — the primary channel, in most models of recursive self-improvement to date — is simply that it will allow us, in effect, to greatly increase the researcher population. Parallelizability constraints could greatly limit the effectiveness of this channel in the long run.

Why isn't AI R&D perfectly parallelizable in our universe?

One might think that AI research can already be fully parallelized. Given enough chips, couldn’t we just run every possible experiment simultaneously and select the best result?

No. Even if each parameter were a single bit, a 10-trillion-parameter model would have 2^{(10 trillion)} possible weightings. Using today’s storage technology, merely storing all of them would require a structure so large that light would take vastly longer than the age of the universe to cross it.

More realistic AI R&D contains at least two distinct bottlenecks.

First, experiments have serial depth. Training runs and experiments consist partly of computations that must wait for earlier computations. More compute lets us run more experiment streams at once, but only technological advances, like faster-running chips, would let us compress each stream.

Second, cognitive work must be coordinated. A million AI researchers are useful only insofar as they can generate nonredundant plans, learn from one another’s results, allocate themselves across promising directions, and integrate their work. Parallelization technologies determine how many researchers can be used productively at once — and at any moment, that number is finite. Raising it is itself a research problem, subject to the same constraints as any other.

Automated researchers may be much easier to coordinate than humans. New instances can begin with identical weights and contexts, and they can share code, experiment logs, and machine-readable outputs without years of education or onboarding. The transition from human to machine research could therefore produce a large one-time jump in what I call “agent parallelizability”. But a large jump is not the same as an infinite ceiling, so with a sufficiently fast-growing population of virtual researchers, progress may eventually be limited by the rate at which we can continue improving parallelization technology.

(I will flag that I believe neuralese recurrence and memory and continual learning, if it worked, would probably solve the second constraint, mostly by giving agents what is essentially light-speed telepathy of very complex internal states/allowing non-redundant plans and while I don't expect it in the near term, I do expect it in the longer term, meaning serial depth of experiments matters more, and in the AI 2027 scenario, this is a non-trivial part of why takeoff was so fast.)

Conclusion

The upshot is not that parallelization bottlenecks will necessarily prevent or even delay an intelligence explosion, just that they constitute one of the parameters determining whether and when an explosion occurs. The value of the relevant parameter is an important empirical question, and one that has barely been studied.

In sum, automating R&D will permit a rapid increase in the number of “researchers”. This will doubtless greatly accelerate innovation in many domains, including AI, but innovation will accelerate only to the extent that our ability to parallelize R&D work keeps pace with this growing researcher population. Though parallelization technology has not featured at all in the models of technological development economists and AI forecasters have used to date, it may be central over the years to come.

Sneak peek at the paper (for more, read the paper that accompanied the linkpost)

Whether parallelizability constraints will meaningfully slow AI R&D in particular, once it has been automated, depends on the parameters in a technology network like that outlined at the end of Section 4. Many of these parameters have not yet been estimated, but they are all estimable in principle with the right experiments. One potential approach would be to document how much progress an ever larger number of AI agents can make on a well-defined problem, such as NanoGPT speedrun, in finite time.22 Another, inspired by Goldreich and Wigderson (2020), would be to study the history of algorithmic improvements on (say) NanoGPT speedrun, to learn the extent to which the improvements are separable (in which case they could have been discovered simultaneously) or exhibit strong positive interaction effects (in which case they could probably only have been discovered in sequence). In any case, I hope the framework presented here motivates efforts to study parallelizability more carefully.