Evidence that would update me towards a software-only fast takeoff

Anders Woodruff

In a software-only takeoff, AIs improve AI-related software at an increasing speed, leading to superintelligent AI. The plausibility of this scenario is relevant to questions like:

How much time do we have between near-human and superintelligent AIs?
Which actors have influence over AI development?
How much warning does the public have before superintelligent AIs arrive?

Knowing when and how much I expect to learn about the likelihood of such a takeoff helps me plan for the future, and so is quite important. This post presents possible events that would update me towards a software-only takeoff.

What are returns to software R&D?

The key variable determining whether software progress alone can produce rapid, self-sustaining acceleration is returns to software R&D (r), which measures how output scales with labor input. Specifically, if we model research output as:

where O is research output (e.g. algorithmic improvements) and I is the effective labor input (AI systems weighted by their capability), then r captures the returns to scale.

If r is greater than 1, doubling the effective labor input of your AI researchers produces sufficient high-quality research to more than double the effective labor of subsequent generations of AIs, and you quickly get a singularity, even without any growth in other inputs. If it's less than 1, software improvements alone can't sustain acceleration, so slower feedback loops like hardware or manufacturing improvements become necessary to reach superintelligence, and takeoff is likely to be slower.

Projected software capacity growth under different returns-to-scale assumptions, holding hardware constant. ASARA is AI Systems for AI R&D Automation. When r > 1, each generation of AI researchers produces more than enough capability gain to accelerate the next generation, yielding explosive growth (red, purple). At r = 1 (orange), gains compound but don't accelerate. When r < 1 (green, blue), diminishing returns cause growth to asymptotically approach the dashed baseline, making hardware or other bottleneck improvements necessary for continued acceleration. From Forethought.

A software-only singularity could be avoided if r is not initially above 1, or if r decreases over time, for example, because research becomes bottlenecked by compute, or because algorithmic improvements become harder to find as low-hanging fruit is exhausted.

Initial returns to software R&D

The most immediate way to determine if returns to software R&D are greater than 1 would be observing shortening doubling times in AI R&D at major labs (i.e. accelerating algorithmic progress), but it would not be clear how much of this is because of increases in labor rather than (possibly accelerating) increases in experimental compute. This has stymied previous estimates of returns.

Posterior distributions of returns to software R&D (r) across four domains. Only SAT solvers have a 90% confidence interval entirely above 1. From Epoch AI.

Evidence that returns to labor in AI R&D are greater than 1:

Progress continues to accelerate after chip supplies near capacity constraints. This would convince me that a significant portion of continued progress is a result of labor rather than compute and would constitute strong evidence.
Other studies show that labor inputs result in compounding gains. This would constitute strong evidence.
1. Any high-quality randomized or pseudorandom trial on this subject.
2. Work that effectively separates increased compute from increased labor input ^[1].
Labs continue to be able to make up for less compute than competitors with talent (like Anthropic in recent years). This would be medium-strength evidence.
A weaker signal would be evidence of large uplifts from automated coders. Pure coding ability is not very indicative of future returns, however, because AIs’ research taste is likely to be the primary constraint after full automation.
1. Internal evaluations at AI companies like Anthropic show exponentially increasing productivity.
2. Y Combinator startups grow much faster than previously (and increasingly fast over time). This is likely to be confounded by other factors like overall economic growth.

Compute bottlenecks

The likelihood of a software-only takeoff depends heavily on how compute-intensive ML research is. If progress requires running expensive experiments, millions of automated researchers could still be bottlenecked. If not, they could advance very rapidly.

Here are some things that would update me towards thinking little compute is required for experiments:

Individual compute-constrained actors continue to make large contributions to algorithmic progress^[2]. This would constitute strong evidence. Examples include:
1. Academic institutions which can only use a few GPUs.
2. Chinese labs that are constrained by export restrictions (if export restrictions are reimposed and effective).
Algorithmic insights can be cross-applied from smaller-scale experimentation. This would constitute strong evidence. For example:
1. Optimizers developed on small-scale projects generalize well to large-scale projects^[3].
2. RL environments can be iterated with very little compute.
Conceptual/mathematical work proves particularly useful for ML progress. This is weak evidence, as it would enable non-compute-intensive progress only if such work does not require large amounts of inference-time compute.

Diminishing returns to software R&D

Even if returns on labor investment are compounding at the beginning of takeoff, research may run into diminishing returns before superintelligence is produced. This would result in the bumpy takeoff below.

Three intelligence explosion/takeoff scenarios. In the rapid scenario, a software-only takeoff reaches a singularity. In the bumpy scenario, software-only takeoff stalls until AI can improve hardware and other inputs. In the gradual scenario, meaningful capability gains only occur once AI can augment the full stack of inputs to production. From Forethought.

The evidence I expect to collect before takeoff is relatively weak, because current progress rates don't tell us much about the difficulty of discovering more advanced ideas we haven't yet tried to find. That said, some evidence might be:

Little slowdown in algorithmic progress in the next few years. Evidence would include:
1. Evidence of constant speed of new ideas, controlling for labor. Results from this type of analysis that don’t indicate quickly diminishing returns would be one example.
2. Constant time between major architectural innovations (e.g. a breakthrough in 2027 of similar size to AlexNet, transformers, and GPT-3)^[4].
3. New things to optimize (like an additional component to training, e.g. RLVR).
4. Advances in other fields like statistics, neuroscience, and math that can be transferred with some effort. For example:
  1. Causal discovery algorithms that let models infer causal structure from observational data.
We have evidence that much better algorithms exist and could be implemented in AIs. For example:
1. Neuroscientific evidence of the existence of much more efficient learning algorithms (which would require additional labor to identify).
2. Better understanding of how the brain assigns credit across long time horizons.

Conclusion

I expect to get some evidence of the likelihood of a software-only takeoff in the next year, and reasonably decisive evidence by 2030. Overall I think evidence of positive feedback in labor inputs to software R&D would move me the most, with evidence that compute is not a bottleneck being a near second.

Publicly available evidence that would update us towards a software-only singularity might be particularly important because racing companies may not disclose progress. This evidence is largely not required by existing transparency laws, and so should be a subject of future legislation. Evidence of takeoff speeds would also be helpful for AI companies to internally predict takeoff scenarios.

Thanks for feedback from other participants in the Redwood futurism writing program. All errors are my own.

^{^}
This paper makes substantial progress but does not fully correct for endogeneity, and its 90% confidence intervals straddle an r of 1, the threshold for compounding, in all domains except SAT solvers.
^{^}
It may be hard to know if labs have already made the same discoveries.
^{^}
See this post and comments for arguments about the plausibility of finding scalable innovations using small amounts of compute.
^{^}
This may only be clear in retrospect, since breakthroughs like transformers weren't immediately recognized as major.

Software-only singularity is about superintelligence, about getting qualitatively smarter than humanity, and plausibly depends on there being a cognitive factor of production that humanity is almost unable to scale or accumulate, but AIs can. Looking at how humans are doing on that factor prior to its scaling getting unlocked for AIs wouldn't be useful. Knowing how long it took for COVID-19 to start, counting from some arbitrary prior year like 2010, when it didn't exist yet, won't tell you how quickly the infection will spread once it exists (starts scaling).

This could be just sample efficiency, being able to come up with good designs or theories with much less feedback (experiments, including compute-expensive ones, or prior theory). But it could also be things like training novel cognitive skills in themselves that are not directly useful, but build up over time like basic science to produce something much more effective a million steps later. Or automated invention of conceptual theory (rather than proving technical results in framings humans have already come up with language for) at such a speed that it would've taken humans 1000 years to get there (without experiments), so that anything you might observe over the next 5 years about human progress would be utterly uninformative about how useful orders of magnitude more of theoretical progress would be for AI design.

I agree that late in the singularity, AI workflows may be so different from humans' that we learn very little from extrapolating from human returns to software R&D, but I expect that early in the takeoff, AIs may look significantly more like large-scale human labor (especially if they are still largely managed by humans). If existing returns are insufficient for a takeoff, that should update us against a software-only takeoff in general because it makes initial compounding less likely.

I also expect to observe relevant returns in the near future as AIs increasingly automate AI R&D (many of the points in the above post would include this). Early automation may give us some evidence on dynamics mid-takeoff.

My point is more that if there isn't some hidden cognitive bottleneck (that humans plausibly don't perceive as such since it was always there, relatively immutable), then there wouldn't be a software-only singularity, it would take long enough that significantly more hardware will have time to come online even after AIs can act as human replacements in the labor market. There will be returns according to factors that can be tracked earlier, but they won't produce a superintelligence and take over the trees and such before tens of trillions of dollars in AI company revenues buy 100x more compute that's now designed by these AIs but still largely follows existing methods and trends.

early in the takeoff, AIs may look significantly more like large-scale human labor (especially if they are still largely managed by humans). If existing returns are insufficient for a takeoff, that should update us against a software-only takeoff

In my model, the relevant part of a software-only takeoff only starts once the AIs become capable of accumulating or scaling these cognitive factors of production that can't be notably scaled for humanity in the relevant timeframes. Thus observing humanity, or transfering learnings across the analogy between human labor and early AI labor, won't inform these (plausibly originally hidden and unknown) cognitive factors. Only looking at the AIs that do start scaling or accumulating these factors becomes informative.

Even worse, if early human labor replacement capable AIs don't trigger software-only singularity, it doesn't mean some later advancement won't. So all that can be observed is that software-only singularity doesn't start for greater and greater levels of capability, each advancement can only rule out that it's sufficient, but it can't rule out that the next one might be sufficient. The probability goes down, but plausibly it should take a while for it to go way down, and at that point a possible singularity won't be clearly software-only anymore.

Would it be useful to consider how much of current progress has been due to algorithmic improvements rather than compute capacity? It seems that the trend so far has been that a significant proportion of capability improvement has been software-based, though whether that can continue for many more orders of magnitude is certainly debatable.