Will AI R&D Automation Cause a Software Intelligence Explosion?

Daniel_Eth

For future work on the software intelligence explosion, I'd like to see 2 particular points focused on here, @Tom Davidson:

1 is estimating the complementarity issue, and more generally pinning down the rho number for software, because whether complementary or substitution effects dominate during the lead up to automating all AI R&D is a huge factor in whether an intelligence explosion is self-sustaining.

More from Tamay Besiroglu and Natalia Coelho here:

https://x.com/tamaybes/status/1905435995107197082

https://x.com/natalia__coelho/status/1906150456302432647

Tamay Besiroglu: One interesting fact is that there’s a strong coincidence in the rate of software progress and hardware scaling both pre- and post the deep learning era in also in other domains of software. That observation seems like evidence of complementarities.
Natalia Coelho: (Just to clarify for those reading this thread) “0.45-0.87” is this paper’s estimate for sigma (the elasticity of substitution) not rho (the substitution parameter). So this indicates complementarity between labor and capital. The equivalent range for rho is -1.22 to -0.15

Thus, we can use the -1.22 to -0.15 as a base rate for rho for software, and argue for that number being up or down based on good evidence/arguments for the number being higher or lower.

My second ask is to get more information on the value of r, which is the returns on software, because right now the numbers you have gotten are way too uncertain for it to be much use (especially for predicting how fast an AI can improve), and I'd like to see more work on estimating the parameter r using many sources of data.

A question: How hard is it to actually figure out if compute scaling is driving most of the progress, compared to algorithms.

You mentioned it's very hard to decompose the variables, but is it the sort of thing you'd need like a 6 month project for, or would we just have to wait several years for things to play out, because if it could be predicted before it happens, it would be very, very valuable evidence.

Tweet below:

https://x.com/TomDavidsonX/status/1905905058065109206

Tom Davidson: Yeah would be great to tease this apart. But hard: - Hard to disentangle compute for experiments from training+inference. - V hard to attribute progress to compute vs researchers when both rise together

Summary

AI companies are increasingly using AI systems to accelerate AI research and development. These systems assist with tasks like writing code, analyzing research papers, and generating training data. While current systems struggle with longer and less well-defined tasks, future systems may be able to independently handle the entire AI development cycle – from formulating research questions and designing experiments, to implementing, testing, and refining new AI systems.

Some analysts have argued that such systems, which we call AI Systems for AI R&D Automation (ASARA), would represent a critical threshold in AI development. The hypothesis is that ASARA would trigger a runaway feedback loop: ASARA would quickly develop more advanced AI, which would itself develop even more advanced AI, resulting in extremely fast AI progress – an “intelligence explosion.”

Skeptics of an intelligence explosion often focus on hardware limitations – would AI systems be able to build better computer chips fast enough to drive such rapid progress? However, there’s another possibility: AI systems could become dramatically more capable just by finding software improvements that significantly boost performance on existing hardware. This could happen through improvements in neural network architectures, AI training methods, data, scaffolding around AI systems, and so on. We call this scenario a software intelligence explosion (SIE). This type of advancement could be especially rapid, since it wouldn’t be limited by physical manufacturing constraints. Such a rapid advancement could outpace society’s capacity to prepare and adapt.

In this report, we examine whether ASARA would lead to an SIE. First, we argue that shortly after ASARA is developed, it will be possible to run orders of magnitude more automated AI researchers than the current number of leading human AI researchers. As a result, the pace of AI progress will be much faster than it is today.

Second, we use a simple economic model of technological progress to analyze whether AI progress would accelerate even further. Our analysis focuses primarily on two countervailing forces. Pushing towards an SIE is the positive feedback loop from increasingly powerful AI systems performing AI R&D. On the other hand, improvements to AI software face diminishing returns from lower hanging fruit being picked first – a force that pushes against an SIE.

To calibrate our model, we turn to empirical data on (a) the rate of recent AI software progress (by drawing on evidence from multiple domains in machine learning and computer science) and (b) the growing research efforts needed to sustain this progress (proxied by the number of human researchers in the field). We find that (a) likely outstrips (b) – i.e., AI software is improving at a rate that likely outpaces the growth rate of research effort needed to achieve these software improvements. In our model, this finding implies that the positive feedback loop of AI improving AI software is powerful enough to overcome diminishing returns to research effort, causing AI progress to accelerate further and resulting in an SIE.

If such an SIE occurs, the first AI systems capable of fully automating AI development could potentially create dramatically more advanced AI systems within months, even with fixed computing power.

We examine two major obstacles that could prevent an SIE: (1) the fixed amount of computing power limits how many AI experiments can be run in parallel, and (2) training each new generation of AI system could take months. While these bottlenecks will slow AI progress, we find that plausible workarounds exist which may allow for an SIE nonetheless. For example, algorithmic improvements have historically increased the efficiency of AI experiments and training runs, suggesting that training runs and experiments could be progressively sped up, enabling AI progress to continually accelerate despite these obstacles.

Finally, because such a dramatic acceleration in AI progress would exacerbate risks from AI, we discuss potential mitigations. These mitigations include monitoring for early signs of an SIE and implementing robust technical safeguards before automating AI R&D.

LESSWRONG
LW

LESSWRONG
LW

19

Will AI R&D Automation Cause a Software Intelligence Explosion?

19

Ω 10

19

Ω 10

Summary