A response to "State of play of AI progress (and related brakes on an intelligence explosion)" by Nathan Lambert.

Nathan Lambert recently wrote a piece about why he doesn't expect a software-only intelligence explosion. I responded in this substack comment which I thought would be worth copying here.

As someone who thinks a rapid (software-only) intelligence explosion is likely, I thought I would respond to this post and try to make the case in favor. I tend to think that AI 2027 is a quite aggressive, but plausible scenario.

I interpret the core argument in AI 2027 as:

• We're on track to build AIs which can fully automate research engineering in a few years. (Or at least, this is plausible, like >20%.) AI 2027 calls this level of AI "superhuman coder". (Argument: https://ai-2027.com/research/timelines-forecast)
• Superhuman coders will ~5x accelerate AI R&D because ultra fast and cheap superhuman research engineers would be very helpful. (Argument: https://ai-2027.com/research/takeoff-forecast#sc-would-5x-ai-randd).
• Once you have superhuman coders, unassisted humans would only take a moderate number of years to make AIs which can automate all of AI research ("superhuman AI researcher"), like maybe 5 years. (Argument: https://ai-2027.com/research/takeoff-forecast#human-only-timeline-from-sc-to-sar). And, because of AI R&D acceleration along the way, this will actually happen much faster.
• Superhuman AI researchers can obsolete and outcompete humans at all AI research. This includes messy data research, discovering new paradigms, and whatever humans might be doing which is important. These AIs will also be substantially faster and cheaper than humans, like fast and cheap enough that with 1/5 of our compute we can run 200k parallel copies each at 40x speed (for ~8 million parallel worker equivalents). Because labor is a key input into AI R&D, these AIs will be able to speed things up by ~25x. (Argument: https://ai-2027.com/research/takeoff-forecast#sar-would-25x-ai-randd)
• These speed ups don't stop just above the human range, they continue substantially beyond the human range, allowing for quickly yielding AIs which are much more capable than humans. (AI 2027 doesn't really argue for this, but you can see here: https://ai-2027.com/research/takeoff-forecast#siar-would-250x-ai-randd)

(There are a few other minor components like interpolating the AI R&D multipliers between these milestones and the argument for being able to get these systems to run at effective speeds which are much higher than humans. It's worth noting that these numbers are more aggressive than my actual median view, especially on the timeline to superhuman coder.)

Ok, now what is your counterargument?

Sections (1) and (2) don't seem in conflict with the AI 2027 view or the possibility of a software-only intelligence explosion. (I'm not claiming you intended these to be in conflict, just clarifying why I'm not responding. I assume you have these sections as relevant context for later arguments.)

As far as I can tell, section (3) basically says "You won't get large acceleration just via automating implementation and making things more compute efficient. Also, ML research is messy and thus it will be hard to get AI systems which can actually fully automate this."

AI 2027 basically agrees with these literal words:

• The acceleration from superhuman coders which fully automate research engineering (and run at high speeds) is "only" 5x, and to get the higher 25x (or beyond) acceleration, the AIs needed to be as good as the best ML researchers and fully automate ML research.
• AI 2027 thinks it would take ~5 years of unaccelerated human AI R&D progress to get from superhuman coder to superhuman AI researcher, so the forecast incorporates this being pretty hard.

There is presumably a quantitative disagreement: you probably think that the acceleration from superhuman coders is lower and the unassisted research time to go from superhuman coders to superhuman AI researchers is longer (much more than 5 years?).

(FWIW, I think AI 2027 is probably too bullish on the superhuman coder AI R&D multiplier, I expect more like 3x.)

I'll break this down a bit further.

Importantly, I think the claim:

For machine learning research to accelerate at these rates, it needs to be entirely bottlenecked by compute efficiency and implementation difficulty.

Is probably missing the point: AI 2027 is claiming that we'll get these large multipliers by being able to make AIs which beat humans at the overall task of AI research! So, it just needs to be the case that machine learning research could be greatly accelerated by much faster (and eventually very superhuman) labor. I think the extent of this acceleration and the returns are an open question, but I don't think whether it is entirely bottlenecked by compute efficiency and implementation difficulty is the crux. (The extent to which compute efficiency and implementation difficulty bottleneck AI R&D is the most important factor for the superhuman coder acceleration, but not the acceleration beyond this.)

As far as the difficulty of making a superhuman AI researcher, I think your implicit vibes are something like "we can make AIs better and better at coding (or other easily checkable tasks), but this doesn't transfer to good research taste and intuitions". I think the exact quantitative difficulty of humans making AIs which can automate ML research is certainly an open question (and it would be great to get an overall better understanding), but I think there are good reasons to expect the time frame for unassisted humans (given a fixed compute stock) to be more like 3-10 years than like 30 years:

• ML went from AlexNet to GPT-4 in 10 years! Fast progress has happened historically. To be clear, this was substantially driven by compute scaling (for both training runs and experiments), but nonetheless, research progress can be fast. The gap from superhuman coder to superhuman AI research intuitively feels like a much smaller gap than the gap from AlexNet to GPT-4.
• For success on harder to check tasks, there are hopes for both extending our ability to check and generalization. See this blog post for more discussion. Concretely, I think we can measure whether AIs produce some research insight that ended up improving performance and so we can RL on this, at least at small scale (which might transfer). We can also train on things like forecasting the results of training runs etc. Beyond this, we might just be able to get AIs to generalize well: many of current AI capabilities are due to reasonably far generalization and I expect this to continue.

As far as section (4), I think the core argument is "solving real-world domains (e.g., things which aren't software only) will be hard". You might also be arguing that RL will struggle to teach AIs to be good ML researchers (it's a bit hard for me to tell how applicable your argument is to this).

I think the biggest response to the concern with real world domains is "the superhuman AI researchers can figure this out even if it would have taken humans a while". As far as RL struggling to teach AIs to be good scientists (really ML researchers in particular), I think this is addressed above, though I agree this is a big unknown. Note that we can (and will) explore approaches that differ some from the current RL paradigm and acceleration from superhuman coders might make this happen a bunch faster.

In reality, with the large sums of capital at play, it is unlikely that labs give free rein to billions of dollars of compute to so called "AI researchers in the datacenter" because of how constrained compute is at all of the top labs.

I find this argument somewhat wild if you assume that the AI researchers in the datacenter are making extremely fast progress. Insofar as the software only singularity works, the AI researchers in the datacenter can rapidly improve efficiency using algorithmic progress saving far more compute than you spent to run them. Currently, inference costs for a given level of performance drop by around 9-40x per year depending on the exact task. (See the graph you included.) So, if AIs can make the progress happen much faster, it will easily pay for itself on inference costs alone. That said, I don't expect this is the largest source of returns...

There will be diminishing returns, but if the software only singularity ends up being viable, then these diminishing returns will be outpaced by ever more effective and smarter AI researchers in the datacenter meaning progress wouldn't slow until quite high limits are reached.

I assume your actual crux is about the returns to AI R&D at the point of full (or substantial) automation by AIs. You think the returns are low enough that companies will (rationally) invest elsewhere.

I agree that the fact that AI companies aren't investing all their compute into AI R&D is some update about the returns to algorithms progress, but the regime might differ substantially after full automation by AI researchers!

(And, I think the world would probably be better if AI companies all avoid doing a rapid intelligence explosion internally!)

Minimally, insofar as AI companies are racing as fast as possible, we'd expect that whatever route the companies take is the one they think is fastest! So, if there is another route than AI accelerating AI R&D which is faster, fair enough companies would do this instead, but if the AI automating AI R&D story is insanely fast, then this route must be even faster.