This is a special post for quick takes by ryan_greenblatt. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
*Ω29440
I put roughly 50% probability on feasibility of software-only singularity.[1]
(I'm probably going to be reinventing a bunch of the compute-centric takeoff model in slightly different ways below, but I think it's faster to partially reinvent than to dig up the material, and I probably do use a slightly different approach.)
My argument here will be a bit sloppy and might contain some errors. Sorry about this. I might be more careful in the future.
The key question for software-only singularity is: "When the rate of labor production is doubled (as in, as if your employees ran 2x faster[2]), does that more than double or less than double the rate of algorithmic progress? That is, algorithmic progress as measured by how fast we increase the labor production per FLOP/s (as in, the labor production from AI labor on a fixed compute base).". This is a very economics-style way of analyzing the situation, and I think this is a pretty reasonable first guess. Here's a diagram I've stolen from Tom's presentation on explosive growth illustrating this:
Basically, every time you double the AI labor supply, does the time until the next doubling (driven by algorithmic progress) increase (fizzle) or decrease (foom)? I'm being a bit sloppy in saying "AI labor supply". We care about a notion of parallelism-adjusted labor (faster laborers are better than more laborers) and quality increases can also matter. I'll make the relevant notion more precise below.
I'm about to go into a relatively complicated argument for why I think the historical data supports software-only singularity. If you want more basic questions answered (such as "Doesn't retraining make this too slow?"), consider looking at Tom's presentation on takeoff speeds.
Here's a diagram that you might find useful in understanding the inputs into AI progress:
And here is the relevant historical context in terms of trends:
- Historically, algorithmic progress in LLMs looks like 3-4x per year including improvements on all parts of the stack.[3] This notion of algorithmic progress is "reduction in compute needed to reach a given level of frontier performance", which isn't equivalent to increases in the rate of labor production on a fixed compute base. I'll talk more about this below.
- This has been accompanied by increases of around 4x more hardware per year[4] and maybe 2x more quality-adjusted (parallel) labor working on LLM capabilities per year. I think total employees working on LLM capabilities have been roughly 3x-ing per year (in recent years), but quality has been decreasing over time.
- A 2x increase in the quality-adjusted parallel labor force isn't as good as the company getting the same sorts of labor tasks done 2x faster (as in, the resulting productivity from having your employees run 2x faster) due to parallelism tax (putting aside compute bottlenecks for now). I'll apply the same R&D parallelization penalty as used in Tom's takeoff model and adjust this down by a power of 0.7 to yield 1.6x per year in increased labor production rate. (So, it's as though the company keeps the same employees, but those employees operate 1.6x faster each year.)
- It looks like the fraction of progress driven by algorithmic progress has been getting larger over time.
- So, overall, we're getting 3-4x algorithmic improvement per year being driven by 1.6x more labor per year and 4x more hardware.
So, the key question is how much of this algorithmic improvement is being driven by labor vs. by hardware. If it is basically all hardware, then the returns to labor must be relatively weak and software-only singularity seems unlikely. If it is basically all labor, then we're seeing 3-4x algorithmic improvement per year for 1.6x more labor per year, which means the returns to labor look quite good (at least historically). Based on some guesses and some poll questions, my sense is that capabilities researchers would operate about 2.5x slower if they had 10x less compute (after adaptation), so the production function is probably proportional to (). (This is assuming a cobb-douglas production function.) Edit: see the derivation of the relevant thing in Deep's comment, my old thing was wrong[5].
Now, let's talk more about the transfer from algorithmic improvement to the rate of labor production. A 2x algorithmic improvement in LLMs makes it so that you can reach the same (frontier) level of performance for 2x less training compute, but we actually care about a somewhat different notion for software-only singularity: how much you can increase the production rate of labor (the thing that we said was increasing at roughly a rate of 1.6x per year by using more human employees). My current guess is that every 2x algorithmic improvement in LLMs increases the rate of labor production by , and I'm reasonably confident that the exponent isn't much below . I don't currently have a very principled estimation strategy for this, and it's somewhat complex to reason about. I discuss this in the appendix below.
So, if this exponent is around 1, our central estimate of 2.3 from above corresponds to software-only singularity and our estimate of 1.56 from above under more pessimistic assumptions corresponds to this not being feasible. Overall, my sense is that the best guess numbers lean toward software-only singularity.
More precisely, software-only singularity that goes for >500x effective compute gains above trend (to the extent this metric makes sense, this is roughly >5 years of algorithmic progress). Note that you can have software-only singularity be feasible while buying tons more hardware at the same time. And if this ends up expanding compute production by >10x using AI labor, then this would count as massive compute production despite also having a feasible software-only singularity. (However, in most worlds, I expect software-only singularity to be fast enough, if feasible, that we don't see this.) ↩︎
Rather than denominating labor in accelerating employees, we could instead denominate in number of parallel employees. This would work equivalently (we can always convert into equivalents to the extent these things can funge), but because we can actually accelerate employees and the serial vs. parallel distinction is important, I think it is useful to denominate in accelerating employees. ↩︎
I would have previously cited 3x, but recent progress looks substantially faster (with DeepSeek v3 and reasoning models seemingly indicating somewhat faster than 3x progress IMO), so I've revised to 3-4x. ↩︎
This includes both increased spending and improved chips. Here, I'm taking my best guess at increases in hardware usage for training and transferring this to research compute usage on the assumption that training compute and research compute have historically been proportional. ↩︎
Edit: the reasoning I did here was off. Here was the old text: so the production function is probably roughly (). Increasing compute by 4x and labor by 1.6x increases algorithmic improvement by 3-4x, let's say 3.5x, so we have , . Thus, doubling labor would increase algorithmic improvement by . This is very sensitive to the exact numbers; if we instead used 3x slower instead of 2.5x slower, we would have gotten that doubling labor increases algorithmic improvement by , which is substantially lower. Obviously, all the exact numbers here are highly uncertain. ↩︎
2