I agree with a lot of this post, but one other motivation (at least for me) is that checking how much algorithmic progress exists/comes from compute is that it's relevant to predictions around recursive self-improvement/software intelligence explosion without increasing compute, assuming you have AIs that can fully automate AI research, and more generally informs takeoff speeds, and I take the Gundleach et al paper as evidence that pure software improvements are more likely to cap out at relatively modest impacts than creating a self-sustaining feedback loop of progress, slowing down AI takeoff by quite a bit until we massively scale up compute for AGIs.
(The evidence value is reduced but not eliminated by the fact that they tested it on LLMs).
The nuance was in saying that their framework can't predict whether or not data or compute scaling made the majority of improvements, nor can they separate out data and compute improvements, but the core finding of algorithmic efficiency being almost all compute-scaling dependent still holds, so if we had a fixed stock of compute now, we would essentially have 0 improvements in AI forever.
Another possibility, in principle, for why automating AI R&D doesn't lead to an intelligence explosion is because a very large percentage of the progress (at that part of development trajectory) is driven by scaling relative to algorithmic progress.
This is actually happening today, so the real question is why algorithmic progress returns will increase once we attempt to fully automate AI R&D, rather than why we won't get an intelligence explosion.
More specifically, the algorithmic progress that has happened is basically all downstream of more compute going into AI, and algorithmic efficiency is dependent on compute scale being larger and larger to reap the gains of better algorithms.
Just LLMs with pretraining, giving superintelligent in-context learning, making AIs able to figure out how to work around their hobblings using just in-context reasoning. This clearly doesn't work at 2024 levels of compute (which only got to be observed in 2025), and after 2026 levels of compute natural text data starts running out. Maybe there are some relevant sparks at 2029-2031 levels of compute (5 GW), but probably not sufficient on its own.
I want to flag that you estimated that post-2028, we'd slow down due to the MoE data wall, and while MoE is more compute efficient, I don't think it's so much more compute efficient that people won't take the hit of less compute efficiency if data efficiency becomes more relevant post-2028, and we can probably continue mostly pre-training up until the early 2030s based on data limitations alone. That said, I agree this is likely insufficient on its own, so I'm not claiming that pure LLM AGI is a likely path.
Continual learning doesn't seem directly relevant, but maybe it helps with teaching LLMs to RLVR LLMs. It doesn't seem directly relevant because it probably doesn't directly train mildly superhuman capabilities. But since the generalizing glue of adaptation would make everything work better, it might enable method (2) to go forward where it would fail on its own. The relevant benchmarks might be the same as for (2), but only if they don't already saturate by the time there's useful continual learning (in some nontrivial sense of the term).
I think continual learning is more likely to end up important in its impact on AI company TAM. If it manages to 10x it, then it will thereby 10x the compute, though it'll proably take until 2030-2035 to actually build the kind of research/training compute (20-50 GW) that a trillion dollars of revenues buys (even if 2026 already shows that this is on track to happen).
I think this is probably true for the near term (as in 5-10 year timelines), but over a 15-20 year span with Moore's law, I'd be more confident that continual learning alone could plausibly train in mildly superhuman capabilities, for the same reasons why certain researchers IRL are way more productive than others.
This does depend on human level or better sample efficient learning to work out, though.
That said, I agree with your methodology much more than Daniel Kokotajlo's methodology on AI prediction (though I do think METR time horizons are more useful and less anchored to specific paradigms than you say it is).
It's actually worse, in that almost all such simulators are hypercomputational and have literally infinite compute, memory and data, as well as their programs being infinitely large, so it's literally useless to think/write with potential simulators as an audience.
1The answer I suspect has to do with a focus on short timelines combined with more bearish views on software-only singularities than most of LW.
Short timelines means there's a lot less compute resources to run AIs because Moore's law hasn't had as much of an effect compared to future dates, combined with less compute being built overall, meaning we can only run millions or at best tens of millions of AIs.
One thing I note also is that he does think that AIs can increase their intelligence, but that this runs into steep diminishing returns early enough that other factors matter more.
Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do.
The bigger issue, as Jackson Wagner says is that there's a very real risk that will be coopted by people who want to talk mostly about present-day harms of AI, and thus at best siphoning resources from actually useful work on gradual disempowerment threats/AI x-risk in general, and at worst creating polarization around gradual disempowerment with one party supporting gradual disempowerment of humans and another party opposing gradual disempowerment, while the anti-gradual disempowerment party/people become totally ineffective at dealing with the problem because it's been taken over by omnicause dynamics.
To be honest, I now think the post has far less value than people thought in in 2024, and this is not because it's factual statements are wrong, but rather that Richard Ngo and a lot of LWers way underestimated how intractable it was to avoid polarization of AI safety issues in worlds where AI safety is highly salient politically without deep changes to our electoral systems.
(To be clear, this is a US-centric view, but this is justified given that the by far most likely countries to have ASI in their borders initially are the US and China, due to both compute and researcher advantages over the rest of the world, and the US's laws are more tractable to change for AI safety than China's laws on AI).
The basic reason for this is that social sorting has fundamentally led to the breakdown of conditions that allowed low polarization without something like proportional voting/score voting, and this is a problem that AI safety people are ill-equipped to solve or even ameliorate significantly, unfortunately.
And this means absent electoral reform, we should assume by default as AI safety becomes higher salience, it will become more polarized, and this means there's a tradeoff between making AI higher salience to achieving political goals.
I'm giving it a 0 for this reason, because in the end the post was basically irrelevant to how AI safety should act, and how to handle the tradeoff of polarization vs getting enough salience to get laws passed is unfortunately more complicated than the post makes it out to be, and much tougher to solve.
I agree with this, but at that point philosophical dilemmas don't matter for alignment purposes, because from an alignment perspective you only need to (ideally) have agreement between the AI and human on the answers, and the answers to philosophical dilemmas don't actually matter (at least for value-laden philosophical dilemmas).
I agree with Eliezer on the hard part of the alignment problem:
So philosophy being hard is not a valid argument for AI alignment being hard.
That said, I do think philosophical dilemmas like this (as well as trying to steer us towards moral-trade futures) is a big reason I've come to think of space governance as important, neglected and (for now) much more tractable than basically everything else with the same level of importance or higher.
More generally, one very important priority is to move ourselves away from a regime where whoever makes a unilateral claim and seizes land in space owns it, and we will need to prohibit interstellar settlement for a time (because otherwise due to light-speed limitations, it becomes effectively impossible for the central government to catch up with them).
Thankfully, this is far easier than ending the AI race for now.
Link to long comments that I want to pin, but are too long to be pinned:
https://www.lesswrong.com/posts/Zzar6BWML555xSt6Z/?commentId=aDuYa3DL48TTLPsdJ
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/?commentId=Gcigdmuje4EacwirD
https://www.lesswrong.com/posts/DCQ8GfzCqoBzgziew/?commentId=RhTNmgZqjJpzGGAaL
https://www.lesswrong.com/posts/NjzLuhdneE3mXY8we?commentId=rcr4j9XDnHWWTBCTq