This is a special post for quick takes by Eli Tyre. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I'm mostly going to use this to crosspost links to my blog for less polished thoughts, Musings and Rough Drafts.
Both incremental improvements (compute scaling and small to moderate algorithmic improvements), and breakthroughs (such as significant "unhobblings" compared to human faculties) advance capabilities. Which of them is the last step that crosses the threshold to recursive self-improvement at AI speed could even be historically contingent, let alone knowable in advance.
So instead we can quantify the rate of incremental improvements (anticipating future conditions that change it), and point out specific important incapabilities of current AIs that might be targeted by possible breakthroughs. For incremental improvements, the funding (and then the industrial base) for the current super-fast compute scaling will be running out in 2027-2029 (absent AGI), and so the compute scaling itself will slow down (by about 3x) after 2028-2030. This will in turn also slow down incremental algorithmic improvements (a bit later still), because they need compute for experiments and a non-increasing amount of compute will lead to low-hanging fruit getting picked and new low-hanging fruit not showing up from the independent source of greater compute scale. So if it's 2029-2032, compute stopped rapidly scaling 2-3 years ago, and AIs still can't do recursive self-improvement at AI speed, then the factor of incremental improvements becomes less important than it is now.
For the current reasoning LLMs, an important incapability is that they can't (at all) deeply adapt to specific roles or sources of tasks, they are always "first day on the job", even if they have relevant professional skills and copious notes from previous efforts. On a recent podcast, Dwarkesh Patel says that Sutskever's SSI is rumored to be working on "test time training" (at 39:25). Another reason to think this "unhobbling" is plausible soon is that it might turn out to be possible to use agentic (tool-using) RLVR to train AIs to prepare datasets for finetuning variants of themselves (not necessarily with RLVR) that will then do better at particular tasks. So if 2-3 years pass and no version of this happens, it might stop being an important candidate for the next plausible breakthrough. And after at least something with potential is done in this direction, it will also fall under the influence of incremental improvements.
I'm mostly going to use this to crosspost links to my blog for less polished thoughts, Musings and Rough Drafts.