Thomas Larsen's Shortform

Thomas Larsen

This is a special post for quick takes by Thomas Larsen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Here are my largest disagreements with AI 2027.

I think the timelines are plausible but solidly on the shorter end; I think the exact AI 2027 timeline to fully automating AI R&D is around my 12th percentile outcome. So the timeline is plausible to me (in fact, similarly plausible to my views at the time of writing), but substantially faster than my median scenario (which would be something like early 2030s).
I think that the AI behaviour after the AIs are superhuman is a little wonky and, in particular, undersells how crazy wildly superhuman AI will be. I expect the takeoff to be extremely fast after we get AIs that are better than the best humans at everything, i.e., within a few months of AIs that are broadly superhuman, we have AIs that are wildly superhuman. I think wildly superhuman AIs would be somewhat more transformative more quickly than AI 2027 depicts. I think the exact dynamics aren't possible to predict, but I expect craziness along the lines of (i) nanotechnology, leading to things like the biosphere being consumed by tiny self replicating robots which double at speeds similar to the fastest biological doubling times (between hours (amoebas) and months (rabbits)). (ii) extremely superhuman persuasion and political maneuvering, sufficient to let the AI steer policy to a substantially greater extent than it did in AI 2027. In AI 2027, the AI gained enough political power to prevent humans from interacting with ongoing intelligence and industrial explosion (which they were basically on track to do anyways), whereas my best guess is that the AI would gain enough political power to do defacto whatever it wanted, and would therefore result in the AI consolidating power faster (and not keep up the charade of humans being in charge for a period of several years). I also think there are many unknown unknowns downstream of ASI which are really hard to account for in a scenario like AI 2027, but nonetheless are likely to change the picture a lot.
I think the slowdown ending is a bit unrealistic: I think it's unlikely (e.g. maybe 40%) that we get a slowdown of a few months, and I think its unlikely that a few month slowdown is sufficient to avoid misaligned AI takeover (e.g. maybe 30%). I do think a small slowdown would reduce risk, and so is good on the margin, I just don't think it's enough. I also don't think the slowdown ending is what we should be aiming for (and AI 2027 doesn't claim that).

I have various other small disagreements with AI 2027, but, overall, I stand by the bulk of it. I continue to think that the main contribution was sketching out a plausible concrete scenario for how AI takeoff could go where all the parts fit together and it makes sense end-to-end, and I continue to think that the picture is essentially correct. I think the overall sketch is much more important than the specific dates and numbers, which, as we've said many times, are just one hyper specific way things could go. I continue to expect something basically like AI 2027 to happen, except with the above updates and with lots of additional, hard to predict texture and details occuring along the way.

I'm starting to suspect that if 2026-2027 AGI happens through automation of routine AI R&D (automating acquisition of deep skills via RLVR), it doesn't obviously accelerate ASI timelines all that much. Automated task and RL environment construction fixes some of the jaggedness, but LLMs are not currently particularly superhuman, and advancing their capabilities plausibly needs skills that aren't easy for LLMs to automatically RLVR into themselves (as evidenced by humans not having made too much progress in RLVRing such skills).

This creates a strange fu... (read more)

5elifland8mo

Roughly agree. [...] With my median parameters, the AIFM says 1.5 years between TED-AI to ASI. But this isn't taking into account hardware R&D automation, production automation, or the industrial explosion. So maybe adjust that to ~1-1.25 years. However, there's obviously lots of uncertainty. Additionally, conditioning on TED-AI in 2027 would make it faster. e.g., looking at our analysis page, p(AC->ASI <= 1year) conditional on AC in 2027 is a bit over 40%, as opposed to 27% unconditional. So after accounting for this, maybe my median is ~0.5-1 years conditional on TED-AI in 2027, again with lots of uncertainty. There's also a question of whether our definition of ASI, the gap between an ASI and the best humans is 2x greater than the gap between the best humans and the median professional, at virtually all cognitive tasks, would count as wildly superhuman. Probably? Anyway, all this is to say, I think my median is a bit slower than yours by a factor of around 2-4, but your view is still not on the edges of my distribution. For a minimum bar for how much probability I assign to TED-AI->ASI in <=3 months, see on our forecast page that I assign all-things-considered ~15% to p(AC->ASI <=3 months), and this is a lower bound because (a) TED-AI->ASI is shorter, (b) the effects described abobe re: conditioning on 2027. (I'm also not sure what the relationship the result with median parameters has compared to the median of TED-AI to ASI across Monte Carlos which we haven't reported anywhere and I'm not going to bother to look up for this comment.) [...] I tentatively agree, but I don't feel like I have a great framework or world model driving my predictions here. [...] Yeah I think we should have mentioned nanotech. The difference between hours and months is huge though, if it's months then I think we have something like AI 2027 or perhaps slower. [...] I'm not sure it would be able to do whatever it wanted, but I think it at minimum could perform somewhat better

0StanislavKrym8mo

Thanks! My biggest disagreement was the ratio of compute of American and Chinese projects. What I expect is Taiwan invasion causing the projects to slow down and to have the two countries set up compute factories, with a disastrous result of causing OpenBrain and DeepCent to be unable to slow down because the other company would have the capabilities lead. Assuming an invasion in 2027, the median by 2029 would require 10 times (edit: by which I mean 10 times more compute than a counterfactual SC in 2027) more compute which China could be on track to obtain first. Additionally, were Anthropic to keep the lead, Claude's newest Constitution kept unchanged could mean that Claude aligned to it is as unfit for empowering a dictatorship as Agent-4 is unfit to serve any humans.

Thomas Larsen's Shortform

6

Ω 3

Thomas Larsen's Shortform

6

Ω 3