I am freaked out, but not because I think the LLMs will get upgraded "directly" to ASI in something approximating their current form. I'm freaked out because I think LLM coding & research assistants will get good enough to radically accelerate R&D progress into new conceptual territory and we will discover Stephen Byrnes' Brain In a Box.
I'm getting more bearish on that take as well.
- LLMs don't seem to be getting much better at directly helping with non-paradigmic/conceptually novel research by directly generating/synthesizing insights (this post seems roughly right). They're good as rubber ducks and as a better Google/Wikipedia combo, but IMO their utility there is already near saturation.
- Regarding indirect help: I've struggled to imagine any ways to slot them into my research workflow that would significantly speed me up, and e. g. @Steven Byrnes seems to report the same. I can see a bunch of ways they may reduce friction or help with various literature reviews, but productivity gains there are limited.
- Regarding math: I guess they may help here?
- Naively, if you can phrase the problem as a formal mathematical conjecture of the sort LLMs may legitimately become superhuman at solving, you've already done 90% of the conceptual work...
- ... But perhaps there are some speed-ups you can get by just blindly spamming conjectures at them until you stumble upon a formulation that reveals rich structures? Or other quantity-is-a-quality-of-its-own unlocks that would become possible once formal proofs are cheap?
- Regarding coding, if your goal is conceptual insight, do you really need a bespoke 100k LoC codebase for running your experiments or whatever, as opposed to a few quick-and-dirty back-of-the-napkin math sketches plus a 300-LoC Python script?
- Note that if LLMs remain as terrible at direct conceptual work as they are now (which is necessarily the case in the LLMs-don't-scale-to-general-ASI timeline), you'd either need to manually produce and "inject" 100k-LoCs' worth of novel insight into your codebase, or the codebase is going to be 100k lines of boilerplate slop that probably could be mostly replaced by a 300-LoC script with a few package imports.
- Another possible place where LLMs may help significantly is the intersection of math and coding, where they translate your mathematical algorithm into a runnable, computationally efficient form, see Step 4 here. I'm personally very excited about this possibility, I hate Step 4. Again, though, this is probably after most of the conceptual work has been done manually, and the load-bearing steps of finding a computationally efficient form may themselves require manually derived insights.
I expect that there are ways to significantly speed up non-paradigmic research via LLMs, I have some ideas on how... But I think this task is mostly itself bottlenecked on novel, likely counterfactual conceptual insights. So I expect the requisite tools just won't be built.
Overall, I do expect speed-ups from all of this, but on the order of 1.5x-2x, not 10x.
The fundamental problem (or "problem") is that, in the LLMs-don't-scale-to-ASI timeline, everything interesting is locked behind conceptual insights, and there are no known ways to produce those except "a determined human with a headache stares at a wall for 1,000 hours". LLMs can add bells and whistles to that, make the wallpaper have dynamic RGB lighting, but short of advanced brain-computer interfaces, the core bottleneck would remain. (Part of me hates it, staring at a wall with a headache is often unfun, but I guess I'd take that over the alternative.[1])
I may be very wrong about this, though. (Also, this obviously doesn't consider impacts on the economy and the balance of power, just the raw time-to-general-superintelligence.)
(Edit: Also, the remaining amount of total conceptual work to general ASI may not be large, so this isn't a "we're decades away" take, just an "LLMs won't speed it up by orders of magnitude compared to the non-LLM counterfactual" take.)
- ^
I guess one other non-obvious avenue by which they may cause speed-ups is by making the process more pleasant/addictive, the way Claude Code does coding for some people, thereby increasing the amount of human-hours going into it? But I expect the top candidates for cracking the problem are already psychotically obsessed with it and have it as the top idea in their mind 24/7, and the ones who would go for it only after it becomes pleasant would be mostly led astray in the manner described here.
1LLMs don't seem to be getting much better at directly helping with non-paradigmic/conceptually novel research by directly generating/synthesizing insights (this post seems roughly right). They're good as rubber ducks and as a better Google/Wikipedia combo, but IMO their utility there is already near saturation.
Just curious: Have you been trying and failing? Or not trying? (It’s the latter for me—I don’t use LLMs as a “thought partner” for novel conceptual research, but also haven’t really tried enough to have a first-hand opinion.)
I frequently rubber-duck at them when thinking through some novel research threads. I find them useful for (a) providing "pseudosocial" intellectual stimulation, (b) bringing up relevant concepts known at large but not to me, and sometimes for (c) sanity-checking me.
The bulk of the value in (c) is in the act of my preparing my argument to be presented to something-that-feels-like-an-intelligent-entity, however. This, by itself, activates different instincts/heuristics in me, making me spot holes in the argument or clarify it in ways I wouldn't have otherwise. The LLM's actual response afterwards is mostly irrelevant; I often only skim it or even dismiss the supposed holes it identifies as red herrings.
What I don't find them useful for is (d) developing the new research thread in new directions. Intuitively, their default behavior there is to look for safety: "pulling back" from speculative directions, "rounding the idea off" to something known, introducing a bunch of caveats that muddle things instead of committing to a strong simplifying assumption, etc. You can of course get away from this default behavior by being sufficiently confident and using big words arranged in clever-seeming arguments... but I'm pretty sure I'd be able to talk them into anything this way, it's the "AI psychosis" basin.
I admittedly haven't tried to see if they could do novel conceptual research fully autonomously, not since a while ago. Trying this again is a low-priority task on my to-do list, but I don't expect much.
1Humans have lots and lots of training data to build on within imitiation learning and culture which one can get a wrong view of when reading steven byrnes imo. He has a very specific focus on reward learning infrastructure which means he skips out on reading some of the cultural evolution literature.
The important point here is basically that the human language corpus is really op for world models and that I still think that there's a relatively large difference between a RL bootstrapped system compared to an LLM, I think you get a lot of bang for your buck by training on the human language corpus.
So I think there will be a sort of language model anyway but probably a weird version of one.
Honestly, I'm not updating my own internal worldview based on the METR time horizon, I'm just pointing to it as some form of external sign of my internal beliefs. My experience is updated more by the fact that I have been saving up lots of roughly-sketched-out alignment project ideas in anticipation of the period of 'alignment researchers get sped up by LLM assistants.'
I've been trying to scaffold every new Gemini and Claude model into helping with these projects since Claude 3 Opus. Now, for the first time, it's actually starting to work with Opus 4.6. Still lots of errors, so it's not really like an independent agent that can do it end-to-end, but nevertheless a dramatic speed-up.
I'm also quite worried about near-term risks that don't even come from accelerating ASI, as there's lots of dangerous stuff that unlocks with better automation between here and ASI, including paperclip-maximizer-style agents that, while perhaps restricted to digital operations, could have devastating economic and military effects, not to mention capacity for use in developing biological and military threats. There's real risk we don't even make it to ASI because the transition period is deadly enough.
1
1I buy something about 80% being a meaningful metric. There are still a wide variety of software tasks I can't trust Opus 4.6 to do autonomously, and in fact, it's unreliably enough that I still need to manually review every line of code. To me the obvious breakpoint is when it's reliable enough that there's no need for human code review to reliably achieve desired outputs, or at least minimal code review with automated code review agents filling the gaps and consistently identify a more limited amount of code that does require human review.
I should note, though, that 4.6 feels capable like a junior engineer in a way that 4.5 and earlier models didn't.
My opinion is that the 50% result is obviously noise downstream of METR's current task suite getting saturated (Opus 4.6 is not that much better than Opus 4.5, like, come on), just like METR itself warns, and people's reactions to this are a good way to tell who is actually trying to track reality and who is just blindly reposting short-timelines memes. (Not a snipe at this post, to be clear, it's doing an okay job pointing out possible problems with this result.)
having read the full ai2027 update and their backing doc explaining their reasoning, I actually think it's very clear they're overestimating both the difficulty (horizon + reliability required) and necessity of "Automated Coder".
I agree that the 50% benchmark is less useful and as stated by METR, more noisy. I do think its worth pointing out that while the 80% bench mark is basically on trend: 1) that trend has been faster since o3 release. though that could just be a general uptick and constant bias against the current trendline; 2) we are now looking at a model that can do hour long tasks at 80% success. At the low end of Cotra's estimate of 3 months from https://www.aifuturesmodel.com/, that gives an automated coder by December 2027. Playing with the bounds (2 days - 3.3 years) of the coding time horizon gives a range of November 2026-December 2027. So a year range for automated coder. In the most modest estimation, hour long tasks at 80% success (which has much tighter CIs than 50%), will speed up researchers by 12% for a standard work day if they just used it for a single task they need to do let alone parallelize it. 3) I think its also worth pointing out the difference between claude code and codex. chatgpt 5.3 is a lot slower than claude opus 4.6. Only openai had access to the fastest versions of chatgpt 5.3. I wonder how much having a model that is not only more capable but gives a tighter REPL affects things going forward. Test time compute being long can have impacts!
Another day, another METR graph update.
METR said on X:
Some people are saying this makes superexponential progress more likely.
Forecaster Peter Wildeford predicts 2-3.5 workweek time horizons by end of year which would have "significant implications for the economy".
Even Ajeya Cotra (who works at METR) is now saying that her predictions from last month are too conservative and 3-4 month doubling time with superexponential progress is more likely.
Should We All Freak Out?
People are especially concerned when looking at the linear graph for the 50% horizon, which looks like this:
I claim that although this is a faster trend than before for the 50% horizon, there are at least two reasons to take these results with a grain of salt:
Why 80% horizon and not 50%? Won't 50% still accelerate the economy and research?
Well, I don't know. I wish I had a better answer here that "I've spent 30 minutes talking to someone who seems to have thought way more about timelines than me and it seems that the thing they really care about is reliably automating coding".
My current model for the AI 2027 -> AI 2030 update goes something like "research taste is hard to bootstrap" and "actually it will take 4 years to get to super long (think years) 80% horizons".
Why Super Long 80% Horizons Though? Isn't 50% Enough?
Again, I wish I had a better answer here. Maybe read that update. And all the supplementary materials.
My understanding is that the main crux in the model is something called "Coding time horizon required to achieve Automated Coder", which you can play with at aifuturesmodel.com.
Right now it says "3.3 work years". That's because for some people, to really get an Automated Coder you need an AI working completely autonomously for like 125 years (~human max lifespan). For other people it's like months, or like 1 year.
For instance, if I change it to one month, I get automated coder by August 2028.
Why does Automated Coder Matter So Much? What about the economy? Vibe researching / Coding?
Those are all valid questions. My guess is that AI 2027 people would say like "not fully automating coding would give you some uplift but not the crazy uplift that completely automating coding would give you".
Something something unless you fully automate coding then you'll still be bottlenecked by human research taste and compute question mark? @elifland @Daniel Kokotajlo