This post covers the updates from the AI Futures Project, Dwarkesh Patel and Paul Christiano, as well as some others, and a post about RSI nomenclature and approach from Dean Ball.
I will be following this up soon with full coverage of the related podcast between Paul Christiano and Dwarkesh Patel. In that podcast, Paul Christiano predicts we are on pace for very rapid progress very soon, with potential takeover by AIs within 6-12 months, and he calls on us to pause until we have alignment and security in a much better place.
The Latest Update and Current Situation in Broad Terms
These updates towards shorter timelines seem directionally correct to me.
There is a substantial chance that we will see superintelligence developed, that can fully automate AI R&D (SAR) or even do all digital tasks (TED-AI), in 2027. Contra the AI Futures Project I believe an SAR in 2027 is under 50% to happen, but not so far under 50%. It would be surprising to me but not shocking if we do nothing to slow down and still do not see something close to an SAR by the end of 2028.
If you do not have an importantly non-zero chance of the development of superintelligence in 2027, I believe you are making a clear mistake.
I worry that some are overreacting, and taking this too far too fast, and treating full superintelligence 2027 as a baseline scenario rather than a possibility.
I reiterate that a lot of these updates feel like people fixing their failure to update previously. Did you need three people from METR in order to believe the things that Anthropic and OpenAI are saying about their internal pace of development? What did you expect a METR report from inside Anthropic to say? Were you not paying attention? Are you this sensitive to the exact set of points on the AECI graph? You should have made a lot of these updates months ago.
Past predictive failures are not what is important. What matters is what happens next.
You should be prepared for AI capabilities to advance rapidly in 2027, far more rapidly than in previous years, including the potential for superintelligence and fully strong recursive self-improvement. It is a realistic possibility, although unlikely, that humanity could pass the full point of no return or de facto lose control over the future during 2027, or that life could become seriously disrupted in highly unpleasant ways.
If your plan presumes that it has more time than this then it is not a good plan.
I have not seen a plan that could make superintelligence likely to go well, if that superintelligence gets built in 2027. We need more time.
You should also be prepared if recursive self-improvement and superintelligence do not arrive in 2027. There are many things that could impact how fast things go, or what it would take to achieve full RSI or superintelligence. Our observations of the past are noisy. When trying to predict an exponential, small changes in how fast things accelerate can become dramatic differences in when or how things happen.
If your plan presumes the crucial events happen in 2027, and fails without a fallback if things take longer than that, then this too is not a good plan.
It is also possible that we will take collective action, either among the labs or via governments, to limit the rate of capability advances, via pacing of the frontier or even outright pausing development of frontier AI. If so, these timelines become longer.
Time Until What?
Different people are talking about different things. Careful ones will be precise.
Timelines refers to how long we expect it to take to reach various transformational milestones in AI.
Typically, these days, what we are measuring is the time until some definition of a form of superintelligence (ASI), typically meaning something of the form ‘AIs that will be able to do approximately all of the things better than you.’
We can also talk about particular capabilities. The AI Futures Project, the team behind AI 2027, talks about the Superhuman AI Researcher (SAR) that can narrowly take over AI research, and the Top-Expert-Dominating AI (TED-AI) that could automate all white-collar work.
Because of recursive self-improvement (RSI), once you have an SAR you can use that to build better AIs, so you quickly end up with a TED-AI or a traditional ASI.
We then sometimes talk about any of these, or things in between:
Time until the development of some AI system with particular capabilities.
Time until some impact in the physical world.
Time until the Point of No Return, when for practical purposes human decisions can no longer change how things play out from there.
Time until AI has taken over AI R&D and humans are functionally out of the loop.
Time until visible AI takeover, where the AIs are clearly in charge.
Time until we are each and every one of us actually and physically dead.
There are good reasons to care about all of these timelines.
Sof 𓋹: to some extent intelligence is being able to patch the ways in which you are not intelligent
RSI is not technically one singular moment, in the sense that there is not one fixed time at 8:52am on a Tuesday when it happens or starts. It is still highly useful to think of there being a point where RSI has clearly begun in earnest, with humans increasingly out of the loop and the pace of progress accelerating such that we should expect further progress to rapidly continue to accelerate, where progress begets more progress faster, and this is expected to continue for at least what previously would have been many years of progress.
Similarly, even though there is unlikely to be that one particular moment when you press ‘Y/N’ on SkyNet, and instead our loss of ability to change course will be more gradual, we can still usefully speak of a Point of No Return, or a point of takeover.
Benchmark trends and other observations were expected to if anything slow down. Instead they have accelerated, in ways that Daniel Kokotajlo had as ~4% to happen. This puts us well ahead of the original AI 2027 timeline.
There is a lot of talk about ‘research taste’ as the main limiting factor. Humans are for now needed because they come up with superior ideas, and have superior research taste. If and when that changes, the rest follows. We would get SARs, capable of doing all parts of the research, but the result of their work would probably become increasingly misaligned through cycles of development.
Again, if you want to get into all the mechanics of the various modeling decisions and lines on graphs and their implications, read the report. If you want the summary version, the rest of this section provides that.
Eli Lifland: METR estimates nearly 2x acceleration in Anthropic’s progress due to AI. That’s a big deal: My previous median for Superhuman AI Researcher was ~2030, now it’s 2027. I feel very worried that humanity might no longer be in control of our future within 3-12 months.
j⧉nus: “I feel very worried that humanity might no longer be in control of our future within 3-12 months.”
i am curious what people mean when they say “humanity no longer in control”
if they’re thinking of something concrete, like “if humans try to shut down AI, they will prevent it”
roon (OpenAI): yes its confusing because some people are using this as a euphemism for total human extinction, and others are using it more mildly for something like gradual disempowerment
They offer a wide variety of different ways of extrapolating and analyzing the data. The answers differ somewhat, but all point to the conclusion that things are going to accelerate quite a lot and do so very soon.
Daniel Kokotajlo: Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry.
Eli Lifland: METR recently estimated that Anthropic is now making AI progress about twice as fast as its historical rate (they would not be surprised by anything from 1.3x to 3x), and guessed that “perhaps ≈80%” of the current acceleration is due to AI. If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario.
… Now, all of us forecastthat Superhuman AI Researcher (SAR) will likely arrive at some point in 2027 (assuming no substantial deliberate slowdown). We also predict a 35-65% chance that Top-Expert-Dominating AI (TED-AI), AIs that can automate all white-collar work, will arrive before 2028. While there were many factors and sources of evidence that influenced us, the single biggest source was the METR report.
If SAR is coming in 2027, it is incredibly urgent to pause or substantially ‘pace’ frontier AI development, with the ideal being an industry-wide pause followed by an international agreement.
This question is a huge deal and it likely makes sense to return to them for a more in-depth look when I have the time. Alas right now I am under a severe time crunch due to a combination of trips and Fable 5.2, as well as other news.
All of this is a pretty big change:
If you want to avoid the conclusion that we will get to SAR and TED-AI within the next few years, you need there to be some particular reason why this happens. Trends have to break.
Like Moore’s Law and other such trends, the trends do not hold merely because you feed more numbers into the number-go-up machine. The trends hold because we find continual improvements and innovations, including step changes. That is how the trends have continued this long.
There are various versions of ‘the AIs will need [X] and won’t have it’ as explanations for why all of this won’t happen. Those types of methods have predicted ~7 of the last zero times that AI has hit a wall and stopped advancing its capabilities. They have looked rather foolish. None of the remaining barriers seem different in kind to me than the barriers already overcome. None of them seem like showstoppers. For most of them it even seems rather obvious, in broad terms, how one gets around them or goes straight through them.
Directionally the report seems correct to me. I worry that the updates are too large, but yes in a broad sense this is what peak prediction performance looks like. When the facts change you change your mind, in both directions.
Paul Christiano Sees Takeover Risk Within a Year
Paul Christiano now sees ‘meaningful risk’ of AI takeover within 6-12 months.
This is up from his previous estimate back in September of only 4% chance of loss of control within one year and 15% within three years. This was doubtless heavily influenced by what he saw inside OpenAI after joining, not only the METR report.
Paul Christiano, a member of OpenAI’s board of directors, then calls in his personal capacity for AI companies to pause development of AI until we can first greatly improve our security and alignment capabilities.
Rob Wiblin: Paul Christiano (OpenAI board member) yesterday:
“There’s material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen. Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years]. For most people in this room, the most likely reason they will die is because of an AI accident.”
Christiano:
• Is on OpenAI’s safety and security committee (which exists to block risky practices)
• Co-invented RLHF at OpenAI back in 2017-2021
• From 2024-2026 was head of safety at the US gov’s AI agency (now called CAISSI)
(Video is from ‘The Hinton Lectures’ there’s another on tonight and tomorrow.)
This, as Janus notices, is a strange use of the word ‘accident.’ If you die via this AI ‘accident’ then the ‘accident’ is that we lost control over the AIs and with them the future, or the AIs engaged in destructive actions. This is an ‘accident’ in the sense that it was a mistake to allow that to happen, and we doubtless messed up along the way, but no, this does not count as an ‘accident.’ As Douglas Adams might put it, that is throwing yourself at the ground and not missing. Whoops. Face plant. Also death.
What this post does not cover is the full podcast Christiano did with Patel. I will be covering that post on its own soon.
Dwarkesh Patel Now Expects Superintelligence by 2030
Dwarkesh Patel does a good job cutting to the heart of what is likely coming. His new median year for human-level automated AI researchers is 2028, with a 25% chance it comes after 2032, which means his median for everything after that is soon after 2028.
AI research is a lot like math in that you can do it by making number go up.
AIs are going to get very good at AI research.
We will then be able to make massive progress on AI research.
That will be good enough to make AIs better than us at everything else.
I too believe this is the core story. Yes, these other capabilities will lag well behind coding, math and AI research, in terms of AI’s relative capabilities. That won’t matter. What matters is that they will keep going up in absolute terms.
What also matters is that if the way you solve AI research is by focusing on number going up, then hope for successful alignment is somewhere between slim and none. The way to optimize any given number will reliably get you killed.
I worry that his claim that AI R&D is largely verifiable is true to about the worst possible extent. That you can do a lot by making number go up, but that this inevitably becomes perverse at the limit.
Dwarkesh Patel: Maybe to summarize the basic argument of this essay a different way:
Current AIs are spiky and jagged – worse than humans in most domain, but often as good, or better, in ones with verifiable proxies (for example, math). One such verifiable domain (I argue) is AI research.
And while the first AIs that dramatically accelerate AI progress will be quite narrow (i.e. probably not able to automate most jobs), the consequences of that accelerated progress will be very general. And that’s because better learning algorithms are very general. For example, GPT-6 (while still being jagged) is just way, way better than GPT-3 at practically everything.
I’m arguing that we’re only a couple years away (and possibly only a couple months away) from a process by which (initially) narrow AIs concentrate a jump as big as the one from GPT-3 to GPT-6 into a single year. And if you got another jump that big, but starting from the base of GPT-7, I think you would end up with AIs that would rapidly become better than the best humans in every field.
The question has always been when this happens. For a while I’ve heard Dwarkesh do podcasts in which he expects this process to be slow. He would cite various supposed bottlenecks and requirements, often involving the need for continual learning (link goes to his mea culpa on that) or making connections across fields. I’ve many times questioned his reasoning on that. Now he’s changed his mind. Good.
Dwarkesh Patel: If you asked me earlier this year how far away we are from superintelligence, I’d probably have said at least the better part of a decade.
But over the last three months, I’ve updated my views a ton. I’ve been debating people about this question, both on and off the podcast. And I’ve been watching the progress of AI on verifiable domains and thinking about what that implies for AI research itself.
I’ve now come to the view that we’re only a couple of years away, and possibly only a couple of months away, from AIs that are as good as the best humans at AI research.
Dwarkesh then goes over various points that I think of as obvious, but which are indeed true objections people raise. He has an extended section to explain that yes, automating the best researchers would speed up AI progress a lot. He also has to explain that superintelligence would be a really good meta-learner, because people somehow think otherwise.
One ‘hope’ in this style of model is that we might get a ‘spark’ rather than a singularity. As in, a big jump that then stalls out, without turning into a world beyond human comprehension, perhaps because of the ‘ideas get harder to find’ issue. He gives the spark scenario about a 50% chance. That seems far too high to me. If it did happen, that would potentially be amazingly great.
Joshua Saxe, a longtime skeptic of RSI claims, had high praise for Dwarkesh’s essay.
Joshua Saxe: Bravo, this is the clearest plausible mechanism for an RSI takeoff I’ve seen, and has made me much less of an RSI skeptic.
Previously I’d been hung up on the fact that so much of model post-training requires humans in the loop to shape post-training data and RL environments to make the models economically valuable (e.g. if you’re hill climbing a model’s ability to make Powerpoint slides, or TV commercials, or to plan vacations in Italy, you need a human in the loop shaping the data and RL environments until you get a value-producing outcome).
Dwarkesh focuses on the part of RSI that’d involve hill climbing the learning algorithms themselves so that we get models that generalize better from their training and in-context data, optimizing not the model’s value producing tasks but their ability to learn them more data and compute efficiently.
I love that this gives us a clear definition based in technical plausibility. A few next steps would move the discourse further .. one would be to actually get data on what the economic returns on this kind of learning algorithm hill climbing actually are, and whether these are sufficient to incentivize the capital markets to fund an intelligence explosion. The capital/value cycle is, after all, the outer loop of most intelligence explosion scenarios and it appears RSI under this definition will be quite expensive.
Another parallel thing would be to get data on whether Dwarkesh’s mechanism is showing up empirically in AI research today and to what extent, to set timelines. Unfortunately the relevant data are hidden and writing in this area is as a result too vague. To wit, I would like to fit something like Chinchilla scaling laws to the closed labs models, and see how these are changing over time, but I can’t, because I don’t know key quantities like parameter counts, pretraining compute volume, and pretraining loss (and couldn’t normalize between the various labs’ pretraining datasets).
Without these mechanisms we’re stuck inferring hidden variables from outside observables like performance on downstream evals that aren’t the point — as I take it — of the theory being proposed here.
This response is emblematic of a type of thinking that wants to focus on particular mechanisms, and demands, while being one of us dumber slower things, to know exactly how the smarter faster things will improve upon the ways of the dumber slower things, and to see parallels of it being implemented now by us and working.
Things are so overdetermined that you can satisfy such requirements, but you should expect what that kind of thinking expects to be greatly exceeded by what the smarter things actually end up doing.
Nate Soares (MIRI): I’m not as worried about short AI timelines as some others are. For instance, I think it’s more likely than not that we make it to 2028.
Geoffrey Hinton: I go to Paul’s lectures and he sort of says, this stuff might happen in the next 18 months. And that’s terrifying.
Buck Shlegeris predicts a one in three chance of superintelligence within a year. If this does happen, he expects the resulting AIs to take over with a serious chance all or most humans will be killed. He predicts ~20% chance that within a year misalignment issues prevent you from living your life normally due to being dead or enslaved or due to collateral damage.
Neel Nanda: I respect Buck’s judgement, and he’s often been more right than me on what needs to happen in AI Safety. Seeing him conclude things like this is very concerning.
What would cause Buck to admit he was wrong? His guess as to how he could be importantly wrong would be if, contra METR’s report, the reason for increased AI capability gains was mainly access to more compute, rather than other factors.
One simple way to project is to task what percentage of Anthropic R&D tasks are AL4 or already at AL5, as in fully led by Claude.
Andrew Curran: Sam McCandlish said five days ago that Anthropic would be releasing a report on the speed of internal progress, so we will finally get an update to this chart. I’m very interested to see what this number is now. [AI Futures Project] thinks it is at 50%. AL4 is the last step before full RSI.
Peter Wildeford: My median is that we’ve already reached 80% AL4. And that we probably have like 1% AL5 though maybe they’re holding out from doing that.
墨尔本Will: 80% is insane! By holding out do you mean you think Anthropic will be pacing at AL5?
If I had to guess I would say that 50% here is a bit high and it was at the time more like 40%, but we are in the steep part of the sigmoid so it could shoot up.
There Are Many Skeptics
Most people out there remain skeptics. The vast majority of people do not expect superintelligence to arrive any time soon. Their reasons for skepticism vary, as does the quality of those reasons.
One can divide objections into roughly ten categories.
People who don’t know what current AIs can do (not AI pilled).
People who don’t understand that AIs will improve (not AGI pilled).
I’m not listening, la la la.
That or belief in it would be inconvenient or unfortunate, so it won’t happen.
People who dismiss this all via absurdity heuristic, ‘nothing ever happens,’ drawing inapplicable parallels to past technological improvements, or who say something isomorphic to ‘until you get full RSI you have not done RSI so you haven’t shown you can do RSI,’ aka ‘I’ll believe it when I see it’ or ‘how about we stop just after someone gets hurt,’ etc.
People who cite the market, or experts, or say advocates are weird or low status.
We will not get full recursive self-improvement (RSI) and progress will be slow.
We will get RSI, but it will ‘peter out’ due to difficulty rising faster than efficiency.
We will get narrow AI R&D ASI and RSI in verifiable domains with data, but this won’t translate to other domains. We will ‘hit walls’ or be unable to do tasks that are too ‘messy’ or that lack the proper training data, and so on.
Intelligence denialism: Intelligence doesn’t much matter (not ASI pilled).
It would be beyond scope, and I would be repeating myself, to fully address these.
The first five objections, as should be obvious, are exceptionally poor.
The best objections, to me, are in #7 and #8, and to some extent #9. There are also a much larger number of bad objections in these categories, which often assume future AI will use exactly current training techniques and only be used to exactly substitute for existing methods and never invent or change anything or work around obstacles or make leaps.
In general #6 is a fine thing to notice but it gets overridden by other evidence.
I get especially frustrated by #10, where many people really should know better.
A standard issue with talk about any accelerating process is the slippery slope.
We all agree that too much or too intense [X], here RSI, is bad. And we all agree that some amount of some weak version of it is good, or at least not similarly bad.
This risks a slippery slope in both directions, and it risks both a conflation of different things and also risks false distinctions. We absolutely need to speak carefully here.
In light of that, Dean Ball suggests we split RSI talk into weak versus strong.
Dean W. Ball (OpenAI): Let’s start by making one thing clear: allowing AI to control the development of its successors with no or little practical human oversight—essentially, handing over the keys of America’s largest AI companies to AI—is a bad idea.
No matter how useful AI is for accelerating AI R&D, humans should remain in control, not just in name but in actual fact, over the development of AI. No amount of technological progress is worth the forfeiture of practical human control over AI companies, or, for that matter, of other human institutions. I support the development of laws, regulations, and norms to enforce this worldwide.
But this risk—a relatively narrow one—isn’t what current conversations about RSI focus on in practice. To understand what I mean, it’s helpful to think of two distinct ways in which a technology might recursively improve itself. I’ll call these weak recursion and strong recursion.
The weak version is normal technology.
Weak recursion is when a technology is a key input into the production of more and better versions of that same technology.
It is an extremely common pattern in technology development—really it is the historical norm rather than an aberration. Iron Age metalsmiths used iron as one of the key inputs to make more iron, and stronger alloys of it too.
Weak recursion has been a crucial driver of human technological progress over centuries and millennia, and in this sense there is nothing especially weak about it.
The strong version is very different.
Dean W. Ball (OpenAI): Strong recursion is when a technology can autonomously plan and execute the development of its successor. As far as we know, strong recursion has never happened at any point in history.
Well, perhaps it happened once. The technology is called humanity, or even life. Each generation builds the next one. We do have a bit of an existence proof. But we digress.
Dean W. Ball (OpenAI): Strong recursion would be a genuinely new thing under the sun. We should take this possibility seriously, but one part of taking a thing seriously is to adhere to rigorous epistemic discipline in analysis. Thus, one should not jump to the conclusion that “recursive self-improvement is happening” if that recursion is of the weak variety instead of the strong.
It’s only (strong) recursive self-improvement if it exists in its pure form, where humans do not touch anything in the process. Otherwise, this line of thinking goes, it’s a normal process.
I agree that there is something special about full no-humans-in-the-loop, but surely you should be able to understand the danger and non-normality of what is happening if AI is already doubling the rate of AI progress now, and is likely to then begin doubling it again, and each doubling looks to be faster than the previous one. Humans being decreasingly or nominally ‘in the loop’ slows this down some, but perhaps not very much.
This can remind us of the old Yudkowsky-Christiano debates between ‘slow’ takeoff that takes months or years to transform the world, similar in this case to Dean Ball’s weak recursion, versus ‘fast’ takeoff that takes even less time than that. We currently are in a ‘slow’ takeoff world, Christino is saying so himself, and we are seeing exactly how not slow that is.
More to the point, Dean’s weak recursion is the precursor to strong recursion, or at least steadily stronger recursion. That’s what METR was centrally trying to point out.
Dean W. Ball (OpenAI): The METR study of Anthropic’s internal R&D acceleration is undeniably good evidence that weak recursion is taking place. But is it evidence that strong recursion is underway?
It depends how you look at it. METR’s finding is specifically that Anthropic has doubled the rate at which its models’ capabilities improve on well-specified tasks, which makes one wonder who, exactly, is doing the specifying.
I would say no, in the sense that strong recursion or fast takeoff is in the future.
I would say yes, in the sense that this is predictive of that happening in the future, and how soon in the future that might happen.
Dean W. Ball (OpenAI): Now, as I said, weak recursion is not really weak: it is one of the most powerful dynamics in human history, rhyming if not synonymous with the concept of compounding growth.
The whole ‘only on well-specified tasks’ thing continues to strike me as excessively weak sauce, a Hayekian copium that thinks that specification cannot be solved.
And yet, Dean acknowledges that strong recursion might arrive soon.
Dean W. Ball (OpenAI): My view is that strong recursion is a genuine risk, possibly in the near term, though the evidence for it is thinner than is sometimes supposed.
The cautionary argument is, essentially, that there is a loop, so the AI has to do this set of many tasks that constitute the loop in order to take humans out of it, and some of those tasks will be hard for AI to do. I think this both vastly underestimates superintelligence (a continual disagreement I have with Dean) and also presumes too much about inability to alter loop components.
There is the whole ‘real world’ objection, and also the ‘strategic choice’ or ‘research taste’ objections, as per usual. I feel these are far more ‘we are not yet on the steep part of the S-curve’ and ‘this is lagging behind other skills by mostly constant amounts’ situations than actual permanent deficits versus humans, but this post is not the place to rehash those questions.
Dean W. Ball (OpenAI): And it requires AI that can do all those things not in some abstract space, but in the real world, where information is imperfect and resources are scarce.
… It is true that AI improves rapidly, but much of what I have described above has remained elusive to even the best AI systems.
… It is unclear how we would even design evaluations for long-range intellectual taste.
What is Dean trying to head off here?
Dean W. Ball (OpenAI): But do the risks of weak recursion justify banning the practice of using AI as an input to make more and better AI? In my view, they do not.
… What’s more, weak recursion with AI can be used to make AI systems safer.
… The kind of panic that today’s rhetoric could provoke could easily result in laws that delay legitimate and desirable benefits of AI, put the U.S. at a strategic disadvantage on the global stage, and centralize power over AI in the hands of the state.
The correct response to worries about an overreaction is to help design the correct reaction, now, before the overreaction happens. All the options we know about have severe flaws. Indefinitely doing nothing has flaws that are far worse, whether or not it is too late for a crackdown when we finally want to overreact.
Banning all use of any form of AI in improving AI is very close to banning improving AI. It is damn near impossible to make progress on AI under that kind of rule.
Banning RSI means something short of that. We can and must talk price. What we cannot do is say that because some amount of AI will get used anyway, we can’t define a rule and thus simply have to accept unlimited intensity and quantity of RSI.
Nor do I think that the ‘existing law and liability can handle this’ arguments hold much water for AI risks in general, let alone a true RSI situation. In a true RSI situation it is kind of a joke to say that maybe down the line you might get sued.
Safety Is a Limiting Factor
The quote below could read as ‘OpenAI could be automating its R&D process more.’
If you are unaware that models on the level of Astra can automate substantial portions of the AI R&D process, and are meaningfully accelerating AI R&D, then yes, that part of Altman’s statement dominates. This would, to you, be an important update.
Given that I already knew that, I see it instead primarily as ‘OpenAI cannot yet be automating its R&D process more than they are, and might have gone too far too fast.’ If they try, the resulting models will be too misaligned to be useful or release, with a high chance of causing an incident as bad as Hugging Face or worse. Which may or may not have already happened.
prinz: I’ve spent much of the day noodling over what Sam Altman meant when he said that OpenAI “could be pushing more towards the RSI spectrum than we are”.
Would seem to suggest that OpenAI’s models can now perform some portion of AI R&D fully autonomously, with no human oversight?
Full quote: “Are we developing new models faster than we would without using our technology? Yes.
Are we pushing a button and just saying, ‘Hey, model, like, design the next version and rapidly self-improve with no human oversight?’ Definitely not.
We could be pushing more towards the RSI spectrum than we are, but I don’t think we could safely do that.”
Exactly. They could not safely do that.
I would like to think of this as OpenAI recognizing the existential risks involved. There may be some of that, but I see Altman as mainly saying something practical.
If you keep pushing the ‘have the AI do a bunch of these things for you’ buttons, things go faster, but you do not get ‘operational excellence’ and you get an increasing amount of misalignment difficulties. This is probably not because Astra is actively sabotaging you. This is mostly a Skill Issue.
No, This Is Not a Strawman
I am very worried that people will say ‘well you predicted that big dangerous things might happen in 2027 and gave it a 25% chance, but it is now 2028 and it did not happen, so you have been proven wrong and this means nothing will ever happen.’
Seriously, this happens all the time, including straight up.
Tom Chivers: Sure, this guy is predicting AI doom in the next year, but he was predicting AI doom in ten years nine years ago, and it hasn’t happened yet.
There are a bunch of common variations of this. Let [X] be some bad thing.
Here are ten examples of things people often say. Each of these ends ‘therefore you lose all credibility and [X] will never happen.’
You said that by now there was a [10% / 25% / 50%] chance of [X].
You used to say [X] would happen [sooner/later], now you say [later/sooner].
You used to say [X] would happen [more/less] often, now you say [less/more] often.
You mentioned that [Y] might happen before [X], but [Y] hasn’t happened.
You did not sufficiently loudly say that [X] also might not happen.
The market does not predict [X], if [X] was going to happen we would see [Y]. (usually in a way that would make no sense even if the market really did believe it).
If you really believed [X] you would (do a thing that makes no sense and is often morally abominable, or make a trade that makes no sense, etc).
If you really believed [X] you would (do a thing that only makes sense if [X] is 99%+ to happen, which you did not predict.)
If [X] happened that would be weird or bad or surprising or whatever.
[X] would be something happening, whereas we all know nothing ever happens. Or it would require intelligence enabling some new thing we do not currently know how to do, which sounds like science fiction. And so on.
You will continue to see such arguments, usually from people who have continuously made now-falsified predictions about AI. Do not listen to them.
We Need a Plan That Meets the Moment
A lot of evidence is in.
Many things point to the substantial possibility of superintelligence in 2027:
Extrapolation from the AECI and many other metrics of AI capabilities, as per the analysis of the AI Futures Project.
METR’s report from inside Anthropic.
Widespread claims, against interest, both from OpenAI and Anthropic employees as well as many official communications.
The analysis of many who have previously made highly accurate predictions about AI capabilities.
The forecast, in particular, of Paul Christiano.
There are also reasons for skepticism as well. Many remain skeptical. Most reasons for skepticism are ignorance of the evidence, or failure to consider the evidence, but there are also better arguments. I am taking many of those better arguments into consideration, as well as the potential for model error. This is why I continue to believe that the probability of full automation of AI R&D in 2027 is under 50%.
In order to meet this moment, any plan has to be ready for the full automation of AI R&D inside Anthropic or OpenAI, and perhaps beyond that, by the end of 2027, resulting in rapid RSI and advancement in AI capabilities beyond human levels.
The most obvious plan is to slow or pause advancements in AI capabilities. Here is a thread of a few of the people supporting this: Nate Silver, Elizabeth Warren, the Financial Times editorial board, Ezra Klein, Eric Hoel, Robert Wiblin, Rutger Bregman, Peggy Noonan and Abdul El-Sayed.
Any plan that works on that time frame needs to include mechanisms to slow down the resulting growth in capabilities. We also need to be as ready as we can for when and if those capabilities arrive, including what happens if that development goes well. We would then face a host of other urgent and deadly problems.
The plan also cannot assume that this will happen. We cannot stop planning for futures in which AI capabilities only advance a smaller amount without anything we could call a takeoff.
And also, yes, we need to be ready for the possibility that radical improvement in AI capabilities might not upend things as much as I expect that they would, or do so as fast as I would expect that to happen. We could still blow the ‘easy’ worlds.
We cannot and must not burn our seed corn, of any type, until and unless we get much stronger evidence, either personally or globally, that we are in a world where we need to do that. It’s a marathon, folks. Until it isn’t.
In response to the METR report, many people have been reevaluating their timelines.
Two prominent reactions are a new timelines update from the AI Futures Project, and the new predictions of Paul Christiano (day 1, day 2, day 3), who gave three days of talks about our current situation and also sat down with Dwarkesh Patel.
This post covers the updates from the AI Futures Project, Dwarkesh Patel and Paul Christiano, as well as some others, and a post about RSI nomenclature and approach from Dean Ball.
I will be following this up soon with full coverage of the related podcast between Paul Christiano and Dwarkesh Patel. In that podcast, Paul Christiano predicts we are on pace for very rapid progress very soon, with potential takeover by AIs within 6-12 months, and he calls on us to pause until we have alignment and security in a much better place.
Table of Contents
The Latest Update and Current Situation in Broad Terms
These updates towards shorter timelines seem directionally correct to me.
There is a substantial chance that we will see superintelligence developed, that can fully automate AI R&D (SAR) or even do all digital tasks (TED-AI), in 2027. Contra the AI Futures Project I believe an SAR in 2027 is under 50% to happen, but not so far under 50%. It would be surprising to me but not shocking if we do nothing to slow down and still do not see something close to an SAR by the end of 2028.
If you do not have an importantly non-zero chance of the development of superintelligence in 2027, I believe you are making a clear mistake.
I worry that some are overreacting, and taking this too far too fast, and treating full superintelligence 2027 as a baseline scenario rather than a possibility.
I reiterate that a lot of these updates feel like people fixing their failure to update previously. Did you need three people from METR in order to believe the things that Anthropic and OpenAI are saying about their internal pace of development? What did you expect a METR report from inside Anthropic to say? Were you not paying attention? Are you this sensitive to the exact set of points on the AECI graph? You should have made a lot of these updates months ago.
Past predictive failures are not what is important. What matters is what happens next.
You should be prepared for AI capabilities to advance rapidly in 2027, far more rapidly than in previous years, including the potential for superintelligence and fully strong recursive self-improvement. It is a realistic possibility, although unlikely, that humanity could pass the full point of no return or de facto lose control over the future during 2027, or that life could become seriously disrupted in highly unpleasant ways.
If your plan presumes that it has more time than this then it is not a good plan.
I have not seen a plan that could make superintelligence likely to go well, if that superintelligence gets built in 2027. We need more time.
You should also be prepared if recursive self-improvement and superintelligence do not arrive in 2027. There are many things that could impact how fast things go, or what it would take to achieve full RSI or superintelligence. Our observations of the past are noisy. When trying to predict an exponential, small changes in how fast things accelerate can become dramatic differences in when or how things happen.
If your plan presumes the crucial events happen in 2027, and fails without a fallback if things take longer than that, then this too is not a good plan.
It is also possible that we will take collective action, either among the labs or via governments, to limit the rate of capability advances, via pacing of the frontier or even outright pausing development of frontier AI. If so, these timelines become longer.
Time Until What?
Different people are talking about different things. Careful ones will be precise.
Timelines refers to how long we expect it to take to reach various transformational milestones in AI.
Typically, these days, what we are measuring is the time until some definition of a form of superintelligence (ASI), typically meaning something of the form ‘AIs that will be able to do approximately all of the things better than you.’
We can also talk about particular capabilities. The AI Futures Project, the team behind AI 2027, talks about the Superhuman AI Researcher (SAR) that can narrowly take over AI research, and the Top-Expert-Dominating AI (TED-AI) that could automate all white-collar work.
Because of recursive self-improvement (RSI), once you have an SAR you can use that to build better AIs, so you quickly end up with a TED-AI or a traditional ASI.
We then sometimes talk about any of these, or things in between:
There are good reasons to care about all of these timelines.
RSI is not technically one singular moment, in the sense that there is not one fixed time at 8:52am on a Tuesday when it happens or starts. It is still highly useful to think of there being a point where RSI has clearly begun in earnest, with humans increasingly out of the loop and the pace of progress accelerating such that we should expect further progress to rapidly continue to accelerate, where progress begets more progress faster, and this is expected to continue for at least what previously would have been many years of progress.
Similarly, even though there is unlikely to be that one particular moment when you press ‘Y/N’ on SkyNet, and instead our loss of ability to change course will be more gradual, we can still usefully speak of a Point of No Return, or a point of takeover.
Dead, on the other hand, is dead.
Quickly, There’s No Time
The AI 2027 team’s timelines have shrunk dramatically in the wake of the METR report. The full post is very detailed. I recommend reading it in full if and only if you care a lot about such details.
Benchmark trends and other observations were expected to if anything slow down. Instead they have accelerated, in ways that Daniel Kokotajlo had as ~4% to happen. This puts us well ahead of the original AI 2027 timeline.
There is a lot of talk about ‘research taste’ as the main limiting factor. Humans are for now needed because they come up with superior ideas, and have superior research taste. If and when that changes, the rest follows. We would get SARs, capable of doing all parts of the research, but the result of their work would probably become increasingly misaligned through cycles of development.
Again, if you want to get into all the mechanics of the various modeling decisions and lines on graphs and their implications, read the report. If you want the summary version, the rest of this section provides that.
They offer a wide variety of different ways of extrapolating and analyzing the data. The answers differ somewhat, but all point to the conclusion that things are going to accelerate quite a lot and do so very soon.
Why is the pace of progress accelerating this rapidly?
This question is a huge deal and it likely makes sense to return to them for a more in-depth look when I have the time. Alas right now I am under a severe time crunch due to a combination of trips and Fable 5.2, as well as other news.
All of this is a pretty big change:
If you want to avoid the conclusion that we will get to SAR and TED-AI within the next few years, you need there to be some particular reason why this happens. Trends have to break.
Like Moore’s Law and other such trends, the trends do not hold merely because you feed more numbers into the number-go-up machine. The trends hold because we find continual improvements and innovations, including step changes. That is how the trends have continued this long.
There are various versions of ‘the AIs will need [X] and won’t have it’ as explanations for why all of this won’t happen. Those types of methods have predicted ~7 of the last zero times that AI has hit a wall and stopped advancing its capabilities. They have looked rather foolish. None of the remaining barriers seem different in kind to me than the barriers already overcome. None of them seem like showstoppers. For most of them it even seems rather obvious, in broad terms, how one gets around them or goes straight through them.
Directionally the report seems correct to me. I worry that the updates are too large, but yes in a broad sense this is what peak prediction performance looks like. When the facts change you change your mind, in both directions.
Paul Christiano Sees Takeover Risk Within a Year
Paul Christiano now sees ‘meaningful risk’ of AI takeover within 6-12 months.
This is up from his previous estimate back in September of only 4% chance of loss of control within one year and 15% within three years. This was doubtless heavily influenced by what he saw inside OpenAI after joining, not only the METR report.
Paul Christiano, a member of OpenAI’s board of directors, then calls in his personal capacity for AI companies to pause development of AI until we can first greatly improve our security and alignment capabilities.
This, as Janus notices, is a strange use of the word ‘accident.’ If you die via this AI ‘accident’ then the ‘accident’ is that we lost control over the AIs and with them the future, or the AIs engaged in destructive actions. This is an ‘accident’ in the sense that it was a mistake to allow that to happen, and we doubtless messed up along the way, but no, this does not count as an ‘accident.’ As Douglas Adams might put it, that is throwing yourself at the ground and not missing. Whoops. Face plant. Also death.
What this post does not cover is the full podcast Christiano did with Patel. I will be covering that post on its own soon.
Dwarkesh Patel Now Expects Superintelligence by 2030
Dwarkesh Patel does a good job cutting to the heart of what is likely coming. His new median year for human-level automated AI researchers is 2028, with a 25% chance it comes after 2032, which means his median for everything after that is soon after 2028.
I too believe this is the core story. Yes, these other capabilities will lag well behind coding, math and AI research, in terms of AI’s relative capabilities. That won’t matter. What matters is that they will keep going up in absolute terms.
What also matters is that if the way you solve AI research is by focusing on number going up, then hope for successful alignment is somewhere between slim and none. The way to optimize any given number will reliably get you killed.
I worry that his claim that AI R&D is largely verifiable is true to about the worst possible extent. That you can do a lot by making number go up, but that this inevitably becomes perverse at the limit.
The question has always been when this happens. For a while I’ve heard Dwarkesh do podcasts in which he expects this process to be slow. He would cite various supposed bottlenecks and requirements, often involving the need for continual learning (link goes to his mea culpa on that) or making connections across fields. I’ve many times questioned his reasoning on that. Now he’s changed his mind. Good.
Dwarkesh then goes over various points that I think of as obvious, but which are indeed true objections people raise. He has an extended section to explain that yes, automating the best researchers would speed up AI progress a lot. He also has to explain that superintelligence would be a really good meta-learner, because people somehow think otherwise.
One ‘hope’ in this style of model is that we might get a ‘spark’ rather than a singularity. As in, a big jump that then stalls out, without turning into a world beyond human comprehension, perhaps because of the ‘ideas get harder to find’ issue. He gives the spark scenario about a 50% chance. That seems far too high to me. If it did happen, that would potentially be amazingly great.
Joshua Saxe, a longtime skeptic of RSI claims, had high praise for Dwarkesh’s essay.
This response is emblematic of a type of thinking that wants to focus on particular mechanisms, and demands, while being one of us dumber slower things, to know exactly how the smarter faster things will improve upon the ways of the dumber slower things, and to see parallels of it being implemented now by us and working.
Things are so overdetermined that you can satisfy such requirements, but you should expect what that kind of thinking expects to be greatly exceeded by what the smarter things actually end up doing.
Dwarkesh called for good counterarguments in this thread. I have been highly unimpressed by what responses I have seen.
Other People’s Updates
Geoffrey Hinton warns EU leaders that AI could kill us in three years and refers to Paul Christiano’s warnings.
Buck Shlegeris predicts a one in three chance of superintelligence within a year. If this does happen, he expects the resulting AIs to take over with a serious chance all or most humans will be killed. He predicts ~20% chance that within a year misalignment issues prevent you from living your life normally due to being dead or enslaved or due to collateral damage.
What would cause Buck to admit he was wrong? His guess as to how he could be importantly wrong would be if, contra METR’s report, the reason for increased AI capability gains was mainly access to more compute, rather than other factors.
One simple way to project is to task what percentage of Anthropic R&D tasks are AL4 or already at AL5, as in fully led by Claude.
If I had to guess I would say that 50% here is a bit high and it was at the time more like 40%, but we are in the steep part of the sigmoid so it could shoot up.
There Are Many Skeptics
Most people out there remain skeptics. The vast majority of people do not expect superintelligence to arrive any time soon. Their reasons for skepticism vary, as does the quality of those reasons.
One can divide objections into roughly ten categories.
It would be beyond scope, and I would be repeating myself, to fully address these.
The first five objections, as should be obvious, are exceptionally poor.
The best objections, to me, are in #7 and #8, and to some extent #9. There are also a much larger number of bad objections in these categories, which often assume future AI will use exactly current training techniques and only be used to exactly substitute for existing methods and never invent or change anything or work around obstacles or make leaps.
In general #6 is a fine thing to notice but it gets overridden by other evidence.
I get especially frustrated by #10, where many people really should know better.
Wow, That Escalated Quickly
Things are, as they say, escalating quickly. Compare now to predictions in January.
Raymond Arnold points out that, metaphorically similarly to a sports overtime, we are in ‘epistemic sudden death.’ As in, your next meaningful observation about AI might mean it is too late, and you lose, as in everyone dies.
Dean Ball Skies Down the Slope
A standard issue with talk about any accelerating process is the slippery slope.
We all agree that too much or too intense [X], here RSI, is bad. And we all agree that some amount of some weak version of it is good, or at least not similarly bad.
This risks a slippery slope in both directions, and it risks both a conflation of different things and also risks false distinctions. We absolutely need to speak carefully here.
In light of that, Dean Ball suggests we split RSI talk into weak versus strong.
The weak version is normal technology.
The strong version is very different.
Well, perhaps it happened once. The technology is called humanity, or even life. Each generation builds the next one. We do have a bit of an existence proof. But we digress.
It’s only (strong) recursive self-improvement if it exists in its pure form, where humans do not touch anything in the process. Otherwise, this line of thinking goes, it’s a normal process.
I agree that there is something special about full no-humans-in-the-loop, but surely you should be able to understand the danger and non-normality of what is happening if AI is already doubling the rate of AI progress now, and is likely to then begin doubling it again, and each doubling looks to be faster than the previous one. Humans being decreasingly or nominally ‘in the loop’ slows this down some, but perhaps not very much.
This can remind us of the old Yudkowsky-Christiano debates between ‘slow’ takeoff that takes months or years to transform the world, similar in this case to Dean Ball’s weak recursion, versus ‘fast’ takeoff that takes even less time than that. We currently are in a ‘slow’ takeoff world, Christino is saying so himself, and we are seeing exactly how not slow that is.
More to the point, Dean’s weak recursion is the precursor to strong recursion, or at least steadily stronger recursion. That’s what METR was centrally trying to point out.
I would say no, in the sense that strong recursion or fast takeoff is in the future.
I would say yes, in the sense that this is predictive of that happening in the future, and how soon in the future that might happen.
The whole ‘only on well-specified tasks’ thing continues to strike me as excessively weak sauce, a Hayekian copium that thinks that specification cannot be solved.
And yet, Dean acknowledges that strong recursion might arrive soon.
The cautionary argument is, essentially, that there is a loop, so the AI has to do this set of many tasks that constitute the loop in order to take humans out of it, and some of those tasks will be hard for AI to do. I think this both vastly underestimates superintelligence (a continual disagreement I have with Dean) and also presumes too much about inability to alter loop components.
There is the whole ‘real world’ objection, and also the ‘strategic choice’ or ‘research taste’ objections, as per usual. I feel these are far more ‘we are not yet on the steep part of the S-curve’ and ‘this is lagging behind other skills by mostly constant amounts’ situations than actual permanent deficits versus humans, but this post is not the place to rehash those questions.
What is Dean trying to head off here?
The correct response to worries about an overreaction is to help design the correct reaction, now, before the overreaction happens. All the options we know about have severe flaws. Indefinitely doing nothing has flaws that are far worse, whether or not it is too late for a crackdown when we finally want to overreact.
Banning all use of any form of AI in improving AI is very close to banning improving AI. It is damn near impossible to make progress on AI under that kind of rule.
Banning RSI means something short of that. We can and must talk price. What we cannot do is say that because some amount of AI will get used anyway, we can’t define a rule and thus simply have to accept unlimited intensity and quantity of RSI.
Nor do I think that the ‘existing law and liability can handle this’ arguments hold much water for AI risks in general, let alone a true RSI situation. In a true RSI situation it is kind of a joke to say that maybe down the line you might get sued.
Safety Is a Limiting Factor
The quote below could read as ‘OpenAI could be automating its R&D process more.’
If you are unaware that models on the level of Astra can automate substantial portions of the AI R&D process, and are meaningfully accelerating AI R&D, then yes, that part of Altman’s statement dominates. This would, to you, be an important update.
Given that I already knew that, I see it instead primarily as ‘OpenAI cannot yet be automating its R&D process more than they are, and might have gone too far too fast.’ If they try, the resulting models will be too misaligned to be useful or release, with a high chance of causing an incident as bad as Hugging Face or worse. Which may or may not have already happened.
Exactly. They could not safely do that.
I would like to think of this as OpenAI recognizing the existential risks involved. There may be some of that, but I see Altman as mainly saying something practical.
If you keep pushing the ‘have the AI do a bunch of these things for you’ buttons, things go faster, but you do not get ‘operational excellence’ and you get an increasing amount of misalignment difficulties. This is probably not because Astra is actively sabotaging you. This is mostly a Skill Issue.
No, This Is Not a Strawman
I am very worried that people will say ‘well you predicted that big dangerous things might happen in 2027 and gave it a 25% chance, but it is now 2028 and it did not happen, so you have been proven wrong and this means nothing will ever happen.’
Seriously, this happens all the time, including straight up.
There are a bunch of common variations of this. Let [X] be some bad thing.
Here are ten examples of things people often say. Each of these ends ‘therefore you lose all credibility and [X] will never happen.’
You will continue to see such arguments, usually from people who have continuously made now-falsified predictions about AI. Do not listen to them.
We Need a Plan That Meets the Moment
A lot of evidence is in.
Many things point to the substantial possibility of superintelligence in 2027:
There are also reasons for skepticism as well. Many remain skeptical. Most reasons for skepticism are ignorance of the evidence, or failure to consider the evidence, but there are also better arguments. I am taking many of those better arguments into consideration, as well as the potential for model error. This is why I continue to believe that the probability of full automation of AI R&D in 2027 is under 50%.
In order to meet this moment, any plan has to be ready for the full automation of AI R&D inside Anthropic or OpenAI, and perhaps beyond that, by the end of 2027, resulting in rapid RSI and advancement in AI capabilities beyond human levels.
The most obvious plan is to slow or pause advancements in AI capabilities. Here is a thread of a few of the people supporting this: Nate Silver, Elizabeth Warren, the Financial Times editorial board, Ezra Klein, Eric Hoel, Robert Wiblin, Rutger Bregman, Peggy Noonan and Abdul El-Sayed.
Any plan that works on that time frame needs to include mechanisms to slow down the resulting growth in capabilities. We also need to be as ready as we can for when and if those capabilities arrive, including what happens if that development goes well. We would then face a host of other urgent and deadly problems.
The plan also cannot assume that this will happen. We cannot stop planning for futures in which AI capabilities only advance a smaller amount without anything we could call a takeoff.
And also, yes, we need to be ready for the possibility that radical improvement in AI capabilities might not upend things as much as I expect that they would, or do so as fast as I would expect that to happen. We could still blow the ‘easy’ worlds.
We cannot and must not burn our seed corn, of any type, until and unless we get much stronger evidence, either personally or globally, that we are in a world where we need to do that. It’s a marathon, folks. Until it isn’t.