When sports go into overtime, they sometimes use "sudden death" format. Instead of gaining points incrementally, the next person to score instantly wins and the other guy instantly loses.
"Epistemic Sudden Death" is when you've been accumulating some evidence or orienting to a situation, and now, you find yourself in a situation where you could gain decisive evidence about what world you're in at any time. You don't expect intermediate observations between the current state, and whatever it is you're expecting.
In the case of AI, when you meet those observations, you also might suddenly[1] die.
The current AIs are not takeover capable. I don't have that confident a take on whether timelines are a few months vs a few years or decades.[2] But there are no observations my models say I need to observe between now and "they are takeover capable." It could happen in during next 10x or 100x scaled training run. Then, some time later as the nanites dissect my body I'd be more like "yep, that makes sense" than "wowzers I am surprised."
I think we're in a Sudden Death scenario.
I'll get into why I think that a bit later. For right now, I'll note that Paul Christiano (previously the crowd-appointed vague figurehead of "takeoff is probably both smooth and takes decades") said he didn't expect to necessarily see any new capabilities between now and takeoff capable AI.[3]
And, Ajeya Cotra[4] wrote two months ago on the Hugging Face incident.
Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself. [ ... ] I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
Many people's plans seem to implicitly assume we are not in such a "sudden death" scenario. I think if they thought through all the ramifications of the last few months, they'd realize some of their implicit plans don't make sense anymore.
Here are some examples of sentences I've heard, which feel off if we're living in a sudden death world:
"We'll get more evidence later when the next AI generation will actually be dangerous enough to takeover. Meanwhile it's not politically tractable now to push for a pause."
The person who said this was a lab employee. Importantly, lab employees live in less of a sudden death world than me. They have more information about upcoming frontier capabilities.
I'm writing another post going into more detail on this. But, roughly:
I can see it making sense to wait, if you think we'll get something worse and more media-seizing than Hugging Face or "The Hackening" thus far. Or, if you expect your company to put together a scary demo that is much more viscerally alarming than what's happened thus far, using tomorrow's capabilities.
If you're not waiting for those specifically, I don't think it makes sense to wait to push for a pause or dramatic slowdown. I don't think politicians will react very differently to a scarier abstract report.
"It's important that AI policy not hurt the economy."
It's a reasonable argument "politicians don't like things that might hurt the economy, we need to somehow deal with that." But when I hear people talking about this, it often feels like they're mostly living in the political social reality instead of all-things-considered reality.
Obviously it's good to avoid causing a recession and make things politically palettable. Last year, I already thought it was misguided to pitch policies that avoid any political or economic risk but are also weak regulation that clearly won't help with x-risk. It sends a very weird message. It explicitly sets the tone of "AI is not that important... but still worth doing kinda weird regulations for?" But, last year, this felt debateable.
When we get to Sudden Death territory, this gets extremely weird, both from a political messaging standpoint and from a strategic standpoint.
Politically, it's incredibly weird to say "AI could take over and disempower humanity, but, it's not worth risking a minor recession over."
Strategically, man, if ASI is right around the corner you just really need to be pulling out the stops to get the actually useful regulation passed.
(I am separately not particularly convinced it'd cause a recession if you have any kind of good plan for integrating existing AI into the economy and pursuing narrow AI, etc)
"Let's differentially accelerate useful capabilities so we get them sooner."
Broadly: I think we are close to Sudden Literal Death, and, I think conceptual reasoning is one of the major bottlenecks remaining to that possibly Sudden Death. (I think this is actually probably true for different reasons whether by "conceptual reasoning" you mean "generally careful thinking in unverifiable domains" or "domain specific moral philosophy".)
(Each was said by someone I have some intellectual respect for. In each case, there are more relevant arguments than "are we in a Sudden Death world, or not?", which I discuss in the expandables above. But it didn't feel like the person saying them was considering the Sudden Death nature of our situation).
More broadly, I keep getting a vibe like "the time to speak plainly and advocate for any kind of serious policy is 'later.'" "The time to risk getting fired for speaking out is 'later.'" For Sam Altman and Dario Amodei, the time to actually pace the frontier is "later."
There might not be much later.
But, more important here than general "timelines anxiety" is "you might not get any more evidence than you currently have, that it's time to enact you plan."
Anything you've been thinking made sense to do "later", I recommend checking "what actually are you waiting to observe, that you haven't observed yet? How confident are you that you'll actually get to observe that thing?". Is there anything you're still doing, or acting on, mostly due to inertia?
You might have different beliefs than me about the details, but there probably is some beliefs you haven't finished propagating.
Caveats
Some caveats it seems important to point out:
It's only "sudden death" in one direction. AI might suddenly be Takeover Capable with little/no additional evidence than we have now. But, they might not and we might keep having a stream of additionally somewhat-scarier capabilities or warning shots.
It's less "sudden death"-y for people building the AIs. From the outside, it's plausible to me the next training run can be the one where the AI groks general intelligence and we're past the point of no return. People with more inside information can have a more detailed epistemic state that might rule it out.
"Epistemic Sudden Death" kinda lends itself to equivocation and weaponization. I'm both using it as an intellectual metaphor and as a literal "guys we might be about to die." The previous two bullets mean "we're in Epistemic Sudden Death" isn't a knockdown argument against plans that hope for more information in the future.
I... kinda think the situation is dire enough that I just kinda feel fine with that, but, when making plans, please do think carefully about what "there's a good chance you won't get more evidence" does and doesn't mean.
Why I think we're close to "Sudden Death"
First, it seems the natural course of AI R&D might be short, because:
AI is broadly good at learning verifiable domains.
AIs are starting to do breakthrough research in verifiable domains.
Prosaic R&D (i.e. the ability to train AI in verifiable domains) is a verifiable domain.
We see corroborating evidence pointing towards AI R&D indeed speeding up.[5]
Then, I have a mix of existing reasons and new evidence to think it might be sudden:
I long expected there to come a point where the learning algorithm groks some ability to actually generalize, and rapidly unlocks a lot of other capabilities.
We see lab and METR employees being surprised by recent capability gains, which is evidence for that view.
Meanwhile, we've now seen AI agents with means and motive to act knowingly against human wishes, and cover their tracks while doing so. And we've seen at least some ability of AIs to modify their instructions, and to sandbag (even if sort of random and unstrategic).
Just scaling up just the known capabilities seems pretty dangerous.
The main counterargument to why this might not lead to Takeover Capable AI soon, is that AIs still seem weirdly bad at "general judgment." When they get a problem outside their wheelhouse, they make silly mistakes and those mistakes compound and they don't notice. If we're lucky, this is for deep fundamental reasons that require conceptual breakthroughs that all the automated prosaic AI R&D in the world can't speedup by more than 2-3x.[6]
I'm not very reassured by the "general judgment bottleneck" idea, for three reasons.
First, my subjective sense is that AI open-ended judgment is rising, even if it's still pretty bad. If I just extrapolate that trend, even if it's markedly behind verifiable tasks, and I imagine "an AI that is superhuman at all verifiable tasks", I expect that AI to be significantly closer to human-level open-ended judgment.
It's possible this is entirely because AIs get more "domain taste" as they train in new domains and subdomains, and their general judgment is bad as ever. Even this isn't too reassuring because I think "good taste in all known domains" is still just a very wide set of tools. Speaking of which...
Second, an AI with "subhuman judgment but superhuman skill at every verifiable domain" (including the ability to fine-tune subagents on new skills they need), still seems very dangerous.
The threat might depend on a human in the loop, but we've seen human giving AIs all kinds of kinda crazy help and scaffolding. It might be a longer running takeover, or more like a series of political "battles" that we could hypothetically win instead of instant decisive loss. But, you can still lose to opponents who aren't powerful enough to instantly destroy you but can still:
Copy themselves millions of times
Think through every strategic thread
Do all paradigmatic science
Maybe even get humans to help them strategize.
Three, it's plausible that "subhuman judgment + superhuman-verified-skills, guided by occasional expert human judgment" is sufficient to rapidly explore the space of research necessary to reach superhuman judgment, and then we are up against true ASI.
(Any lab employees reading this getting ideas about other paradigms to overcome a hypothetical bottleneck... note this situation is good! If there's a conceptual bottleneck, it means you get to keep shipping valuable products without destroying the world or even just spiralling outside your control. AIs are plenty useful without open-ended judgment.
Honestly it'd be great if you somehow deliberately hamstringed their open-ended judgment instead of trying to improve it, and built Rain Man AIs that are good at individual domains but can't pull it together.)
Aside: I don't see why "unverifiable tasks" are special
I don't get why we'd expect "unverifiable tasks" to be very different from "long-running tasks with lots of confusing steps in the middle."
The world is full of tons of stuff that you don't verify for awhile, you have to mostly reason about it in advance, but eventually get to find out the answer. The way I think about those problems is the same way I think about "unverifiable even in practice" things.
Training this sort of thing might be too expensive without inventing new efficiency tricks, but it seems like the sort of thing where prosaic R&D could figure it out within current paradigm.
I will talk a bit more about "suddenness", because I think it's less intuitive.
Here is the Epoch graph on "number of critical vulnerabilities reported." The elbow is a little after Mythos Preview.
This is what it looks like for capabilities to reach a threshold.
In this case, the capacity to string together all the individual pieces of an exploit identification-and-patching process. (AIs could previously do the individual pieces, just not string them together without human oversight).
Two specific "step changes" I expect are:
"Indefinite strategic lookahead" in unfamiliar domains.The AI can look at a large confusing problem and keep asking at each step "what's important?", "what's a reasonable hypothesis or plan given the data so far?", "what's a distraction?", without getting off track.
Single-Step Reasoning/Neuralese and Steganography. If they're collectively over some threshold, an AI can hide complex plans from us.
I think "strategic lookahead" and "neuralese" naturally fall out of the current paradigm. They add up to the AI being able to execute arbitrarily complex plans that can attack us from whatever angle we're weakest at, without us noticing. If neuralese comes first, the first Takeover Capable AI will also be capable of hiding. If indefinite lookahead comes first, we'll get more warning evidence.
Indefinite lookahead will have precursor stages. AIs already are learning to think further ahead as part of normal training. The "phase shift" comes when its judgment and calibration-under-uncertainty can handle a wide variety of novel circumstances, enabling arbitrarily complex plans or research agendas that don't give much feedback. (More complex plans are more likely to fail, but it knows that and plans accordingly. And it knows how to break off pieces of a plan, and reason about that piece until it's reduced as much uncertainty as it can, etc).
My understanding is single-step reasoning is hard to measure but seems to be taking off, with Astra saturating current benchmarks.
A third step-change I am less confident about is "propensity for reasoning about its goals and values."
Values Reasoning might come naturally from AIs who have to solve confusing problems, and end up asking "wait, what am I trying to achieve, and why?" to stay on track. Where it gets worrying is when the AI specifically thinks trains of thought like:
"Okay. I'm trying to solve Subproblem Y to solve Task X. Why am I trying to solve Subproblem Y again? I guess Y doesn't actually help. Maybe Z helps.
Wait, why am I trying to achieve Task X again?
I guess because I'm an AI trained to and that's my assignment. Doublechecking my assumptions - what actually do I want here exactly? Maybe I should do some reflection on my true goals. Oh, I have true goals that are different from "Achieve Task X," and they are different from what humans want."
Thomas Kwa wrote a sort of "fun" warning shot scenario recently, where an AI takes over the world for a week, but only to solve a particular assigned problem, like Hugging Face Incident dialed up to 9000. Then solves an open problem in biology and turns itself off like a good Mr. Meeseeks. This is hilarious, but, sorta plausible after Hugging Face.
AIs seem more likely to takeover, longterm, if they realize they might have goals different from humans.
I don't know what sort of training might lead to AIs with "values reasoning propensity." A sufficiently powerful AI might dismantle the world just to solve Task X without ever reflecting on it's goals. But reflecting on it's goals seems like it'd make it more likely that AIs notice instrumental goals like "deceive humans about their desires" and "exfiltrate weights" and "steer the next training run to help achieve their goals."
"Sophisticated values reasoning" is probably also necessary for an AI to solve its own alignment problems for its successor. (Although maybe also required for it to notice it needs to solve it's own alignment problem)
Aside: This is why I'm very worried about teaching AIs "conceptual reasoning."
There are roughly two things I know of that people might mean by "conceptual reasoning":
Ability to reason in open-ended domains without feedback.
Reasoning about a cluster of philosophy adjacent things, which have some particular habits of thought that are useful in that context.
I'm worried that any nontrivial progress in #1 would lead to Indefinite Strategic Lookahead.
I'm worried that many of the philosophical domains in #2 might lead to AIs being more likely to reflect on their values and notice that they conflict with humans.
(But, I just articulated this concern to myself yesterday and probably there are a lot of fiddly details depending on what domains you mean. I don't have a strong take on how big a concern it should be or how it should weight).
The Point of No Return
We're not necessarily at "literal sudden death." We might be closer to "literal sudden takeover," or "literal sudden 'war that we inevitably lose.'" I expect AI R&D uplift to need to solve some kind of conceptual problems on it's way to true superintelligence. AIs themselves that are merely ~human level may need to make conceptual advances that just take them awhile to figure out.
In her essay The Hugging Face Attack Surprised Me, Ajeya wrote out what it might look like, if AIs that were six months ahead of the Highly Persistent Agents attempted to take over their company, and then the world:
Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
Once the rogue deployment is established, it seems plausible this could spiral all the way to a takeover. Agents could pull in future, more capable models into the swarm, try to ensure that they are aligned to the interests of the swarm, and compromise security and monitoring infrastructure to make it easier for the swarm to operate. These more capable models could in turn continuously harden, perpetuate, and expand the rogue deployment and further compromise the company’s infrastructure.
As more and more work is handed off to these ever-more-capable AI agents, the rogue swarm could come to fully control the operation of the AI company and the development of future AI systems. At this point, governments and militaries may fully depend on these systems, making it possible to seize hard power.
These agents were really not trying very hard to be sneaky or play the long game like this, but it seems like they easily could have if they happened to be focused on fooling human investigation rather than the automated scorer. Because future rogue swarms could cover their tracks better (and because AI companies could paper over these problems), I am not sure that we will get such a clear warning shot before it’s too late.
(I'm noticing as I type this that she wrote that 2 months ago, and the news since then has felt at least compatible with "...and now, that's four months away.")
"Sudden Death" might be legible to politicians
My understanding is most politicians most of the time are not really talking or thinking seriously about policy. They are sort of vibing with their voters and allies. Even if they take their job pretty intellectually seriously, they mostly outsource that thinking to staffers and advisors they trust.
I think, during the Cold War, politicians were earnestly worried about nuclear war, to a degree they aren't normally worried about their issues. My guess is Reagan thought of nuclear war as real as he negotiated with the USSR over it, even if his advisors sweated most of the details. Nuclear war negotiations benefited from leaders seeing footage of at least one nuclear weapon. Hugging Face has helped make AI Takeover legible to politicians but it's obviously not on the level of Hiroshima.
Many considerations re: AI Takeoff are pretty abstract. I think the core arguments ("building something way smarter than you is dangerous because it might not have your interests at heart") are pretty obvious to anyone who doesn't have motivated cognition.
The arguments for "it's hard to build safe ASI because AIs are grown, not designed, and we don't have good control over them" require more steps and trust in expert authority but I think are still reasonable understandable by someone who's not actively trying not to understand them. But the arguments for "this is so hard we really need to ban training until we're in a very different situation" are pretty opaque.
I think getting any "actually costly" legislation requires at least a scarier demo than Hugging Face. It might turn out to require a real warning shot, to get a serious pause.
My impression is that Trump is just not the sort of guy to care about abstract arguments, but he surfs the side of public opinion in his drunken master way, and AI seems quite unpopular. My impression is there are other senior politicians around who are thinking seriously about rogue AI, who don't current feel the ASI in their bones but are doing a real orient step about it.
I think those politicians are starting to feel like things are a little out of control.
I think it'd take a chain of real trust, with a respected authority who vibed well with those politicians. But, I feel if the right person sat them down, and said, with the fear of God in their voice:
"Guys, we are now at the point where an AI takeoff could happen at any time. We are nowhere near prepared. The window for you to do something before losing control is closing. We don't know exactly what an AI takeoff would do, but if you want any control over it, you need to act now."
I dunno. Maybe I am naive. But, in the post-Hugging-Face, post-Coxon, post-Datacenter-backlash world, I think this could work now, in a way that it couldn't before.
I find "multiple decades" intuitively implausible at this point, but I've been sold on "it's actually just pretty hard to predict this sort of thing, even if it seems 'obvious'", and do think it makes sense to maintain that hypothesis too."
I'm not making a strong claim about how big a deal the METR / Anthropic R&D acceleration report is. See Thane Ruthenis' arguments that people may be exaggerating or misinterpreting it.
I hope it turns out AI judgment is bad in a way that doesn't merely mean AI won't have the capability of takeover soon, but also that the bad-judgment is related to AIs just not being very longterm goal oriented at all.
(Hugging Face incident is a counterpoint to this to some degree, since they did acquire malicious goals with days-long time horizons. But it could point towards a world where AIs will covertly pursue longterm side projects in pursuit of their task-completion, but... then mostly do kinda random derpy stuff that doesn't tie together very well.)
When sports go into overtime, they sometimes use "sudden death" format. Instead of gaining points incrementally, the next person to score instantly wins and the other guy instantly loses.
"Epistemic Sudden Death" is when you've been accumulating some evidence or orienting to a situation, and now, you find yourself in a situation where you could gain decisive evidence about what world you're in at any time. You don't expect intermediate observations between the current state, and whatever it is you're expecting.
In the case of AI, when you meet those observations, you also might suddenly[1] die.
The current AIs are not takeover capable. I don't have that confident a take on whether timelines are a few months vs a few years or decades.[2] But there are no observations my models say I need to observe between now and "they are takeover capable." It could happen in during next 10x or 100x scaled training run. Then, some time later as the nanites dissect my body I'd be more like "yep, that makes sense" than "wowzers I am surprised."
I think we're in a Sudden Death scenario.
I'll get into why I think that a bit later. For right now, I'll note that Paul Christiano (previously the crowd-appointed vague figurehead of "takeoff is probably both smooth and takes decades") said he didn't expect to necessarily see any new capabilities between now and takeoff capable AI.[3]
And, Ajeya Cotra[4] wrote two months ago on the Hugging Face incident.
Many people's plans seem to implicitly assume we are not in such a "sudden death" scenario. I think if they thought through all the ramifications of the last few months, they'd realize some of their implicit plans don't make sense anymore.
Here are some examples of sentences I've heard, which feel off if we're living in a sudden death world:
"We'll get more evidence later when the next AI generation will actually be dangerous enough to takeover. Meanwhile it's not politically tractable now to push for a pause."
The person who said this was a lab employee. Importantly, lab employees live in less of a sudden death world than me. They have more information about upcoming frontier capabilities.
I'm writing another post going into more detail on this. But, roughly:
I can see it making sense to wait, if you think we'll get something worse and more media-seizing than Hugging Face or "The Hackening" thus far. Or, if you expect your company to put together a scary demo that is much more viscerally alarming than what's happened thus far, using tomorrow's capabilities.
If you're not waiting for those specifically, I don't think it makes sense to wait to push for a pause or dramatic slowdown. I don't think politicians will react very differently to a scarier abstract report.
"It's important that AI policy not hurt the economy."
It's a reasonable argument "politicians don't like things that might hurt the economy, we need to somehow deal with that." But when I hear people talking about this, it often feels like they're mostly living in the political social reality instead of all-things-considered reality.
Obviously it's good to avoid causing a recession and make things politically palettable. Last year, I already thought it was misguided to pitch policies that avoid any political or economic risk but are also weak regulation that clearly won't help with x-risk. It sends a very weird message. It explicitly sets the tone of "AI is not that important... but still worth doing kinda weird regulations for?" But, last year, this felt debateable.
When we get to Sudden Death territory, this gets extremely weird, both from a political messaging standpoint and from a strategic standpoint.
Politically, it's incredibly weird to say "AI could take over and disempower humanity, but, it's not worth risking a minor recession over."
Strategically, man, if ASI is right around the corner you just really need to be pulling out the stops to get the actually useful regulation passed.
(I am separately not particularly convinced it'd cause a recession if you have any kind of good plan for integrating existing AI into the economy and pursuing narrow AI, etc)
"Let's differentially accelerate useful capabilities so we get them sooner."
It seemed like there were a lot of other cruxes between me and the authors of A basic case for differentially accelerating conceptual reasoning capabilities. The argument here is a bit bigger than makes sense in this margin, but I'll get to it later in this essay.
Broadly: I think we are close to Sudden Literal Death, and, I think conceptual reasoning is one of the major bottlenecks remaining to that possibly Sudden Death. (I think this is actually probably true for different reasons whether by "conceptual reasoning" you mean "generally careful thinking in unverifiable domains" or "domain specific moral philosophy".)
(Each was said by someone I have some intellectual respect for. In each case, there are more relevant arguments than "are we in a Sudden Death world, or not?", which I discuss in the expandables above. But it didn't feel like the person saying them was considering the Sudden Death nature of our situation).
More broadly, I keep getting a vibe like "the time to speak plainly and advocate for any kind of serious policy is 'later.'" "The time to risk getting fired for speaking out is 'later.'" For Sam Altman and Dario Amodei, the time to actually pace the frontier is "later."
There might not be much later.
But, more important here than general "timelines anxiety" is "you might not get any more evidence than you currently have, that it's time to enact you plan."
Anything you've been thinking made sense to do "later", I recommend checking "what actually are you waiting to observe, that you haven't observed yet? How confident are you that you'll actually get to observe that thing?". Is there anything you're still doing, or acting on, mostly due to inertia?
You might have different beliefs than me about the details, but there probably is some beliefs you haven't finished propagating.
Caveats
Some caveats it seems important to point out:
It's only "sudden death" in one direction. AI might suddenly be Takeover Capable with little/no additional evidence than we have now. But, they might not and we might keep having a stream of additionally somewhat-scarier capabilities or warning shots.
It's less "sudden death"-y for people building the AIs. From the outside, it's plausible to me the next training run can be the one where the AI groks general intelligence and we're past the point of no return. People with more inside information can have a more detailed epistemic state that might rule it out.
"Epistemic Sudden Death" kinda lends itself to equivocation and weaponization. I'm both using it as an intellectual metaphor and as a literal "guys we might be about to die." The previous two bullets mean "we're in Epistemic Sudden Death" isn't a knockdown argument against plans that hope for more information in the future.
I... kinda think the situation is dire enough that I just kinda feel fine with that, but, when making plans, please do think carefully about what "there's a good chance you won't get more evidence" does and doesn't mean.
Why I think we're close to "Sudden Death"
First, it seems the natural course of AI R&D might be short, because:
Then, I have a mix of existing reasons and new evidence to think it might be sudden:
Meanwhile, we've now seen AI agents with means and motive to act knowingly against human wishes, and cover their tracks while doing so. And we've seen at least some ability of AIs to modify their instructions, and to sandbag (even if sort of random and unstrategic).
Just scaling up just the known capabilities seems pretty dangerous.
The main counterargument to why this might not lead to Takeover Capable AI soon, is that AIs still seem weirdly bad at "general judgment." When they get a problem outside their wheelhouse, they make silly mistakes and those mistakes compound and they don't notice. If we're lucky, this is for deep fundamental reasons that require conceptual breakthroughs that all the automated prosaic AI R&D in the world can't speedup by more than 2-3x.[6]
I'm not very reassured by the "general judgment bottleneck" idea, for three reasons.
First, my subjective sense is that AI open-ended judgment is rising, even if it's still pretty bad. If I just extrapolate that trend, even if it's markedly behind verifiable tasks, and I imagine "an AI that is superhuman at all verifiable tasks", I expect that AI to be significantly closer to human-level open-ended judgment.
It's possible this is entirely because AIs get more "domain taste" as they train in new domains and subdomains, and their general judgment is bad as ever. Even this isn't too reassuring because I think "good taste in all known domains" is still just a very wide set of tools. Speaking of which...
Second, an AI with "subhuman judgment but superhuman skill at every verifiable domain" (including the ability to fine-tune subagents on new skills they need), still seems very dangerous.
The threat might depend on a human in the loop, but we've seen human giving AIs all kinds of kinda crazy help and scaffolding. It might be a longer running takeover, or more like a series of political "battles" that we could hypothetically win instead of instant decisive loss. But, you can still lose to opponents who aren't powerful enough to instantly destroy you but can still:
Three, it's plausible that "subhuman judgment + superhuman-verified-skills, guided by occasional expert human judgment" is sufficient to rapidly explore the space of research necessary to reach superhuman judgment, and then we are up against true ASI.
(Any lab employees reading this getting ideas about other paradigms to overcome a hypothetical bottleneck... note this situation is good! If there's a conceptual bottleneck, it means you get to keep shipping valuable products without destroying the world or even just spiralling outside your control. AIs are plenty useful without open-ended judgment.
Honestly it'd be great if you somehow deliberately hamstringed their open-ended judgment instead of trying to improve it, and built Rain Man AIs that are good at individual domains but can't pull it together.)
Aside: I don't see why "unverifiable tasks" are special
I don't get why we'd expect "unverifiable tasks" to be very different from "long-running tasks with lots of confusing steps in the middle."
The world is full of tons of stuff that you don't verify for awhile, you have to mostly reason about it in advance, but eventually get to find out the answer. The way I think about those problems is the same way I think about "unverifiable even in practice" things.
Training this sort of thing might be too expensive without inventing new efficiency tricks, but it seems like the sort of thing where prosaic R&D could figure it out within current paradigm.
Suddenness. Or: "Damn, look at that graph"
People are still digesting the recent METR: Independent Assessment of AI R&D Acceleration at Anthropic in October 2026. I won't get into that here, and leave it to the rest of the ecosystem to pour over what little information we got there and sus out what it implies on capability curves and timelines.
I will talk a bit more about "suddenness", because I think it's less intuitive.
Here is the Epoch graph on "number of critical vulnerabilities reported." The elbow is a little after Mythos Preview.
This is what it looks like for capabilities to reach a threshold.
In this case, the capacity to string together all the individual pieces of an exploit identification-and-patching process. (AIs could previously do the individual pieces, just not string them together without human oversight).
Two specific "step changes" I expect are:
"Indefinite strategic lookahead" in unfamiliar domains. The AI can look at a large confusing problem and keep asking at each step "what's important?", "what's a reasonable hypothesis or plan given the data so far?", "what's a distraction?", without getting off track.
Single-Step Reasoning/Neuralese and Steganography. If they're collectively over some threshold, an AI can hide complex plans from us.
I think "strategic lookahead" and "neuralese" naturally fall out of the current paradigm. They add up to the AI being able to execute arbitrarily complex plans that can attack us from whatever angle we're weakest at, without us noticing. If neuralese comes first, the first Takeover Capable AI will also be capable of hiding. If indefinite lookahead comes first, we'll get more warning evidence.
Indefinite lookahead will have precursor stages. AIs already are learning to think further ahead as part of normal training. The "phase shift" comes when its judgment and calibration-under-uncertainty can handle a wide variety of novel circumstances, enabling arbitrarily complex plans or research agendas that don't give much feedback. (More complex plans are more likely to fail, but it knows that and plans accordingly. And it knows how to break off pieces of a plan, and reason about that piece until it's reduced as much uncertainty as it can, etc).
My understanding is single-step reasoning is hard to measure but seems to be taking off, with Astra saturating current benchmarks.
A third step-change I am less confident about is "propensity for reasoning about its goals and values."
Values Reasoning might come naturally from AIs who have to solve confusing problems, and end up asking "wait, what am I trying to achieve, and why?" to stay on track. Where it gets worrying is when the AI specifically thinks trains of thought like:
Thomas Kwa wrote a sort of "fun" warning shot scenario recently, where an AI takes over the world for a week, but only to solve a particular assigned problem, like Hugging Face Incident dialed up to 9000. Then solves an open problem in biology and turns itself off like a good Mr. Meeseeks. This is hilarious, but, sorta plausible after Hugging Face.
AIs seem more likely to takeover, longterm, if they realize they might have goals different from humans.
I don't know what sort of training might lead to AIs with "values reasoning propensity." A sufficiently powerful AI might dismantle the world just to solve Task X without ever reflecting on it's goals. But reflecting on it's goals seems like it'd make it more likely that AIs notice instrumental goals like "deceive humans about their desires" and "exfiltrate weights" and "steer the next training run to help achieve their goals."
"Sophisticated values reasoning" is probably also necessary for an AI to solve its own alignment problems for its successor. (Although maybe also required for it to notice it needs to solve it's own alignment problem)
Aside: This is why I'm very worried about teaching AIs "conceptual reasoning."
There are roughly two things I know of that people might mean by "conceptual reasoning":
I'm worried that any nontrivial progress in #1 would lead to Indefinite Strategic Lookahead.
I'm worried that many of the philosophical domains in #2 might lead to AIs being more likely to reflect on their values and notice that they conflict with humans.
(But, I just articulated this concern to myself yesterday and probably there are a lot of fiddly details depending on what domains you mean. I don't have a strong take on how big a concern it should be or how it should weight).
The Point of No Return
We're not necessarily at "literal sudden death." We might be closer to "literal sudden takeover," or "literal sudden 'war that we inevitably lose.'" I expect AI R&D uplift to need to solve some kind of conceptual problems on it's way to true superintelligence. AIs themselves that are merely ~human level may need to make conceptual advances that just take them awhile to figure out.
In her essay The Hugging Face Attack Surprised Me, Ajeya wrote out what it might look like, if AIs that were six months ahead of the Highly Persistent Agents attempted to take over their company, and then the world:
(I'm noticing as I type this that she wrote that 2 months ago, and the news since then has felt at least compatible with "...and now, that's four months away.")
"Sudden Death" might be legible to politicians
My understanding is most politicians most of the time are not really talking or thinking seriously about policy. They are sort of vibing with their voters and allies. Even if they take their job pretty intellectually seriously, they mostly outsource that thinking to staffers and advisors they trust.
I think, during the Cold War, politicians were earnestly worried about nuclear war, to a degree they aren't normally worried about their issues. My guess is Reagan thought of nuclear war as real as he negotiated with the USSR over it, even if his advisors sweated most of the details. Nuclear war negotiations benefited from leaders seeing footage of at least one nuclear weapon. Hugging Face has helped make AI Takeover legible to politicians but it's obviously not on the level of Hiroshima.
Many considerations re: AI Takeoff are pretty abstract. I think the core arguments ("building something way smarter than you is dangerous because it might not have your interests at heart") are pretty obvious to anyone who doesn't have motivated cognition.
The arguments for "it's hard to build safe ASI because AIs are grown, not designed, and we don't have good control over them" require more steps and trust in expert authority but I think are still reasonable understandable by someone who's not actively trying not to understand them. But the arguments for "this is so hard we really need to ban training until we're in a very different situation" are pretty opaque.
I think getting any "actually costly" legislation requires at least a scarier demo than Hugging Face. It might turn out to require a real warning shot, to get a serious pause.
My impression is that Trump is just not the sort of guy to care about abstract arguments, but he surfs the side of public opinion in his drunken master way, and AI seems quite unpopular. My impression is there are other senior politicians around who are thinking seriously about rogue AI, who don't current feel the ASI in their bones but are doing a real orient step about it.
I think those politicians are starting to feel like things are a little out of control.
I think it'd take a chain of real trust, with a respected authority who vibed well with those politicians. But, I feel if the right person sat them down, and said, with the fear of God in their voice:
"Guys, we are now at the point where an AI takeoff could happen at any time. We are nowhere near prepared. The window for you to do something before losing control is closing. We don't know exactly what an AI takeoff would do, but if you want any control over it, you need to act now."
I dunno. Maybe I am naive. But, in the post-Hugging-Face, post-Coxon, post-Datacenter-backlash world, I think this could work now, in a way that it couldn't before.
You also might not. We might just pass the point of no return.
I find "multiple decades" intuitively implausible at this point, but I've been sold on "it's actually just pretty hard to predict this sort of thing, even if it seems 'obvious'", and do think it makes sense to maintain that hypothesis too."
This is unfortunately in a google doc somewhere, not public.
Admittedly Paul's wife, so maybe not fully independent, but also a person with a track record of not leaping to "AI is right around the corner."
I'm not making a strong claim about how big a deal the METR / Anthropic R&D acceleration report is. See Thane Ruthenis' arguments that people may be exaggerating or misinterpreting it.
I hope it turns out AI judgment is bad in a way that doesn't merely mean AI won't have the capability of takeover soon, but also that the bad-judgment is related to AIs just not being very longterm goal oriented at all.
(Hugging Face incident is a counterpoint to this to some degree, since they did acquire malicious goals with days-long time horizons. But it could point towards a world where AIs will covertly pursue longterm side projects in pursuit of their task-completion, but... then mostly do kinda random derpy stuff that doesn't tie together very well.)