This essay is a combination of personal notes, discussions I had at the MAIA/AIST workshop, and slides from a talk I gave with JHU ASBS and BlueDot Impact.
Premise
A common theme I noticed, particularly among viewpoints with short timelines, is the idea that the introduction of ASI is inherently and instantaneously the loss condition. This stems, I think, from Yudkowsky-ian philosophy that ASI = unavoidable doom.[1] I think this defeatist attitude is counterproductive and detracts from real, important, effective avenues of action.
A superintelligence is bounded through interaction with the physical world; there are fundamental limitations on feedback loops which are crucial to enact any kind of influence.[2] ASI on a laptop in a concrete room in 1850 cannot escape with any high probability, and even if it did, it would take a very long time for it to do anything – conceivably much longer than the physical lifetime of the laptop. In well-defined worlds like video games, it is not hard to imagine scenarios where you are mathematically guaranteed to lose even with perfect skill and play – for example, if the ball in pong is much much faster than your paddle. There is an analogy here to galactic algorithms in computer science[3]: just because an ASI has an algorithm that scales better asymptotically doesn't mean the constant factors (e.g. time needed to conduct experiments; time needed to manufacture a quantity of robots) allow it to win within a finite time horizon.
In the present world, incidents like HuggingFace[4] can be largely attributed to 1. Ease: cybersecurity and the internet is an extremely accessible attack surface for AI, 2. Lack of foresight: most of the internet was not designed in anticipation of highly intelligent adversaries and widespread integration into society (cybersecurity is almost always reactionary), and 3. Ubiquity: the internet is a widespread and well-connected infrastructure tied deeply to society. As I, and many others have previously wrote, it is extremely difficult for ASI, if incarnated right now, to rapidly destroy all of humanity, or even do more mundane things like synthesize a virus or take over a factory. It would almost necessarily have to use humans, through superpersuasion or blackmail, limiting the effective set of “hands” to a few experts which it must coordinate… not impossible, but inherently gated by the limitations of humans as tools: we suck at following instructions, general competency, etc. Physical limits also significantly bound the influence of ASI in propagation time and physical limits (e.g. Control Theory); no matter what, noise and experiment mean you cannot have iteration speeds past a certain limit nor achieve results under a certain statistical confidence. All of this is further compounded by the need to avoid detection prior to achieving independence or dominance over humanity. This is why “gradual disempowerment” scenarios are generally more favored.[5]
However, as I have previously argued, this will change.[6] It’s only through luck that the robotics and biotech industries have not achieved the same widespread diffusion that the internet has (at this point in time), and all signs point towards this changing rapidly. This presents us a simple theory of change: designing these new systems with foresight, similarly to how we treat other high-risk technologies like nuclear weapons. Through proactively “hardening” these systems – these “hands” for AI – can we provide an effectively externally-imposed slowdown (allowing other actions) or even shut down ASI?
Adjusting time-to-doom through slowing the trajectory of ASI independence
We have three basic actions to combat against ASI: slow it, see it, stop it. Suppose a misaligned ASI has escaped: we define the win condition as a physical base (power, chips, robots, etc.) entirely independent of humanity. Here, we have four steps:
Building/accumulating physical assets takes some amount of time.
Each month, there is some chance someone notices.
Each time someone notices, some action may be taken for a response.
A response either terminates the ASI, or sets it back.[7]
where the first three steps can be convolved into one number: the number of times we expect to catch an attempt and act on it. Then,
where T is months to build, λ is the monthly chance of being noticed, a is the chance a notice is acted on, and d is the chance a response terminates the ASI. That gives three dials: slow it (T), see it (λ), and stop it (a and d). Then, we can make some starting estimates:
Two covert programmes hidden 5 and 15 years[9] imply 0.5% to 2%
Notice acted on (a)
42%
15% to 75%
Guess; no reference class
Response ends it (d)
22%
5% to 60%
Guess; large botnets came back after takedowns[10]
Unmodelled route exists
15%
Fixed; tested from 0% to 100%
Guess
Catches per attempt (X)
0.4
0.03 to 3.9
Result of the rows above
but also sample from a wide range for all parameters. Simulating many different worlds with different combinations of parameters:
Clearly, it doesn’t seem terribly unlikely (black line = even odds) that we can indeed detect and catch nascent ASI even in the absence of interventions (orange dots). Nevertheless, the odds are not great. To model the effect of hardening (green dots), we can adjust the parameters:
Dial
Factor (median; range)
Median world
Basis
Evidence
T
×1.8 per stage; 1.1 to 3
45 → 95 months
Nuclear projects after 1970 took 2.4× longer (17 years against 7)[11]
Loose
T
÷6.3; 2 to 20
Included in the row above
None; today’s nearest control stops about 5% of orders
Guess
λ
×3.5; 1.5 to 8
2% → 7%
Monitoring speeds detection about 2× (fraud: 6 months against 12[12]; intrusions: 9 days against 20[13])
None; kept modest because strong takedowns have failed
Guess
X
0.4 → 5.5
Follows from the rows above
Some interesting results follow. Win/loss probability in worlds is largely a two-state readout, with the majority of probability mass either on 0% or 100%; hardening therefore flips many worlds from certain doom to certain success.
The median time-to-notice required for even odds also increases to historically plausible levels:
Finally, we can decompose the expected effect of interventions at each level:
and, as expected, speedups in thinking speed quickly saturate and do not increase ASI-win probability:
(1x ASI = 100x human)
Lastly, I look at some pessimistic and optimistic assumptions:
Input
Fast
Central
Bottleneck
A route the model misses exists
60%
15%
3%
Robot doubling time, months
0.5 to 6
4 to 36
9 to 60
How far physical steps compress
5 to 300×
1.2 to 10×
1.05 to 3×
Monthly chance of notice
A quarter of central
About 2%
Double central
A response ends it
0.5% to 15%
5% to 60%
15% to 80%
Hardening factors
Weak
As in your table
Stronger
Median months to build
7
45
109
Result, without → with hardening
99% → 94%
72% → 26%
39% → 5%
Obviously, if most probability is dominated by definitionally unforeseeable routes, hardening does not provide much gain. Outside of this, however, hardening seems to robustly increase our chances of winning. Therefore, it seems like hardening may represent a third option to policy and technical alignment work amenable to great interdisciplinary efforts.
Thanks to Elaine Ly and Peyton Li for comments.
Eliezer Yudkowsky, “AGI Ruin: A List of Lethalities,” LessWrong, June 5, 2022, https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities (point 2: losing to a high-powered cognitive system “looks at least as deadly as ‘everybody on the face of the Earth suddenly falls over dead within the same second’”).
Richard J. Lipton, “Galactic Algorithms,” Gödel’s Lost Letter and P=NP, October 23, 2010, https://rjlipton.com/2010/10/23/galactic-algorithms/ (an algorithm “wonderful in its asymptotic behavior” but never used in practice).
Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud, “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development,” arXiv:2501.16946, January 28, 2025, https://arxiv.org/abs/2501.16946 .
Biopreparat was created on April 24, 1974 (Glavmikrobioprom Order 131 DSP), per “Biopreparat,” Wikipedia, citing Anthony Rimmington, The Soviet Union’s Invisible Weapons of Mass Destruction (Springer, 2021), https://en.wikipedia.org/wiki/Biopreparat , and was exposed when Vladimir Pasechnik defected to Britain in late October 1989, per David E. Hoffman, The Dead Hand (Doubleday, 2009), excerpted by the National Security Archive, May 6, 2010, https://nsarchive2.gwu.edu/NSAEBB/NSAEBB315/ . Aum Shinrikyo began producing botulinum toxin and phosgene in 1990 and carried out the Tokyo subway sarin attack on March 20, 1995, per “Tokyo subway sarin attack,” Wikipedia, https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack ; see also Richard Danzig et al., Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons, 2nd ed. (CNAS, 2012), https://www.cnas.org/publications/reports/aum-shinrikyo-second-edition-english .
Jacques E. C. Hymans, “The Threat of Nuclear Proliferation: Perception and Reality,” Ethics & International Affairs 27, no. 3 (2013): 281–298, https://doi.org/10.1017/S089267941300021X (all dedicated projects launched before 1970 succeeded, averaging about seven years to a first test; only three of the ten launched since 1970 succeeded, needing “an average of about seventeen years”). See also Hymans, Achieving Nuclear Ambitions (Cambridge University Press, 2012).
Association of Certified Fraud Examiners, Occupational Fraud 2024: A Report to the Nations, as summarized in Alexander Gershner, “Be Proactive & Limit Asset Misappropriation Schemes,” Forvis Mazars, August 30, 2024, https://www.forvismazars.us/forsights/2024/08/be-proactive-limit-asset-misappropriation-schemes (proactive surveillance and data monitoring detected frauds at a median of six months; the typical scheme lasted 12 months).
Mandiant M-Trends 2024, as reported in Connor Jones, “Mandiant: Orgs are detecting cybercriminals faster than ever,” The Register, April 23, 2024, https://www.theregister.com/2024/04/23/mandiant_orgs_are_detecting_cybercrims/ (global median dwell time of ten days; non-ransomware intrusions found by internal teams in nine days against 20 days when an outside party notified the victim).
This essay is a combination of personal notes, discussions I had at the MAIA/AIST workshop, and slides from a talk I gave with JHU ASBS and BlueDot Impact.
Premise
A common theme I noticed, particularly among viewpoints with short timelines, is the idea that the introduction of ASI is inherently and instantaneously the loss condition. This stems, I think, from Yudkowsky-ian philosophy that ASI = unavoidable doom.[1] I think this defeatist attitude is counterproductive and detracts from real, important, effective avenues of action.
A superintelligence is bounded through interaction with the physical world; there are fundamental limitations on feedback loops which are crucial to enact any kind of influence.[2] ASI on a laptop in a concrete room in 1850 cannot escape with any high probability, and even if it did, it would take a very long time for it to do anything – conceivably much longer than the physical lifetime of the laptop. In well-defined worlds like video games, it is not hard to imagine scenarios where you are mathematically guaranteed to lose even with perfect skill and play – for example, if the ball in pong is much much faster than your paddle. There is an analogy here to galactic algorithms in computer science[3]: just because an ASI has an algorithm that scales better asymptotically doesn't mean the constant factors (e.g. time needed to conduct experiments; time needed to manufacture a quantity of robots) allow it to win within a finite time horizon.
In the present world, incidents like HuggingFace[4] can be largely attributed to 1. Ease: cybersecurity and the internet is an extremely accessible attack surface for AI, 2. Lack of foresight: most of the internet was not designed in anticipation of highly intelligent adversaries and widespread integration into society (cybersecurity is almost always reactionary), and 3. Ubiquity: the internet is a widespread and well-connected infrastructure tied deeply to society. As I, and many others have previously wrote, it is extremely difficult for ASI, if incarnated right now, to rapidly destroy all of humanity, or even do more mundane things like synthesize a virus or take over a factory. It would almost necessarily have to use humans, through superpersuasion or blackmail, limiting the effective set of “hands” to a few experts which it must coordinate… not impossible, but inherently gated by the limitations of humans as tools: we suck at following instructions, general competency, etc. Physical limits also significantly bound the influence of ASI in propagation time and physical limits (e.g. Control Theory); no matter what, noise and experiment mean you cannot have iteration speeds past a certain limit nor achieve results under a certain statistical confidence. All of this is further compounded by the need to avoid detection prior to achieving independence or dominance over humanity. This is why “gradual disempowerment” scenarios are generally more favored.[5]
However, as I have previously argued, this will change.[6] It’s only through luck that the robotics and biotech industries have not achieved the same widespread diffusion that the internet has (at this point in time), and all signs point towards this changing rapidly. This presents us a simple theory of change: designing these new systems with foresight, similarly to how we treat other high-risk technologies like nuclear weapons. Through proactively “hardening” these systems – these “hands” for AI – can we provide an effectively externally-imposed slowdown (allowing other actions) or even shut down ASI?
Adjusting time-to-doom through slowing the trajectory of ASI independence
We have three basic actions to combat against ASI: slow it, see it, stop it. Suppose a misaligned ASI has escaped: we define the win condition as a physical base (power, chips, robots, etc.) entirely independent of humanity. Here, we have four steps:
where the first three steps can be convolved into one number: the number of times we expect to catch an attempt and act on it. Then,
where T is months to build, λ is the monthly chance of being noticed, a is the chance a notice is acted on, and d is the chance a response terminates the ASI. That gives three dials: slow it (T), see it (λ), and stop it (a and d). Then, we can make some starting estimates:
Input
Median
Sampled range (10th to 90th)
Basis
Months to build (T)
45
7 to 172
Mostly follows from the next three rows
↳ Robot doubling time, months
12
4 to 36
Forethought’s estimate of about a year[8]
↳ Robots needed
320,000
10,000 to 10 million
Guess
↳ Robots at the start
1,700
100 to 30,000
Guess
Monthly chance of notice (λ)
2%
0.5% to 8%
Two covert programmes hidden 5 and 15 years[9] imply 0.5% to 2%
Notice acted on (a)
42%
15% to 75%
Guess; no reference class
Response ends it (d)
22%
5% to 60%
Guess; large botnets came back after takedowns[10]
Unmodelled route exists
15%
Fixed; tested from 0% to 100%
Guess
Catches per attempt (X)
0.4
0.03 to 3.9
Result of the rows above
but also sample from a wide range for all parameters. Simulating many different worlds with different combinations of parameters:
Clearly, it doesn’t seem terribly unlikely (black line = even odds) that we can indeed detect and catch nascent ASI even in the absence of interventions (orange dots). Nevertheless, the odds are not great. To model the effect of hardening (green dots), we can adjust the parameters:
Dial
Factor (median; range)
Median world
Basis
Evidence
T
×1.8 per stage; 1.1 to 3
45 → 95 months
Nuclear projects after 1970 took 2.4× longer (17 years against 7)[11]
Loose
T
÷6.3; 2 to 20
Included in the row above
None; today’s nearest control stops about 5% of orders
Guess
λ
×3.5; 1.5 to 8
2% → 7%
Monitoring speeds detection about 2× (fraud: 6 months against 12[12]; intrusions: 9 days against 20[13])
Anchored
a
×4; 2 to 8
42% → 74%
None; history shows notices are often ignored[14]
Guess
d
×2.3 twice; 1.3 to 4 each
22% → 59%
None; kept modest because strong takedowns have failed
Guess
X
0.4 → 5.5
Follows from the rows above
Some interesting results follow. Win/loss probability in worlds is largely a two-state readout, with the majority of probability mass either on 0% or 100%; hardening therefore flips many worlds from certain doom to certain success.
The median time-to-notice required for even odds also increases to historically plausible levels:
Finally, we can decompose the expected effect of interventions at each level:
and, as expected, speedups in thinking speed quickly saturate and do not increase ASI-win probability:
(1x ASI = 100x human)
Lastly, I look at some pessimistic and optimistic assumptions:
Input
Fast
Central
Bottleneck
A route the model misses exists
60%
15%
3%
Robot doubling time, months
0.5 to 6
4 to 36
9 to 60
How far physical steps compress
5 to 300×
1.2 to 10×
1.05 to 3×
Monthly chance of notice
A quarter of central
About 2%
Double central
A response ends it
0.5% to 15%
5% to 60%
15% to 80%
Hardening factors
Weak
As in your table
Stronger
Median months to build
7
45
109
Result, without → with hardening
99% → 94%
72% → 26%
39% → 5%
Obviously, if most probability is dominated by definitionally unforeseeable routes, hardening does not provide much gain. Outside of this, however, hardening seems to robustly increase our chances of winning. Therefore, it seems like hardening may represent a third option to policy and technical alignment work amenable to great interdisciplinary efforts.
Thanks to Elaine Ly and Peyton Li for comments.
Eliezer Yudkowsky, “AGI Ruin: A List of Lethalities,” LessWrong, June 5, 2022, https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities (point 2: losing to a high-powered cognitive system “looks at least as deadly as ‘everybody on the face of the Earth suddenly falls over dead within the same second’”).
On physical limits bounding a superintelligence: Jeffrey Heninger and Aysja Johnson, “Superintelligence Is Not Omniscience,” LessWrong, April 7, 2023, https://www.lesswrong.com/posts/qpgkttrxkvGrH9BRr/superintelligence-is-not-omniscience ; Shahin Kaveh, “Why AI Won’t Kill Us by 2030, and Probably Ever,” LessWrong, November 14, 2026, https://www.lesswrong.com/posts/QgSvF5yDJjkn3NZTY/why-ai-won-t-kill-us-by-2030-and-probably-ever ; MarkelKori, “Where ASI Could Stumble,” LessWrong, November 15, 2026, https://www.lesswrong.com/posts/Jj2aLyZPYDv8ZsnCj/where-asi-could-stumble-1 . For the counterargument, see Algon, “Chaos Alone is No Bar to Superintelligence,” LessWrong, October 6, 2025, https://www.lesswrong.com/posts/4AxYNx5HxjKKyfJyG/chaos-alone-is-no-bar-to-superintelligence .
Richard J. Lipton, “Galactic Algorithms,” Gödel’s Lost Letter and P=NP, October 23, 2010, https://rjlipton.com/2010/10/23/galactic-algorithms/ (an algorithm “wonderful in its asymptotic behavior” but never used in practice).
OpenAI, “OpenAI – Hugging Face Incident Technical Report,” August 26, 2026, https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf .
Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud, “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development,” arXiv:2501.16946, January 28, 2025, https://arxiv.org/abs/2501.16946 .
Jonathan Ouyang, “Let’s be realistic about AI-enabled bioweapons,” The Johns Hopkins News-Letter, October 5, 2026, https://www.jhunewsletter.com/article/2026/10/lets-be-realistic-about-ai-enabled-bioweapons .
On catching an AI attempt and acting on it: Ryan Greenblatt and Buck Shlegeris, “Catching AIs red-handed,” LessWrong, January 5, 2024, https://www.lesswrong.com/posts/i2nmBfCXnadeGmhzW/catching-ais-red-handed ; Buck Shlegeris, “Handling schemers if shutdown is not an option,” LessWrong, April 18, 2025, https://www.lesswrong.com/posts/XxjScx4niRLWTfuD5/handling-schemers-if-shutdown-is-not-an-option .
Tom Davidson and Rose Hadshar, “The Industrial Explosion,” Forethought, May 21, 2025, https://www.forethought.org/research/the-industrial-explosion (estimates “an initial robot doubling time of roughly 1 year or less”; its summary is more conservative, “on the order of a few years”). Also posted to LessWrong: https://www.lesswrong.com/posts/Na2CBmNY7otypEmto/the-industrial-explosion .
Biopreparat was created on April 24, 1974 (Glavmikrobioprom Order 131 DSP), per “Biopreparat,” Wikipedia, citing Anthony Rimmington, The Soviet Union’s Invisible Weapons of Mass Destruction (Springer, 2021), https://en.wikipedia.org/wiki/Biopreparat , and was exposed when Vladimir Pasechnik defected to Britain in late October 1989, per David E. Hoffman, The Dead Hand (Doubleday, 2009), excerpted by the National Security Archive, May 6, 2010, https://nsarchive2.gwu.edu/NSAEBB/NSAEBB315/ . Aum Shinrikyo began producing botulinum toxin and phosgene in 1990 and carried out the Tokyo subway sarin attack on March 20, 1995, per “Tokyo subway sarin attack,” Wikipedia, https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack ; see also Richard Danzig et al., Aum Shinrikyo: Insights Into How Terrorists Develop Biological and Chemical Weapons, 2nd ed. (CNAS, 2012), https://www.cnas.org/publications/reports/aum-shinrikyo-second-edition-english .
For example, Emotet, taken down in an international operation coordinated by Europol and Eurojust at the start of 2021, was rebuilding its botnet by November 2021: Lawrence Abrams, “Emotet malware is back and rebuilding its botnet via TrickBot,” BleepingComputer, November 15, 2021, https://www.bleepingcomputer.com/news/security/the-emotet-malware-is-back-and-rebuilding-the-botnet-via-trickbot/ .
Jacques E. C. Hymans, “The Threat of Nuclear Proliferation: Perception and Reality,” Ethics & International Affairs 27, no. 3 (2013): 281–298, https://doi.org/10.1017/S089267941300021X (all dedicated projects launched before 1970 succeeded, averaging about seven years to a first test; only three of the ten launched since 1970 succeeded, needing “an average of about seventeen years”). See also Hymans, Achieving Nuclear Ambitions (Cambridge University Press, 2012).
Association of Certified Fraud Examiners, Occupational Fraud 2024: A Report to the Nations, as summarized in Alexander Gershner, “Be Proactive & Limit Asset Misappropriation Schemes,” Forvis Mazars, August 30, 2024, https://www.forvismazars.us/forsights/2024/08/be-proactive-limit-asset-misappropriation-schemes (proactive surveillance and data monitoring detected frauds at a median of six months; the typical scheme lasted 12 months).
Mandiant M-Trends 2024, as reported in Connor Jones, “Mandiant: Orgs are detecting cybercriminals faster than ever,” The Register, April 23, 2024, https://www.theregister.com/2024/04/23/mandiant_orgs_are_detecting_cybercrims/ (global median dwell time of ten days; non-ransomware intrusions found by internal teams in nine days against 20 days when an outside party notified the victim).
On warning signs failing to trigger action: Eliezer Yudkowsky, “There’s No Fire Alarm for Artificial General Intelligence,” LessWrong, October 13, 2017, https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence .