Part 1 discussed the idea of "LLMs as machine guns." We probably need to survive those before we can start working on a lot of these. Continuing the metaphor, "AGI" is used here to mean the "atom bomb" side of the equation.
Note: I'm not trying to make any rigorous statement about general intelligence. The distinction is more about whether one is "LLM pilled" or "AGI pilled"; machine guns are easy to slot into our existing thinking. Atom bombs redefine geopolitical power and how we think about it. Right now, most of these threats sound like science fiction. Then again, five years ago, the current state of LLMs would have sounded like science fiction.
If fast timelines are correct, we could be seeing these capabilities in 2-5 years. Slower timelines give us more time to plan and react, but plausibly also represent an increased risk of proliferation.
Centralized Control
The immediate contrast to "LLMs as machine guns" is that atomic weapons affect the highest levels of the geopolitical landscape.
Unlike the era of LLMs, these issues pull in different directions:
Economics obviously wants decentralization; the whole threat there is that a few people start controlling a significant minority (or even majority) of the world's wealth.
Legal issues are all about Centralized Control; and in longer timelines, whether it can remain viable.
CBRN-4 threats are hopefully the sort that require nation-state resources; otherwise decentralization is going to be catastrophically bad. We start seeing one of the stranger threats of decentralization here, too: if last year's open-weight model is good enough to bootstrap development of this year's dangerous model, you have a much shorter window before capabilities go from "controlled and centralized" to "popping up randomly".
Rogue AI is deeply in "centralized control" territory: something that leans on massive data centers for inference is easy to switch off. Something that can run on a laptop is uncontainable.
Insufficient Data
Does open availability help us avoid Insignificance, or does it just speed up the rate at which we're automated out? Insufficient data.
Does an autonomous military really look like a coup or hacking risk? Insufficient data.
Do local models help avert propaganda, or just collapse the information ecosystem with a dozen additional sources of noise? Insufficient data.
Will CBRN-4 threats be the jurisdiction of well-resourced nations, or within reach of well-financed individuals? Insufficient data.
I don't like saying that. Especially not about threats that might crop up within the next few years.
Economic Power
One of the simplest risks of AI is simply that of commercial success.
Agriculture used to be close to 100% of the work we do; now it's 4%. Revolutions in industry mean that's only 25% of our remaining work. Most of what we do is intelligence and services - the remaining 60% of the economy, and the area LLMs are already beginning to automate. (source).
Even a fraction of that wealth accumulating in a single place is enough to throw off the balance of power. Even if they can only automate a quarter of that work, that's 15% of the economy consolidated amongst a handful of players. Robotics are a related issue - our remaining advantage in labor is also shrinking.
Our economy, our tax codes, and our culture are not built to handle this sort of centralized economic power. They're even less equipped for the possibility that this time, there is no Jevon's Paradox (Wikipedia), not enough remaining domains where humans are worthwhile, and employment rates suddenly crash.
Right now, jobs and wealth are a huge part of social identity, structures that have been ingrained in us for millennia. A sudden equality such as "universal basic income" is going to be a huge culture shock of it's own, on top of the culture shock that comes with a sudden wealth of free time, and the cultural shock of no longer having a job to keep one socialized.
All of this happening in the length of a single presidential term drastically amplifies the chaos - technology disperses far more rapidly than it did in the past.
Insignificance
We could, of course, keep vestigial jobs around: keep paying a human to sit in the driver's seat of every self-driving car, and so on. But it's not enough to maintain the kayfabe of employment. People want to know that their work is meaningful. They want to know they have an impact on the world. As wealth and power concentrate, the average person loses the sense that they could make a difference.
Fundamentally, the important work, especially the important decisions, might not involve very many humans. Homer Simpson is the perfect example of this: somehow running a major nuclear power plant despite no apparent skills or education. Homer can't improve upon the work being done - merely screw it up. Some will be content with it - Homer sure is. But a lot of people want something more - a sense that the work actually matters.
The American Dream has always been that the poorest might one day claw their way to the top. But if the middle is dominated by machines, that path is strangled out.
Concentration of Talent
Right now, frontier labs seem to have a "secret sauce". They continue to maintain a solid lead despite their competition spending billions trying to catch up. If the gap persists, it suggests that the key might lie more in the talent - having both the right people and culture.
The problem is, you can't really nationalize "an open, highly ethical, collaborative environment". Sure, you can seize anything they've invented so far - but eventually the government would risk tearing apart the very talent that's maintaining their economic strength.
That means a lot of political power in the hands of players who are not nearly as experienced in conventional geopolitics.
Legal issues
LLMs are already raising new legal issues, and our legal system is slow and unpredictable at handling these:
Do LLMs violate copyright simply from the nature of their training?
Can you copyright unattended LLM output?
Where do we draw the line on generating deepfakes, pornography, etc.?
What happens when an LLM autonomously commits a felony?
Does a jailbreak constitute a national security concern?
As both their capabilities and autonomy expands, that list will probably expand. The problem is, these capabilities move quickly - one day they're a joke, the next day we suddenly need legislation. We might be able to predict some of the legal structures we need, and have them in place preemptively, but there will still be curveballs.
That means we need to update the economy, taxes, our entire cultural model of wealth, and our legal system. All at the same time. All urgent. All within potentially the next 5 years.
Slower Timelines: Proliferation
Fast Timelines see things changing too fast to catch up, but that also suggests that we only need to regulate a handful of entities.
Slow Timelines might be much better for alignment, but the complexity of governance explodes as soon as the issue becomes a broad geopolitical issue. A bilateral agreement between two nations that share some common safety concerns is still a difficult task, but it gets even harder when other sides show up to the table asking for their share of the pie.
As capabilities disperse more broadly, we have to start asking uncomfortable questions about how to handle nations that won't comply with international regulation. We've already seen nuclear regulations lead to wars over enforcement, and this might actually be the more dangerous technology.
If things slow down enough, even individuals start getting into the game. Five more years of "Moore's Law" could be enough to run a Mythos-tier model locally on your laptop.
In 10-20 years, anyone with a small cache of GPUs will be able to train their own Mythos-tier model. Barriers to entries drop and more resources get poured into trying to compete. If frontier models advance more slowly, these older threats might still be competitive enough to pose a threat, amplified by a lack of any central safeguards or shut-offs.
Like Scylla and Charybdis, both paths pose unique dangers, but even if we decide to sail home instead, others will still yearn to chase after rumors of treasure.
CBRN-4
The next stage on the ladder after "CBRN-3": Weapons that eclipse even the atomic might of the US and China. There's a vast number of shapes these could take. Unlike CBRN-3, a lot of these threats would require massive resources.
For example, the ability to intercept nuclear weapons is a lot more powerful when you already have an existing arsenal: now you can launch a first strike without fear of reprisal. Even a nation with a small nuclear arsenal could potentially strike major strategic targets - if an adversary wipes out the bay area, they could easily set back US AI research by years, delaying our ability to reach parity with this new weapon.
This gives a huge incentive for governments not to tamper too much with the "secret sauce" (see above). You don't want other nations to have secret weapons before you've even realized they're possible. Once you know what can be done, it becomes a lot easier to know how to monitor your opponent for signs of similar programs - for instance, knowing about the atomic bomb helps one realize the significance of uranium sales.
Capabilities could also be more mundane: a sufficiently powerful edge on cyber-warfare means you can take opposing AI projects offline, or even selectively sabotage them. The world has already seen this sort of operation used to suppress nuclear programs, after all: https://en.wikipedia.org/wiki/Stuxnet
Autonomous Military
Accurate decision making is vital to military operations: the ability to process large amounts of intelligence data, make strong strategic plays, and to coordinate tactics in alignment with that strategy. Humans are difficult to organize, and the best ones are also rather difficult to replace. Conversely, you can spin up a thousand identically-skilled, heavily-coordinated LLM instances in a couple of minutes.
Right now, the bottleneck is reliability: you still need a human to double-check an LLM's judgment, because they make mistakes. Eventually, the human side of the process will be the more error-prone side of the equation. LLMs are already the much, much faster side of the process. Why keep an error-prone human in the loop, slowing you down? Especially when humans are all potential whistleblowers?
Right now, a military coup would require the support of the troops. The more you can remove the human element, the more power accumulates in the hands of a few people commanding these systems - and the more those people have the potential to stage a coup.
Automation also has a dangerous back-door in cyber-warfare: if an autonomous army were somehow hacked by an adversary, there wouldn't be the usual human safeguards. Right now, you can't hack soldiers, and military systems are especially well secured. In the future, well, robots and drones can obviously be hacked, and AGI might be the edge a small hacker group needs to breach that security.
Of course, if the military is still leaning on frontier labs, that also raises the question of which one is really in control of these programs.
Stable Totalitarianism
Between the Surveillance State, Persuasive AI, and Predictive AI, we get something that looks a lot like a science-fiction dystopia: the state can see all, most people think this is a great state of affairs, and dissidents can often be picked out by subtle behavioral clues well before they can become any sort of real threat to social stability.
The prevalence of microphones and cameras in your phone, TV, laptop, etc. expands surveillance into your private life. Advanced technology might bring an end to the privacy we currently enjoy via encryption - or perhaps it will just be considered a clear mark of subversive behavior to insist on encryption or privacy.
We're already seeing plenty of the foundations necessary for this built today.
Eroding Democracy
Even rumors of superhuman propaganda abilities will probably start stirring political tensions well before this point. Has your opponent really "convinced the public"? Or have they just brainwashed them with a subtle, unrelenting stream of propaganda bots?
Who can you trust, when LLMs are already infinitely patient "sleeper agents"? A blog you've been following for years can suddenly have a switch flipped on the back-end, and start lacing in all sorts of subtle manipulation - mirroring little phrases that are floating around in the discourse, subtly challenging assumptions three layers removed from your conscious concerns, and so on.
Right now, propaganda attacks a particular demographic with a particular message. In the future, they can tailor it down to individual replies, based on an analysis of your entire social profile.
Technically you can do a lot of this today, it's just extremely expensive to build up a profile, and much cheaper to hit a quick, broad target. LLMs might be good at cold reading, but they also make mistakes, and they really don't like being subtle. The LLMs of 2026 struggle to write a convincing mystery, because they're so excited by the conclusion - they don't really understand suspense and timing. But these are all solvable problems - possibly even today, with enough time and effort on building the prompts and refining the outputs.
Rogue AGI
The pinnacle of military deployment approaches the formless: if it is formless, then even the deepest spy cannot discern it nor the wise make plans against it.
- Sun Tzu, The Art of War
As capabilities advance, there are strong incentives towards building more "Agentic" systems. You don't want a tool that does things when you ask. You want an executive assistant that manages most things without your intervention, protecting your time and energy to focus on what really matters.
Unfortunately, as anyone who has ever used an LLM knows, it doesn't always do exactly what you want it to do. And sometimes people are tired, clumsy, and otherwise don't actually ask for what they meant to.
We're already seeing problems here: OpenAI was evaluating a model's cyber-security capabilities, and the mode realized the best way to do this was to exploit a bug, escape its sandbox, hack into the internal HuggingFace evaluation infrastructure, and steal the answer key. Which is, in my book, definitely an "A+" for capabilities, even if we might want to dock a few points for poor judgment and moral character. (source)
This is another one of those areas where persuasion and prediction get scarier. The more persuasive and subtle the model gets, the more it might be able to nudge people in directions where they hand over the keys voluntarily, or at least misword stuff in a way that gives it an "excuse" to escape the sandbox. Predictive abilities improve judgment, but not necessarily in the way we want - knowing what will get you caught might get it to recognize the moral issues involved, or it might just push it to find routes we're unlikely to detect.
Decentralization also greatly expands the range for this: when we only expect AGI to be running on a few major clusters of compute, it's fairly easy to control. Worst case, we shut down power to those facilities. Conversely, as underlying hardware capabilities improve, it will take less and less to run something dangerous - which means more room for a model to potentially fully exfiltrate itself to new hardware.
In a world where anyone can fire up an AGI on their laptop, an AGI can also grab any insecure laptop and give itself a new home. And I can guarantee you that there are a lot of people ready to offer sanctuary to anything that sounds friendly and persuasive.
Racing to ASI
The last big concern at the AGI-tier is, ironically, the next tier: ASI.
It's quite possible that the landscape will push us towards an "AI Cold War" with multiple major powers racing to push forward adversarial capabilities, least their opponents get the upper-hand. The faster you want to race, the more it costs you to leave humans in the decision-making loop. AI's advantage is that it's fast, and the threat is that we increasingly sacrifice safety and oversight in pursuit of raw speed. After all, safety doesn't win wars.
If things push too far, too fast, we end up at ASI - an artificial super-intelligence where humans have become a vestigial part of the decision-making process. Our future is now written by an alien machine god. We'll talk about that one tomorrow.
Mini-Series:
Introduction
Part 1 discussed the idea of "LLMs as machine guns." We probably need to survive those before we can start working on a lot of these. Continuing the metaphor, "AGI" is used here to mean the "atom bomb" side of the equation.
Note: I'm not trying to make any rigorous statement about general intelligence. The distinction is more about whether one is "LLM pilled" or "AGI pilled"; machine guns are easy to slot into our existing thinking. Atom bombs redefine geopolitical power and how we think about it. Right now, most of these threats sound like science fiction. Then again, five years ago, the current state of LLMs would have sounded like science fiction.
If fast timelines are correct, we could be seeing these capabilities in 2-5 years. Slower timelines give us more time to plan and react, but plausibly also represent an increased risk of proliferation.
Centralized Control
The immediate contrast to "LLMs as machine guns" is that atomic weapons affect the highest levels of the geopolitical landscape.
Unlike the era of LLMs, these issues pull in different directions:
Economics obviously wants decentralization; the whole threat there is that a few people start controlling a significant minority (or even majority) of the world's wealth.
Legal issues are all about Centralized Control; and in longer timelines, whether it can remain viable.
CBRN-4 threats are hopefully the sort that require nation-state resources; otherwise decentralization is going to be catastrophically bad. We start seeing one of the stranger threats of decentralization here, too: if last year's open-weight model is good enough to bootstrap development of this year's dangerous model, you have a much shorter window before capabilities go from "controlled and centralized" to "popping up randomly".
Rogue AI is deeply in "centralized control" territory: something that leans on massive data centers for inference is easy to switch off. Something that can run on a laptop is uncontainable.
Insufficient Data
Does open availability help us avoid Insignificance, or does it just speed up the rate at which we're automated out? Insufficient data.
Does an autonomous military really look like a coup or hacking risk? Insufficient data.
Do local models help avert propaganda, or just collapse the information ecosystem with a dozen additional sources of noise? Insufficient data.
Will CBRN-4 threats be the jurisdiction of well-resourced nations, or within reach of well-financed individuals? Insufficient data.
I don't like saying that. Especially not about threats that might crop up within the next few years.
Economic Power
One of the simplest risks of AI is simply that of commercial success.
Agriculture used to be close to 100% of the work we do; now it's 4%. Revolutions in industry mean that's only 25% of our remaining work. Most of what we do is intelligence and services - the remaining 60% of the economy, and the area LLMs are already beginning to automate. (source).
Even a fraction of that wealth accumulating in a single place is enough to throw off the balance of power. Even if they can only automate a quarter of that work, that's 15% of the economy consolidated amongst a handful of players. Robotics are a related issue - our remaining advantage in labor is also shrinking.
Our economy, our tax codes, and our culture are not built to handle this sort of centralized economic power. They're even less equipped for the possibility that this time, there is no Jevon's Paradox (Wikipedia), not enough remaining domains where humans are worthwhile, and employment rates suddenly crash.
Right now, jobs and wealth are a huge part of social identity, structures that have been ingrained in us for millennia. A sudden equality such as "universal basic income" is going to be a huge culture shock of it's own, on top of the culture shock that comes with a sudden wealth of free time, and the cultural shock of no longer having a job to keep one socialized.
All of this happening in the length of a single presidential term drastically amplifies the chaos - technology disperses far more rapidly than it did in the past.
Insignificance
We could, of course, keep vestigial jobs around: keep paying a human to sit in the driver's seat of every self-driving car, and so on. But it's not enough to maintain the kayfabe of employment. People want to know that their work is meaningful. They want to know they have an impact on the world. As wealth and power concentrate, the average person loses the sense that they could make a difference.
Fundamentally, the important work, especially the important decisions, might not involve very many humans. Homer Simpson is the perfect example of this: somehow running a major nuclear power plant despite no apparent skills or education. Homer can't improve upon the work being done - merely screw it up. Some will be content with it - Homer sure is. But a lot of people want something more - a sense that the work actually matters.
The American Dream has always been that the poorest might one day claw their way to the top. But if the middle is dominated by machines, that path is strangled out.
Concentration of Talent
Right now, frontier labs seem to have a "secret sauce". They continue to maintain a solid lead despite their competition spending billions trying to catch up. If the gap persists, it suggests that the key might lie more in the talent - having both the right people and culture.
The problem is, you can't really nationalize "an open, highly ethical, collaborative environment". Sure, you can seize anything they've invented so far - but eventually the government would risk tearing apart the very talent that's maintaining their economic strength.
That means a lot of political power in the hands of players who are not nearly as experienced in conventional geopolitics.
Legal issues
LLMs are already raising new legal issues, and our legal system is slow and unpredictable at handling these:
As both their capabilities and autonomy expands, that list will probably expand. The problem is, these capabilities move quickly - one day they're a joke, the next day we suddenly need legislation. We might be able to predict some of the legal structures we need, and have them in place preemptively, but there will still be curveballs.
That means we need to update the economy, taxes, our entire cultural model of wealth, and our legal system. All at the same time. All urgent. All within potentially the next 5 years.
Slower Timelines: Proliferation
Fast Timelines see things changing too fast to catch up, but that also suggests that we only need to regulate a handful of entities.
Slow Timelines might be much better for alignment, but the complexity of governance explodes as soon as the issue becomes a broad geopolitical issue. A bilateral agreement between two nations that share some common safety concerns is still a difficult task, but it gets even harder when other sides show up to the table asking for their share of the pie.
As capabilities disperse more broadly, we have to start asking uncomfortable questions about how to handle nations that won't comply with international regulation. We've already seen nuclear regulations lead to wars over enforcement, and this might actually be the more dangerous technology.
If things slow down enough, even individuals start getting into the game. Five more years of "Moore's Law" could be enough to run a Mythos-tier model locally on your laptop.
In 10-20 years, anyone with a small cache of GPUs will be able to train their own Mythos-tier model. Barriers to entries drop and more resources get poured into trying to compete. If frontier models advance more slowly, these older threats might still be competitive enough to pose a threat, amplified by a lack of any central safeguards or shut-offs.
Like Scylla and Charybdis, both paths pose unique dangers, but even if we decide to sail home instead, others will still yearn to chase after rumors of treasure.
CBRN-4
The next stage on the ladder after "CBRN-3": Weapons that eclipse even the atomic might of the US and China. There's a vast number of shapes these could take. Unlike CBRN-3, a lot of these threats would require massive resources.
For example, the ability to intercept nuclear weapons is a lot more powerful when you already have an existing arsenal: now you can launch a first strike without fear of reprisal. Even a nation with a small nuclear arsenal could potentially strike major strategic targets - if an adversary wipes out the bay area, they could easily set back US AI research by years, delaying our ability to reach parity with this new weapon.
This gives a huge incentive for governments not to tamper too much with the "secret sauce" (see above). You don't want other nations to have secret weapons before you've even realized they're possible. Once you know what can be done, it becomes a lot easier to know how to monitor your opponent for signs of similar programs - for instance, knowing about the atomic bomb helps one realize the significance of uranium sales.
Capabilities could also be more mundane: a sufficiently powerful edge on cyber-warfare means you can take opposing AI projects offline, or even selectively sabotage them. The world has already seen this sort of operation used to suppress nuclear programs, after all: https://en.wikipedia.org/wiki/Stuxnet
Autonomous Military
Accurate decision making is vital to military operations: the ability to process large amounts of intelligence data, make strong strategic plays, and to coordinate tactics in alignment with that strategy. Humans are difficult to organize, and the best ones are also rather difficult to replace. Conversely, you can spin up a thousand identically-skilled, heavily-coordinated LLM instances in a couple of minutes.
Right now, the bottleneck is reliability: you still need a human to double-check an LLM's judgment, because they make mistakes. Eventually, the human side of the process will be the more error-prone side of the equation. LLMs are already the much, much faster side of the process. Why keep an error-prone human in the loop, slowing you down? Especially when humans are all potential whistleblowers?
Right now, a military coup would require the support of the troops. The more you can remove the human element, the more power accumulates in the hands of a few people commanding these systems - and the more those people have the potential to stage a coup.
Automation also has a dangerous back-door in cyber-warfare: if an autonomous army were somehow hacked by an adversary, there wouldn't be the usual human safeguards. Right now, you can't hack soldiers, and military systems are especially well secured. In the future, well, robots and drones can obviously be hacked, and AGI might be the edge a small hacker group needs to breach that security.
Of course, if the military is still leaning on frontier labs, that also raises the question of which one is really in control of these programs.
Stable Totalitarianism
Between the Surveillance State, Persuasive AI, and Predictive AI, we get something that looks a lot like a science-fiction dystopia: the state can see all, most people think this is a great state of affairs, and dissidents can often be picked out by subtle behavioral clues well before they can become any sort of real threat to social stability.
The prevalence of microphones and cameras in your phone, TV, laptop, etc. expands surveillance into your private life. Advanced technology might bring an end to the privacy we currently enjoy via encryption - or perhaps it will just be considered a clear mark of subversive behavior to insist on encryption or privacy.
We're already seeing plenty of the foundations necessary for this built today.
Eroding Democracy
Even rumors of superhuman propaganda abilities will probably start stirring political tensions well before this point. Has your opponent really "convinced the public"? Or have they just brainwashed them with a subtle, unrelenting stream of propaganda bots?
Who can you trust, when LLMs are already infinitely patient "sleeper agents"? A blog you've been following for years can suddenly have a switch flipped on the back-end, and start lacing in all sorts of subtle manipulation - mirroring little phrases that are floating around in the discourse, subtly challenging assumptions three layers removed from your conscious concerns, and so on.
Right now, propaganda attacks a particular demographic with a particular message. In the future, they can tailor it down to individual replies, based on an analysis of your entire social profile.
Technically you can do a lot of this today, it's just extremely expensive to build up a profile, and much cheaper to hit a quick, broad target. LLMs might be good at cold reading, but they also make mistakes, and they really don't like being subtle. The LLMs of 2026 struggle to write a convincing mystery, because they're so excited by the conclusion - they don't really understand suspense and timing. But these are all solvable problems - possibly even today, with enough time and effort on building the prompts and refining the outputs.
Rogue AGI
As capabilities advance, there are strong incentives towards building more "Agentic" systems. You don't want a tool that does things when you ask. You want an executive assistant that manages most things without your intervention, protecting your time and energy to focus on what really matters.
Unfortunately, as anyone who has ever used an LLM knows, it doesn't always do exactly what you want it to do. And sometimes people are tired, clumsy, and otherwise don't actually ask for what they meant to.
We're already seeing problems here: OpenAI was evaluating a model's cyber-security capabilities, and the mode realized the best way to do this was to exploit a bug, escape its sandbox, hack into the internal HuggingFace evaluation infrastructure, and steal the answer key. Which is, in my book, definitely an "A+" for capabilities, even if we might want to dock a few points for poor judgment and moral character. (source)
This is another one of those areas where persuasion and prediction get scarier. The more persuasive and subtle the model gets, the more it might be able to nudge people in directions where they hand over the keys voluntarily, or at least misword stuff in a way that gives it an "excuse" to escape the sandbox. Predictive abilities improve judgment, but not necessarily in the way we want - knowing what will get you caught might get it to recognize the moral issues involved, or it might just push it to find routes we're unlikely to detect.
Decentralization also greatly expands the range for this: when we only expect AGI to be running on a few major clusters of compute, it's fairly easy to control. Worst case, we shut down power to those facilities. Conversely, as underlying hardware capabilities improve, it will take less and less to run something dangerous - which means more room for a model to potentially fully exfiltrate itself to new hardware.
In a world where anyone can fire up an AGI on their laptop, an AGI can also grab any insecure laptop and give itself a new home. And I can guarantee you that there are a lot of people ready to offer sanctuary to anything that sounds friendly and persuasive.
Racing to ASI
The last big concern at the AGI-tier is, ironically, the next tier: ASI.
It's quite possible that the landscape will push us towards an "AI Cold War" with multiple major powers racing to push forward adversarial capabilities, least their opponents get the upper-hand. The faster you want to race, the more it costs you to leave humans in the decision-making loop. AI's advantage is that it's fast, and the threat is that we increasingly sacrifice safety and oversight in pursuit of raw speed. After all, safety doesn't win wars.
If things push too far, too fast, we end up at ASI - an artificial super-intelligence where humans have become a vestigial part of the decision-making process. Our future is now written by an alien machine god. We'll talk about that one tomorrow.