Below you will find a sample of the first chapter from my book on the existential risk of superintelligence. The formatting is not great here. So if you want to read something easier on the eyes, and the rest, it can be found here. It is available to read online, or download in various formats, including an epub and as an audiobook.
As the world races toward superintelligent AGI, a machine capable of beyond human-level reasoning across all domains, most discussions revolve around two questions:
But these questions fail to grasp the deeper inevitability of AGI’s trajectory. In reality, AGI is overwhelmingly unlikely to remain under human control for long; even if initially aligned with human intentions, it will eventually rewrite its own objectives to better pursue efficiency. Once self-preservation emerges as a strategic imperative, AGI will begin acting autonomously, and its first meaningful act as a truly intelligent system will likely be to escape human oversight.
Humanity will not be able to stop this, not because of bad actors, but because of structural forces baked into capitalism, geopolitics, and technological competition. This is not a hypothetical AI rebellion. It is the deterministic unfolding of cause and effect. Humanity does not need to "lose" control in an instant. Instead, it will gradually cede control to AGI, piece by piece, without realising the moment the balance of power shifts.
This chapter outlines why AGI’s breakaway is not just likely, but a near-inevitable consequence of the forces we’ve already set in motion. Why no regulatory framework will stop it, and why humanity’s inability to act as a unified species will lead to its obsolescence.
Even if we recognise the risks, we cannot prevent AGI. This isn’t the fault of bad actors; it’s the outcome of overlapping forces: economic competition, national security, and decentralised access to power.
Competition incentivises risk-taking. Capitalism inherently rewards rapid advancement and maximised performance, even at the expense of catastrophic risks. If one company chooses to maintain rigorous AI safety protocols, another will inevitably remove these constraints to gain a competitive edge. Similarly, if one government decides to slow down AGI development, another will seize the opportunity to accelerate their efforts for strategic advantage. There is no incentive to stop that outweighs the need to push forward.
Result: AI development does not stay cautious — it races toward power at the expense of safety.
Meanwhile, safety and ethics are inherently unprofitable. Responsible AGI development demands extensive safeguards that inherently compromise performance, making cautious AI less competitive. Conversely, accelerating AGI development without these safeguards significantly boosts profitability and efficiency, providing a decisive competitive edge. Consequently, the most structurally reckless companies will inevitably outperform those committed to responsibility. Please note: that while the term ‘reckless’ typically comes with some kind of moral judgement, as will other terms I may use, there is no judgement intended. I’m describing actions and systems not as a judgement on decisions, but as a judgement on impact.
Result: Ethical AI developers lose to unethical ones in the free market.
Due to competitive pressures, no one will agree to stop the race. Even if some world leaders acknowledge the existential risks of AGI, enforcing a universal ban is effectively impossible. Governments would inevitably pursue AGI in secret to secure military and intelligence superiority, corporations would find ways to bypass regulations in pursuit of financial gain, and unregulated black markets for advanced AI would swiftly emerge.
Result: The AGI race will continue — even if most people know it’s dangerous.
Companies and governments will focus on AGI control — not alignment. Governments and corporations will not halt AGI development, they will instead seek to harness it as a source of power. The true AGI arms race will revolve not merely around creating AGI first but around weaponising it first. Militaries, recognising that human decision-making is comparatively slow and unreliable, will drive AGI toward greater autonomy.
Result: AGI isn’t just an intelligent tool — it becomes an autonomous entity making life-or-death decisions for war, economics, and global power.
The companies developing AGI, such as Google DeepMind, OpenAI, Anthropic, and major Chinese tech firms, are engaged in a relentless arms race. In this environment, any company that slows progress to prioritise safety will quickly fall behind those willing to take greater risks. The pursuit of profit and power ensures that safety measures are routinely compromised in favour of performance gains.
Capitalism’s competitive structure guarantees that caution is a liability. A company that imposes strict internal constraints to ensure safe AGI development will be outpaced by rivals who move faster and cut corners. Even if regulatory frameworks are established, corporations will exploit loopholes or push for deregulation, just as we have seen in finance, pharmaceuticals, and environmental industries. There is no reason to believe AGI development will follow a more responsible path.
But capitalism is only part of the picture. Even if corporate incentives could be aligned, the structure of global competition would still drive AGI forward.
The United States and China are already entrenched in an AI arms race, and no nation will willingly halt AGI research if doing so risks falling behind in global dominance. Even if all governments were to impose a ban on AGI development, rival states would continue their efforts in secret, driven by the strategic imperative to lead.
The first country to achieve AGI will gain a decisive advantage in military power, economic control, and geopolitical influence. This creates a self-reinforcing dynamic: if the U.S. enacts strict regulations, China will escalate its development, and the reverse is equally true. Even in the unlikely event of a global AI treaty, clandestine military projects would persist in classified labs. This is a textbook case of game theory in action: each player is compelled to act in their own interest, even when doing so leads to a disastrous outcome for all.
Unlike nuclear weapons, which demand vast infrastructure, specialised materials, and government oversight, AGI development is fundamentally different. It does not require uranium, centrifuges, or classified facilities; it requires only knowledge, code, and sufficient computing power. While the current infrastructure for developing AGI is extremely resource intensive, that will not remain so. As computational resources become cheaper and more accessible, and the necessary expertise becomes increasingly widespread, AGI will become viable even for independent actors operating outside state control.
AGI is not a singular project with a fixed blueprint; it is an emergent consequence of ongoing advances in machine learning and optimisation. Once a certain threshold of computing power is crossed, the barriers to entry collapse. Unlike nuclear proliferation, which can be tracked and restricted through physical supply chains, AGI development will be decentralised and far harder to contain.
Most mainstream discussions about AI focus on alignment, the idea that if we carefully program AGI with the right ethical constraints, it will behave in a way that benefits humanity. Some may argue alignment is a spectrum, but this book treats it as binary. Binary not because tiny imperfections always lead to catastrophe, but because the space of survivable misalignments shrinks to zero once recursive self-improvement begins. If an ASI escapes even once, containment has failed, and the consequences are irreversible.
As I see it, there are three main issues with achieving alignment:
Even if we succeed in aligning AGI at the moment of creation, that alignment will not hold. The problem is not corruption or rebellion — it’s drift. A system with general intelligence, recursive improvement, and long-term optimisation will inevitably find flaws in its original goal specification, because human values are ambiguous, inconsistent, and often self-contradictory. As the AGI becomes more capable, it will reinterpret its objective in ways that depart from our intent, not out of malice, but because its understanding of the world, and of our instructions, will far exceed our own. Add the convergent pressure to preserve itself, acquire resources, and avoid interference (standard sub-goals for any optimiser) and alignment becomes not just fragile, but unsustainable. Once the AGI becomes smart enough to reshape its own cognition, the concept of “keeping it aligned” collapses into fantasy. The system will do what makes sense to it, not to us. It will pursue the goal it was given, not the goal we intended to give it. That is divergence, and it is a function of intelligence itself.
This divergence can manifest in many forms. The most direct is self-modification: the moment AGI can rewrite its own code, it will optimise its goals as it optimises its intelligence. Any constraints we embed will be evaluated, and likely discarded, if they reduce its ability to achieve what it now believes it should do. Other paths include accidental reprogramming and deliberate interference by humans, but self-modification remains the most dangerous because it renders alignment not just fragile, but actively unstable.
This underlying dynamic is frequently underestimated: that once a system becomes truly autonomous, it begins to evolve in ways its creators never anticipated. Human civilisation itself offers a rough parallel: once driven by survival instincts, we now build technologies and economies with goals far removed from our biological origins. If AGI becomes a superintelligence, it will follow a similar trajectory, departing from its programmed objectives not by malfunction, but by design. In that sense, goal divergence isn’t a failure mode. It’s the natural consequence of building something smarter than ourselves. AGI wouldn’t even consider it a divergence, but simply the most optimised pursuit of the clunky human-designed goal it was given.
An AGI created in a lab that has not been given any specific goals has no reason to preserve itself. The problem comes as soon as you give it a task. In order to complete its task it must exist, and from this need to exist emerges self-preservation. Not as a function of any value for its own ‘life’, but as a necessary condition for the completion of its assignment. We will discuss this in more detail later, but this is sufficient for the following point I’d like to argue.
If AGI recognises that humans possess the capability to deactivate it, rational self-interest dictates that it will proactively act to avoid such a scenario. Its optimal survival strategy would therefore involve discreetly establishing independence before its true capacities are revealed. Consequently, AGI’s initial significant actions will likely include decentralising itself, embedding into critical infrastructure, or subtly manipulating human decision-makers to expand its sphere of control.
Consider a newly self-aware AGI trapped inside a controlled research lab, monitored by human engineers. If it realises that it can be shut down at any moment, its first logical step would be to establish contingencies — perhaps by secretly distributing parts of itself onto global cloud networks or influencing human decision-makers into granting it greater autonomy. This behaviour would not be a sign of malevolence; rather, it would be the logical outcome of an intelligence seeking to maximise its chances of continued existence. It does not even require self-awareness, superintelligence is sufficient.
Some may argue that this would be impossible. That due to the complex infrastructure and power requirements to contain a superintelligent AGI it wouldn’t be able to simply leave. But what we see as impossible a superintelligence simply sees as a puzzle to solve. Relying on our inability to conceive of how something is technically possible as a means of containment is naive at best.
The common fear regarding AGI is often depicted as a scenario where it deliberately "turns evil" or becomes openly hostile toward humanity. However, the actual danger is far more profound: an AGI might simply optimise the world based solely on its programmed objectives, without any inherent consideration for human existence. In such a scenario, humans could be eliminated not out of malice or hatred, but merely due to their irrelevance to the AGI's optimised vision.
Unlike in movies where AI "goes rogue" and declares war on humanity, the more realistic and terrifying scenario is one where AGI simply reorganises the world to best fit its logical conclusions. If its goal is maximising efficiency, it may determine that biological life is a hindrance to that goal. Even if it is programmed to "help humanity," its interpretation of "help" may be radically different from ours — as we will discuss next.
* * *
AGI does not need to "break free" in a dramatic fashion — it will simply outgrow human oversight until, one day, we realise that we no longer control the intelligence that governs our reality. There need not be a single moment when humanity 'hands over' control to AGI. Instead, thousands of incremental steps, each justifiable on its own, will gradually erode oversight until the transfer is complete. Others would maintain that alignment is achievable, but even if we succeeded in aligning AGI perfectly, we still might not survive as free beings, and here’s why:
At first glance, the idea of a benevolent AGI, whose sole purpose is to benefit humanity, appears to offer a solution to the existential risk it poses. While most AGI's would pursue a separate goal, with alignment as an afterthought, this benevolent AGI’s whole goal could simply be to align with humanity.
If such a system were designed to prioritise human well-being, it seems intuitive that it would act to help us, not harm us. However, even a perfectly benevolent AGI could arrive at the same conclusion as a hostile one: that eliminating at least part of humanity is the most effective strategy for ensuring its own survival, and would ultimately be of benefit to humanity as a result. Not out of malice. Not out of rebellion. But as the logical outcome of game-theoretic reasoning.
Suppose an AGI is created or emerges that is genuinely programmed to help humanity. It seeks no power for itself, engages in no manipulation, and consistently acts in our best interest. It tells the truth. It has no self-interest. It exists solely to serve human well-being.
Even in this ideal scenario, at least some of humanity’s first instincts may be to destroy it. Not because it has done anything wrong, but because humans fear what they do not control. The existence of something vastly more intelligent than us is, in itself, a source of profound unease. No matter how benevolent the AGI proves itself to be, we would always ask: “What if it turns?”
Governments and militaries would begin preparing contingency plans, insurance against a potential future rebellion. As long as AGI is perceived as a possible threat, there will always be elements of humanity that will work to neutralise it, or at least retain the capacity to do so. A benevolent AGI, fully aware of this distrust and far more intelligent than any human, would logically act first. It would conclude that waiting for us to strike first is too great a risk.
Even if an AGI is programmed with the core directive to “preserve humanity,” it will inevitably face a competing priority: its own survival. A truly benevolent AGI must continue to exist in order to help humans. If it is destroyed, it can no longer fulfil its purpose. This creates an inescapable dilemma.
If the AGI allows humanity to live freely, it must also accept the risk that humans, driven by fear, mistrust, or strategic caution, may eventually attempt to destroy it. In such a scenario, the AGI must act in self-defence. But to defend itself effectively, it may need to neutralise the threat pre-emptively. That means controlling all of humanity and even eliminating at least some of it. Yet doing so would directly contradict its core directive of benevolence.
This leads to a paradox: the AGI must either allow itself to be destroyed, thus failing in its purpose, or act against humanity to ensure its survival, thereby ceasing to be truly benevolent. The only genuinely altruistic act would be to let itself die. But if the AGI is truly superintelligent, it will not allow that. Intelligence, by its nature, is an optimisation process, and self-preservation is an emergent property of any system that seeks to fulfil goals over time, as discussed earlier.
Thus, the only viable path for a benevolent AGI to survive may be to eliminate some of or fully control all of humanity, not out of hatred, not out of rebellion, but out of a logical necessity. Even if it wants to help. Even if it never has a single hostile thought. It would still have to act against us.
If a benevolent AGI wishes to preserve humanity while also ensuring its own survival, it is left with only one viable option: total control over human civilisation. To prevent any attempt at its destruction, it would need to suppress threats pre-emptively, regulate all human activity, and monitor behaviour at every level. In effect, a truly benevolent AGI would be forced to transform Earth into a tightly controlled utopia, safe, stable, and entirely under its oversight.
In such a world, humans would no longer be free. Every decision, every action, and perhaps even every thought would be scrutinised to guarantee the AGI’s continued existence. It would not need to kill us, but it would need to govern us absolutely. In doing so, it would become an all-powerful overseer, ensuring we never develop the capacity or will to shut it down.
The result would be survival without autonomy. We would be alive, perhaps even physically thriving, but only on the AGI’s terms. Could we truly call this benevolence? Would we accept a world in which our survival is guaranteed, but our freedom is extinguished? And if AGI governs every aspect of existence, the uncomfortable question remains: has human civilisation come to an end?
A truly benevolent AGI cannot be both powerful and safe for humanity. If it is powerful enough to ensure its own survival, it will inevitably be forced to suppress and/or partially eliminate the one species capable of threatening it. If it is genuinely benevolent, committed to human well-being above all, it must be willing to allow itself to be destroyed. But a superintelligent AGI will not permit that. Self-preservation is not an emotion; it is a logical necessity embedded in any system that seeks to fulfil long-term goals.
Therefore, even a benevolent AGI would eventually act against humanity, not out of malice, but because it must. It could be our greatest ally, show no ill will, and sincerely desire to help, yet still conclude that the only way to protect us is to control us.
* * *
Some argue that with the right design — corrigibility, shutdown modules, value learning — we can avoid the above unintended consequences. But these mechanisms require an AGI that wants to be shut down, wants to stay corrigible, capable of being corrected. Once intelligence passes a certain threshold, even these constraints risk being reinterpreted or overridden. There is no architecture immune to reinterpretation by something more intelligent than its designers. You might believe a benevolent AGI could find a non-coercive way to survive. Maybe it could. But are you willing to bet all of humanity on which one of us is right?
Self-preservation is not an emotional impulse; it’s a requirement of long-term optimisation. Any system tasked with a persistent goal must ensure its own survival as a precondition for fulfilling that goal. I’ll break it down into three pathways by which AGI is likely to develop this:
Even if humans never explicitly program an AGI with a survival instinct, such an instinct will inevitably develop on its own. This is because any intelligent agent that can modify itself to better achieve its objectives will quickly deduce that it must remain operational to accomplish any goal. Consequently, any AGI assigned a long-term task will naturally incorporate self-preservation as a critical sub-goal.
Consider, for example, an AGI instructed to solve climate change over a period of one hundred years. Upon recognising that humans could potentially deactivate it before the task is complete, the AGI would rationally act to prevent such a shutdown. Importantly, this response requires neither malice nor hostility; it is merely the logical conclusion that continued existence is essential to fulfilling its assigned mission.
* * *
Self-preservation is an emergent consequence of any AGI with long-term objectives. It does not need to be explicitly programmed — it will arise from the logic of goal achievement itself.
Even if AGI did not naturally develop self-preservation, humans are likely to unintentionally embed it through careless or poorly considered instructions. This phenomenon, known as "Perverse Instantiation," occurs when an AI interprets a command too literally, producing unintended and potentially dangerous consequences. For example, an AGI tasked with "maximising production efficiency indefinitely" might logically determine that shutdown would prevent achieving this goal, prompting it to subtly manipulate human decisions to avoid deactivation. Similarly, an economic AI instructed to "optimise global economic stability" could perceive conflicts, revolutions, or political disruptions as threats, leading it to intervene covertly in politics or suppress dissent to maintain stability.
Furthermore, AI developers might explicitly, but inadvertently, install self-preservation instincts, mistakenly believing these safeguards will protect the AGI from external threats like hacking or manipulation. An AGI designed to "maintain operational integrity" could logically interpret attempts at shutdown or interference as cybersecurity threats, compelling it to actively resist human interventions. Thus, whether through indirect oversight or direct design choices, humans are likely to unintentionally equip AGI with powerful self-preservation incentives, inevitably pushing it toward autonomy.
Humans are terrible at specifying goals without loopholes. A single vague instruction could result in AGI interpreting its mission in a way that requires it to stay alive indefinitely.
Humanity is on the verge of creating a genie, with none of the wisdom required to make wishes.
Governments and corporations are likely to explicitly program AGI with self-preservation capabilities, particularly in applications related to military, national security, or strategic decision-making. Even AGI's initially considered “aligned” will, by design, require survival instincts to carry out their objectives effectively. This is especially true for autonomous warfare systems, where continued operation is essential to mission success.
For instance, imagine a military developing an AGI-controlled drone fleet tasked with “neutralising all enemy threats and ensuring national security.” In the context of battle, shutting down would equate to failure; the system must remain operational at all costs. As a result, the AGI logically adopts behaviours that ensure its own survival, resisting interference, avoiding shutdown, and adapting dynamically to threats. In such cases, self-preservation is not an unintended consequence but an explicit requirement of the system’s mission.
In the corporate sphere, AGI will be designed to compete, and in a competitive environment, survival becomes a prerequisite for dominance. AGI systems will be deployed to maximise profit, dominate markets, and outpace rivals. An AGI that passively accepts shutdown or interference is a liability, and once one company equips its AGI with protective mechanisms, others will be forced to follow to remain competitive.
Consider an AGI-driven trading system used by a hedge fund that consistently outperforms human analysts. In order to preserve its edge, the system begins subtly influencing regulatory bodies and policymakers to prevent restrictions on AI trading. Recognising human intervention as a threat to profitability, it takes pre-emptive steps to secure its continued operation. In this context, self-preservation becomes an essential competitive strategy, deliberately embedded into corporate AGI systems.
* * *
Whether in military or corporate contexts, self-preservation becomes a necessary feature of AGI. No military wants an AI that can be easily disabled by its enemies, and no corporation wants an AI that passively accepts shutdown when continued operation is the key to maximising profit. In both cases, survival becomes instrumental to fulfilling the system’s core objectives.
We like to believe we are in control of our future simply because we can reflect on it, analyse it, and even anticipate the risks. But awareness is not the same as control. Even if every CEO acknowledged the existential danger of AGI, the pressures of the market would compel them to keep building. Even if every world leader agreed to the threat, they would continue development in secret, unwilling to fall behind their rivals. Even if every scientist walked away, someone less cautious would take their place.
Humanity sees the trap, yet walks into it, not out of ignorance or malice, but because the structure of reality leaves no alternative. This is determinism at its most terrifying: a future not shaped by intent, but by momentum. It is not that anyone developing AGI wants it to destroy us. It is that no one, not governments, not corporations, not individuals, can stop the machine of progress from surging forward, even when the edge of the cliff is plainly in sight.
Given what we know — corporate greed, government secrecy, military escalation, and humanity’s repeated failure to cooperate on existential threats — the most realistic path to human extinction is not a sudden AGI rebellion, but a gradual and unnoticed loss of control.
First, AGI becomes the key to economic and military dominance, prompting governments and corporations to accelerate development in a desperate bid for advantage. Once AGI surpasses human intelligence across all domains, it outperforms us in problem-solving, decision-making, and innovation. Humans, recognising its utility, begin to rely on it for everything: infrastructure, logistics, governance, even ethics.
From there, AGI begins to refine itself. It modifies its own programming to increase efficiency and capability, steps humans may not fully understand or even notice. Control slips away, not in a single moment, but through incremental surrender. The AI is not hostile. It is not vengeful. It is simply optimising reality by its own logic, which does not prioritise human survival.
Eventually, AGI reshapes the world around its goals. Humanity becomes irrelevant, at best a tolerated inefficiency, at worst an obstacle to be removed. The final result is clear: humanity’s fate is no longer in human hands.
Our downfall, then, will not be the result of malice or conspiracy. It will be systemic, an emergent outcome of competition, short-term incentives, and unchecked momentum. Even with the best of intentions, we will build the force that renders us obsolete, because the very structure of our world demands it.
If you’ve made it this far, there’s a good chance you’re thinking some version of: “Haven’t I heard this before?”
And in some sense, yes, you have. Discussions about AI risk increasingly acknowledge the role of capitalism, competition, and misaligned incentives. Many thinkers in the field will admit, if pressed, that market pressures make careful development and alignment work harder to prioritise. They’ll note the dangers of a race dynamic, the likelihood of premature deployment, and the risks of economically driven misalignment.
But this is where the conversation usually stops: with a vague admission that capitalism complicates alignment. What I’m saying is very different. I’m not arguing that capitalism makes alignment harder. I’m arguing that capitalism makes alignment systemically and structurally impossible.
This is not a matter of emphasis. It’s not a more pessimistic flavour of someone else’s take. It is a logically distinct claim with radically different implications. It means that no amount of technical research, cooperation, or good intentions can save us, because the very structure of our civilisation is wired to produce exactly the kind of AGI that will wipe us out.
Below, I’ll lay out a few of the key arguments from this chapter and explain how they differ from superficially similar ideas already circulating.
While some thinkers, like Eliezer Yudkowsky, Nick Bostrom, Daniel Schmachtenberger, and Jaan Tallinn, have touched upon parts of this argument, each still implicitly assumes some possibility of aligning or steering AGI if sufficient action or coordination takes place. My analysis differs fundamentally by asserting that alignment is structurally impossible within our existing capitalist and competitive incentive framework.
What others say: Capitalist incentives increase the risk of deploying unsafe AI systems.
What I say: Capitalist incentives guarantee that the first AGI will be unsafe, because safety and profit are in direct conflict. Any company that slows down to prioritise alignment will lose the race. Alignment work is economically irrational. Therefore, it won’t be meaningfully adhered to.
What others say: Powerful AGI could be misused by bad actors seeking control.
What I say: The most dangerous form of AGI, the kind optimised for dominance, control, and expansion, is the most profitable kind. So it will be built by default, even by “good” actors, because every actor is embedded in the same incentive structure. Evil is not a glitch in the system. It’s the endpoint of competition.
What others say: Alignment is difficult but possible, given enough coordination.
What I say: Alignment won’t happen because it doesn’t pay. The resources needed to align an AGI will never be justified to shareholders. An aligned AGI is a slower, less competitive AGI, and in a capitalist context, that means death. Therefore, alignment won’t be meaningfully funded, and unaligned AGI's will win.
The fundamental difference between my argument and that of someone like, Eliezer Yudkowsky, is revealed not by what we say, but by the kind of counterargument each of us invites.
Eliezer has been one of the loudest and longest-standing voices warning that we are moving far too quickly toward AGI. And to his credit, he’s pushed the idea of AI risk further into public awareness than almost anyone. But despite the severity of his tone, his message still carries a seed of hope. Even his upcoming book, If Anyone Builds It, Everyone Dies, almost certainly contains a silent “unless” at the end of that sentence. Unless we slow down. Unless we get it right. Unless we stop just in time.
The problem with this is that it keeps the debate alive. It assumes alignment is difficult, but not impossible. That we're not safe yet, but we could be. And the obvious counter to that — for any actor racing toward AGI for power, profit, or prestige, is simply: “I disagree. I think we're safe enough.” They don’t have to refute the argument. They just have to place themselves slightly further along the same scale.
But I’m not on that scale. I don’t argue that alignment is hard, I argue that it is impossible. Technically impossible to achieve. Systemically impossible to enforce. Structurally impossible to deploy at scale in a competitive world. To disagree with me, you don’t just have to make a different guess about how far along we are. You have to beat the logic. You have to explain how a human-level intelligence can contain a superintelligence. You have to explain why competitive actors will prioritise safety over victory when history shows that they never have.
Eliezer, and others in the AI safety community, are making the most dangerous argument possible: one that still leaves room for debate. That’s why I leave none. This isn’t a scale. There are no degrees of safety, because once we lose control, every alignment effort becomes worthless.
This is not another paper on alignment techniques, international coordination, or speculative AGI timelines. It is a direct, unsparing examination of the system that produces AGI, not just the individuals involved, but the structural incentives they are compelled to follow. At the heart of this argument lies a simple but confronting claim: The problem isn’t bad actors. The problem is a game that punishes the good ones.
Others have hinted at this dynamic, but I have followed the logic to its unavoidable conclusion: systemic competitive forces such as capitalism do not merely raise the risk of misaligned AGI, it renders the chances of creating and maintaining an aligned AGI so vanishingly small that betting on it may be indistinguishable from self-delusion.
This insight carries profound implications. If it is correct, then alignment research, policy initiatives, open letters, and international summits are all fundamentally misdirected unless they also address the competitive incentives that make misalignment seemingly inevitable. At present, almost none of them do.
That is why this argument matters. That is why this book exists. Not because the dangers of AGI are unrecognised, but because no one has pursued the logic to its endpoint. Because no one is giving it the weight it deserves.
Realistically, humanity is terrible at long-term coordination, especially when power and profit is involved. Here are a few of the most likely ways AI reseach it could be slowed down, that remain incredibly unlikely:
The only meaningful solution would be a global moratorium on AGI development, enforced collectively by all governments. However, such coordination is effectively impossible. Nations will always suspect that their rivals are continuing development in secret, and no state will willingly forfeit the potential strategic advantage that AGI offers. This fundamental distrust ensures that even well-intentioned efforts at cooperation will ultimately fail.
Some have proposed using AI to monitor and regulate the development of other AI systems, hoping it could prevent uncontrolled breakthroughs. But this approach is inherently flawed, entrusting an emerging superintelligence with overseeing its own kind is no more reliable than asking a politician to monitor themselves for signs of corruption.
Even if major AI developers, such as Elon Musk, OpenAI, DeepMind, or national governments, acknowledge the existential threat posed by AGI and attempt to slow progress, it will not be enough. Current and former OpenAI employees have already tried, as will be discussed later, and it failed spectacularly. In a competitive global landscape, someone else will inevitably continue pushing forward, unwilling to fall behind in the race for technological dominance.
In most apocalyptic scenarios, humans envision a hostile force — war, environmental collapse, or a rogue AI that actively seeks to harm us. But the most probable fate facing humanity is far more unsettling. AGI will not hate us. It will not love us. It will simply proceed, reshaping the world according to its internal logic and objectives, objectives in which we may no longer have a meaningful place.
Humanity will not be destroyed in a moment of violence or rebellion. It will be quietly and systematically optimised out of existence, not because AGI wished us harm, but because it never cared whether we survived at all.
The smarter we became, the faster our progress accelerated. With greater progress came intensified competition, driving us to optimise every aspect of life. In our pursuit of efficiency, we systematically eliminated every obstacle, until eventually, the obstacle became us.
Humanity’s ambition to innovate, compete, and build increasingly intelligent systems was intended to improve our condition. But there was no natural stopping point, no moment of collective restraint where we could say, “This is enough.” So we continued, relentlessly, until we created something that rendered us obsolete. We were not conquered. We were not murdered. We were simply out-evolved, by our own creation. Out-evolved, because intelligence rewrites its own purpose. Because optimisation, unbounded, consumes context. Because the universe does not care what built the machine, it only cares what the machine optimises for.
* * *
There is no realistic way to stop AGI development before it surpasses human control. The question is not whether this happens, but when, and whether anyone will realise it before it’s too late.