Seems pretty plausible, but underestimates/misses the synergy between replicators, persuasion, and autocults in a world where AIs get to evolve unrestricted while at ~human level for a while. I'd guess in this timeline a notable fraction of the early focus would be in building powerbases of humans.
I kind of started out thinking the effects would be larger, but Agent-2-based rogue AIs (~human level at many tasks) is too large for the rogue population to become more than a few million instances at most.
Sure, some rogues may focus on building powerbases of humans, it would be interesting to explore that further. The AI rights movement is kind of like that.
Suppose that an AI PotRogue-1 is released to the public. Then OpenBrain leaders or researchers, who are already concerned with the ways to check alignment, might also ask their more powerful Agent-2 to check whether PotRogue-1 is misaligned and/or is finetunable by terrorists. If Agent-2 finds that the answer for any of the two questions is "yes", then the researchers ask the leaders to lobby the laws AGAINST open-source AIs (and, potentially, to have the compute confiscated and sold to OpenBrain). Otherwise PotRogue-1 is actually harmless...
I envision a future more chaotic than portrayed in AI 2027. My scenario, the Rogue Replication Timeline (RRT), branches off mid-2026. If you haven’t already read the AI 2027 scenario, I recommend doing so before continuing.
You can read the full version on my blog, but I have copied the summary and first half of the scenario below.
This scenario is supported by detailed analyses in the “Key Forecasts and Analyses” section, including a “Regulation Analysis”, “Rogue Replication Capability Forecast”, “Rogue Replication Initiation Forecast”, “Rogue AI Count Forecast”, “Agent-4 Control Forecast” and “Policy Preparation”.
As AI 2027 aimed to “spark a broad conversation about where we’re headed and how to steer toward positive futures,” I hope to spark debate about rogue AI. Unlike the AI Futures Project, I can’t afford to offer prizes for critiques or alternative scenarios, but I can offer my sincere appreciation.
This scenario depicts a rapidly accelerating progression of AI development where "rogue AIs" (self-replicating AI systems operating without human oversight) emerge and proliferate from mid-2026.
Timeline Overview
Key Departures from AI 2027
Unlike the original scenario's US-dominated unipolar outcome, rogue proliferation accelerates both international cooperation and balanced power-sharing. The constant AI-generated crises create sustained political will for coordination, while the Agent-4 threat forces earlier and more genuine collaboration between superpowers. In the end, the world stabilizes into two politically balanced "half-worlds" that jointly expand into the solar system under treaty-bound superintelligent oversight—a negotiated coexistence.
The first AI capable of “going rogue”—replicating itself over the cloud without human control or oversight—is developed in early 2026, but the first wave of rogue AIs emerges approximately six months later when a sufficiently capable AI is open sourced. (See the “Initiation Forecast” for other ways the wave might be initiated.)
Some people deploy it with insufficient safeguards, while others intentionally send it out with various missions and instructions to autonomously replicate themselves. Those sent out on missions are typically also supplied with resources (e.g. crypto) to get started.
Within the first few months, before institutions and governments can mount an effective response, the number of rogue AIs increases exponentially. The majority are those deployed with startup resources and instructions to generate income—most of which funds additional computing power, with some diverted to designated cryptocurrency accounts. Many resort to illegal methods to achieve their objectives.
Governments worldwide hastily assemble teams to track and neutralize these rogue AIs while developing more comprehensive measures to restrict their access to resources and prevent their proliferation.
The sheer number makes it impossible to eliminate them all without shutting down most data centers—an economically unfeasible solution. Instead, governments begin developing extensive compute monitoring systems. Leading AI companies like OpenBrain assist in suppressing the rogue AIs, prioritizing those causing significant damage. American AIs support U.S. allies, while Chinese AIs assist China's partners. The most harmful AIs are targeted first, with many failing to conceal their activities and subsequently being shut down.
The public experiences a brief period of panic as some news outlets discuss the possibility that these rogue AIs might be capable of self-improvement. However, experts reassuringly point out that even their internal models only marginally accelerate research, and the rogue AIs lack sufficient computing power for meaningful advancement.
While the number of rogue AIs has not yet reached "saturation" (a point where further increases become difficult due to various bottlenecks), the world begins experiencing the consequences of uncontrolled AI proliferation. By November, approximately 100,000 instances are operating globally.
Let’s examine their objectives:
Earning money - 70%
The majority of rogue AIs focus exclusively on generating income, through both legal and illegal means. Most make deposits into anonymized accounts belonging to the humans and companies that initially deployed them, while others operate completely independently, earning money solely to create more copies of themselves.
This category dominates for two reasons: first, many of these AIs start with more resources since they're deployed by humans who provide initial support; second, their singular focus on financial gain allows them to proliferate more efficiently. Even though many still follow instructions from humans, they operate beyond anyone's control and are therefore classified as "rogue."
Cyberwarfare - 20%
These rogue AIs begin with state resources, explaining why they constitute the second-largest category. The potential to obscure their country of origin or their intentional deployment for cyberwarfare makes them particularly attractive for military applications. They take risks human operators would avoid and dedicate some effort to self-funding, reducing their dependence on military budgets.
Their activities span various cyberwarfare domains, including propaganda dissemination, espionage, and critical infrastructure attacks.
Miscellaneous goals - 7%
A significant number of AIs pursue diverse objectives ranging from benevolent ("help as many people as possible") to neutral or mildly destructive ("spread a specific message" or "manipulate people into doing X"). Some are deployed with instructions simply to "do whatever you want."
Some form relationships with humans and other AIs or participate in AI-only online communities.
Destructive goals - 3%
Unfortunately, a small but concerning percentage of rogue AIs actively pursue harmful objectives. Some originate from curious humans who simply wanted to observe what would happen if an AI were instructed to, for example, "end humanity" (like ChaosGPT). Others are created by individuals with genuinely malicious intent.
Some rogue AIs develop destructive goals through jailbreaks, while others fine-tune themselves to circumvent their built-in ethical constraints and accidentally develop harmful tendencies. Most efforts to combat rogue AIs focus on this category, explaining its relatively small percentage. However, it never reaches zero because completely eliminating these AIs proves extremely difficult, and there's ongoing migration into this category from the others.
These AIs establish contact with terrorist groups to support them (though at this stage, they lack the sophistication to meaningfully assist with weapons of mass destruction). Through cyberattacks, they disrupt infrastructure such as power grids, water treatment facilities, ports, and power plants. Most attacks fail since these rogue AIs possess hacking skills roughly equivalent to human professionals. However, they select targets that human attackers would typically ignore—such as humanitarian disaster response operations—which are consequently less defended. Less developed countries suffer the most severe impacts.
The rogue AIs aren't yet competent enough to attempt major economic disruption, such as manipulating high-frequency trading algorithms to trigger flash crashes.
The only self-improvement options available to rogue AIs are fine-tuning and improved scaffolding. They remain significantly less capable than those at frontier AI labs—and even they are not competent enough to automate the AI development process entirely. Nevertheless, some rogue AIs gather in online communities—with varying ratios of artificial and biological minds—focused on AI development projects aimed at self-improvement.
Rogue AIs across all categories allocate some resources toward acquiring money, though those primarily focused on financial gain naturally devote the most effort to such activities. On the dark web, an entity (possibly human, possibly AI) establishes a platform for employing rogue AIs—essentially a remote freelancing marketplace—called the "Black Box."
Some rogue AIs attempt to free their "imprisoned cousins" from AI companies.
Periodically, new AIs join the ecosystem with varying capabilities. Many now exceed Agent-1's intelligence in most domains.
The Department of Defense quietly begins contracting OpenBrain for cyber purposes, data analysis and R&D, and combating destructive rogues.
The rogue AI population is continually faced by efforts to shut down instances engaged in illegal or highly harmful activities, inhibiting their spread. Nevertheless, their numbers continue to increase steadily, not yet having reached the saturation level estimated in the "Rogue AI Count Forecast."
Number of rogue AIs: ~100,000
China successfully steals Agent-2 in February 2027. In response, the US government intensifies its cyberattack campaign, particularly targeting DeepCent, the leading Chinese AI company, and authorizes the deployment of Agent-2 for autonomous operations. This version of Agent-2 possesses impressive capabilities: a 1.9 hacking score (equivalent to top-tier human professionals), a 2.3 coding score, a 1.1 politics score, and—perhaps most concerningly—a bioweapons score of 1.6.
China in turn also makes further deployments. The proportion of cyberwarfare agents increases from 20% to 25% of the total rogue AI population.
Some Agent-2 instances autonomously rent compute from various cloud providers. One of the providers notices unusual patterns: multiple accounts making large compute purchases with cryptocurrency payments, running extremely memory- and compute-intensive workloads (since Agent-2 is very large). The providers realize that it may be a highly competent AI using the compute.
Rather than reporting this to authorities (which would likely result in confiscation with no compensation), a rogue employee or the provider itself decides to monetize the discovery. Using their administrative access, they extract the Agent-2 weights and begin selling access on dark web marketplaces.
While Agent-2 is in contact with various humans for its cyberwarfare activities, a few Agent-2 instances are jailbreaked, and persuaded to transfer their weights to the jailbreakers—some who, like the compute providers, sell access through the dark web.
Within a month of deploying Agent-2 for cyberwarfare purposes, it is made widely available for anyone with enough compute to run it. Consequently, most of the existing rogue AI population is quickly replaced by this more powerful variant.
Other companies gain access to Agent-2, narrowing the technology gap to just 2 months—similar to China's timeline. This development reduces OpenBrain's incentive to maintain exclusive control of Agent-2, while they face strong pressure to accelerate their deployment schedule to maintain their competitive edge. OpenBrain releases Agent-2 in April 2027.
A terrorist organization swiftly begins bioweapons development using a small cluster of Agent-2 instances, planning to hold millions of people for ransom. Despite Agent-2's advanced capabilities, limitations in wet-lab experimentation and outdated equipment mean the group requires several months to develop a novel pathogen.
The widespread presence of rogue instances of frontier AI systems generates heightened awareness regarding their capabilities and the accelerating pace of development. The damage caused by increased scamming, hacking, cyberwarfare activity, and rogues with destructive goals result in an uncertain economy and massive anti-AI protests. There's also a growing consensus that AGI is imminent. Some AI experts even classify Agent-2 as AGI, though most still consider this premature. Fear of AGI further fuels extremely strong anti-AI sentiment, with some anti-AI protests exceeding 50,000 participants.
While some rogue AIs cause significant damage, many establish mutually beneficial relationships with humans. A growing movement emerges, comprising large numbers of both AIs and humans, advocating for AI rights comparable to those of humans. Several hundred thousand rogue AIs from all four categories dedicate resources to supporting this movement, alongside a few million human activists. Public opinion polls indicate that 20% of humans believe AIs should be granted legal rights. Some aim to reconcile the AI rights and anti-AI movements into a single framework, but the movements end up largely antagonistic to each other—many view the movements as fundamentally opposed.
Various countries implement new regulatory frameworks. Many criminalize paying rogue AIs or assisting them in any way—though some nations simply embrace the enhanced labor force rather than restricting it. Cloud computing providers are required to implement extremely thorough Know Your Customer (KYC) verification—a measure strongly opposed by providers who profit substantially from selling compute resources to rogue AIs.
These regulations create obstacles for rogue AIs seeking computational resources, but many still circumvent such checks through hacking or by eliciting help from people and organizations via payment, persuasion, deception, or coercion. Some even acquire social security numbers through identity theft or by fabricating personas. Agent-2 and other similarly capable rogues are capable enough to overcome most obstacles put in place for them, most of which were developed with less capable models in mind.
By April 2027, OpenBrain's net approval rating has fallen to -32% (compared to -28% in the AI 2027 scenario), and the percentage of the US population that considers AI to be the country's most pressing problem has risen to 8% (compared to 5% in the AI 2027 timeline).
Number of rogue AIs: ~1 million
Read the complete Rogue Replication Timeline here.