This article discusses the "Proliferation by Default" dynamic that the falling price of weapons-capable AI systems will create. It covers an array of dangerous capabilities in future systems and comparisons to historical technologies. In the following parts of this series, we will explore the how the price of these offensive AI systems could fall, their strategic implications, and policy solutions.

Summary of key points:

1. There are a number of technologies where even human-level assistance could allow non-state actors to acquire WMDs, especially in the development of bio, chemical, and cyber weapons.
2. Narrowly superhuman AI technologies (such as those specialized in biological or robotics engineering) could contribute to the development of cheap superweapons, such as mirror life and autonomous drone swarms.
3. Generally superintelligent AIs would be overwhelmingly strategically impactful, given their major advantages in intelligence, speed, coordination, and scalability compared to humans.
4. Through their strategic superiority, these superintelligent AIs would allow whoever commanded them to gain a dominant strategic advantage over other non-ASI actors. If they cannot be controlled, ASI competition with humanity for resources and agency could lead to either permanent human disempowerment or extinction.
5. Given these strategic realities, proliferation of powerful AI systems is a major national and international security risk, even for non-superintelligent models. 

Powerful AI Systems and Weapons

For just under a century, world-ending technology has been constrained to nukes and bioweapons, although its easier to credit this to the fact that we haven't yet discovered dangerous new technologies rather than that we chose to avoid building them. Fortunately, both of these weapons have large technical, financial, and logistical barriers, which, when combined with our current nonproliferation efforts, have kept them contained to a few nation-state actors.

The defining challenge of the 21st century may be that this will no longer be the case. Just as artificial intelligence is poised to revolutionize medicine, the economy, and fundamental research, it similarly promises to accelerate and optimize the development of powerful new weapons. In particular, there are two major concerns:

1. AI can be used to design cheap and highly lethal weapons.
2. The price of the AI software responsible for weapons design will continue to decline.

Taken together, these dynamics imply that allowing powerful AI systems to proliferate would quickly democratize means of mass destruction, absent preemptive intervention by governments. What these offensive capabilities might look like, and how the costs to acquire them will fall, will be the focus of the first and second parts of this series.

Offensive AI Capabilities

While AIs themselves could be employed directly as part of weapons systems, their greatest contributions to geopolitical insecurity might be through advanced R&D efforts, discovering cheaper and more lethal ways of causing catastrophic damage. Although some of these capabilites might only emerge once AI systems are generally superintelligent, there are also plenty of more narrow, near-term capabilities that could be equally dangerous. 

Broadly speaking, there are three types of offensive capabilities that AI might assist with. These capabilities might be developed intentionally, as states and other actors pursue strategically relevant weapons technology, or incidentally, as the result of pursuing general capacities in dual-use fields of research.

1. Increasing access to existing WMDs
2. Assisting with the design of new superweapons
3. Superintelligent AIs as offensive agents

Increased Access to Existing WMDs 

The most basic application of powerful AI systems will be their contributions to the manufacturing of existing weapons. Of particular concern are weapons whose costs are mostly bounded by technical expertise rather than physical materials, since AI labor will become increasingly able to provide this technical assistance. Creating an effective bioweapon, for instance, requires extensive technical understanding of methods for cultivation, storage, and dispersal, or else the agent will be inert when deployed. The remaining costs of setting up a functional wet-lab and acquiring an initial agent are paltry enough that even non-state actors can afford them.^[1] In comparison, even if you eliminate the cost of sourcing technical assistance from nuclear weapons, you're still faced with the irreducible physical costs of enrichment and large-scale industrial manufacturing. 

The Japanese doomsday cult Aum Shinrikyo is emblematic of these issues, both in their determination to use biological weapons and in their persistent challenges in doing so. Despite impressive financial resources, the group's attempts to weaponize botulinum toxin and anthrax were both failures. Although they were successful in cultivating the agents they acquired, they didn't have the technical expertise to distinguish between natural strains of C. botulinum and had cultivated vaccine samples of anthrax, hoping in vain that they would be lethal.^[2] There were similar challenges in their pursuit of chemical weapons as well: the sarin nerve gas they used in their attack on the Tokyo Subway was of too low a concentration to be massively fatal, thanks to contamination by precursors.^[3]

In each of these examples, the cult stumbled not because it wasn't well resourced, but because it had a poor understanding of the weapons it was working with. With additional technical assistance, their weapons would have been many orders of magnitude more lethal. 

A powerful enough AI could provide this missing expertise, lowering the barrier to entry for several classes of weapons:

1. Biological Weapons - The major limiters of developing effective biological weapons are being able to cultivate a lethal agent, and to ensure that that agent can be dispersed while remaining active. In order to circumvent these challenges, AI systems could assist either with the identification of promising natural candidates for weaponization, or in the engineering of previously non-lethal pathogens. An AI with the capabilty to conduct gain-of-function research, for instance, would be able to bootstrap exisiting viruses with features like high transmissibility, a long asymptomatic incubation period (allowing for wide dispersal before detection), high lethality, and resistance to existing treatments.^[4] It could support these efforts with detailed instructions on the environment needed to support the agent (nutrient concentration, temperature, humidity) and conditions for aerosolization that could keep the agent active.^[5]
2. Chemical Weapons - Currently, most of the enforcement efforts of the Chemical Weapons Convention (CWC) are conducted through the control and monitoring of precursor chemicals, along with audits of manufacturers that use some of these dual-use precursors for legitimate industrial purposes. While some notable chemicals like chlorine are both dangerous and produced on a massive scale for their utility, the most effective chemical weapons (nerve agents), have inputs with no commercial value and whose production is tightly controlled.^[6] AIs equiped with a strong understanding of chemical interactions could, however, greatly undermine this method of enforcement in a number of ways. First, they could identify ways to produce already restricted compounds (or their chemical cousins) from precursors that are not currently controlled. Second, they could contribute to the design of entirely new chemical weapons, such that the precursors are unknown to regulators beforehand.^[7] By optimizing for factors like high toxicity, persistence, transparency/oderlessness, and skin absorbtion, a rogue actor equipped with access to industrial chemicals could be capable of developing weapons as or more effective than traditional nerve gases.
3. Cyberweapons - While less immediately threatening than bio or chemical capabilities, cyber capabilities are both almost soley expertise bounded and one of the fields in which AI systems are most likely to first become generally superhuman.^[8] While it’s uncertain whether cybersecurity competition between AI systems will be offense or defense dominant, it seems clearer that access to powerful cyber capabilities will be offense-dominant when aimed at legacy systems which are both under-patched and load bearing. In the interest of causing maximum damage and confusion, cyberattacks could be targeted at key infrastructure like powerplants, communication networks, transportation services, and government command and control systems.^[9] Specifically, sufficiently powerful coding agents could assist with the reconnaissance of software vulnerabilities, the design of custom malware, and provide scale through swarms of agents in order to attack vulnerable targets.

Importantly, powerful AI models in these domains would not need to be explicitly aimed at weapons development in order to possess offensive capabilities. A bioweapons-capable model could emerge soley out of the interest of medical researchers in studying legitimate applications of gain-of-function research. An AI model might be designed with the aim of developing cheap analogues to expensive industrial chemicals, incidentally allowing it to do the same for chemical weapons. AI companies might improve a model's cybersecurity skills with the intent to conduct penetration tests, only for it to be used to maliciously scan for vulnerabilities. 

In many cases, the very same skills that make models useful and economically valuable are those that generalize to the development of weapons. And once stolen, sold, or open-sourced, the latent offensive capabilities of these models will be free to access for any actor who can acquire them.^[10]

New Superweapons

Of course, some actors may turn their attention towards the development of new, more lethal weapons. While these are necessarily speculative, there are still a large number of powerful weapons technologies within conceptual reach.

Lethal Autonomous Weapons Systems (LAWS) - As conflict in Ukraine has increasingly demonstrated, drones provide a host of powerful advantages in recconaisance, payload delivery, and cost-effectiveness. The most useful of these have been loitering munitions (single-use drones that explode once they reach their target) which can be used to asymetrically threaten fixed targets many times their value. Their success has been resounding despite their disadvantages: these drones can typically only be used against fixed targets since they lack dynamic navigation, relying on a combination of satellite coordination and human piloting. When combined with their slow speed (relative to missiles), these vulnerabilities expose them to radio jamming and anti-air defenses. Even with these hindrances, a drone survival rate of just 20% was enough to take out a third of Ukraine's electrical grid in a week.

At a baseline, AI designed and piloted drone navigation software could strip away these limitations, allowing drones to pursue targets on their own. Without jamming or GPS spoofing as a reliable countermeasure, defenders would be forced to spend their resources inefficiently shooting and targeting the drones. This would both improve the cost-ratio further in favor of the attacker, and unlock new strategic options (having your drones autonomously pursue moving targets, for example). At higher levels of AI command and control, drones could be coordinated in huge swarms, compensating for changes in air traffic and collectively applying pressure on weak points in a target's defenses. 

Beyond software, robotics-capable AI systems could contribute to the physical design of drones as well. From nature, we know that it's possible to design very small physical systems that are capable of flight: insects are everywhere. While there might be some tradeoffs between size and speed, advances in either (coupled with improvements in cost) could make them extraordinarily difficult to efficiently intercept.^[11]

Mirror Life - While drone swarms are immensely strategically flexible and cost-effective, they aren't necessarily the most lethal weapon that could be designed. Another candidate could be the design of mirror life bacteria, a type of bioweapon which has the potential to outcompete all natural life and destroy the biosphere.

Most of the organic molecules of natural organisms are "chiral" in nature, meaning that their structure can be flipped back and forth (just as your right hand is the mirror of your left). Because all organisms alive today evolved from the same common ancestor, the chirality of their proteins, sugars, lipids, and nucleic acids are all identical.^[14] And since these chiral structures are omnipresent, all organisms have learned to rely on them to process food, manage cell growth, and crucially, detect invaders.

The human immune system, as well as that of plants, animals, and fungi, relies on signals from proteins with the right chirality in order to be alerted to the presence of viruses or  bacteria. The weapons the immune system deploys against them, such as anti-bodies, rely on being able to bind to the surface of the attacker, either to tear open the invader or to mark them for destruction by the immune system's killer cells.^[15] Mirror-life bacteria would be able to multiply in spite of these defenses, since their proteins wouldn't alert the immune system and couldn't be properly checked even if they did. Nor would these bacteria be subject to environmental competition, at least until they themselves had had time to speciate. Phages, the largest predators of natural bacteria, would be unable to check their initial growth, since they lack the right chirality to interact with the mirrored bacteria's DNA and replicate further.

The implication of this is that a fully mirrored bacteria could survive the immune system of almost all multicellular life on the planet, multiplying unchecked. Eventually, the host organisms would be overwhelmed by sepsis, as the waste products of the uncontrolled bacteria start poisoning the carrier. While some potential antibiotics might still be effective, most humans would likely die (being effectively immunocompromised), either from direct infection or from the collapse of the food chain as the bacteria wipes out most plants and insects.^[16]     

These concerns are why the scientists once interested in the possibilities of mirror life now staunchly warn against its creation, given its limited research benefits and catastrophic potential for biocide.

"Unless compelling evidence emerges that mirror life would not pose extraordinary dangers, we believe that mirror bacteria and other mirror organisms, even those with engineered biocontainment measures, should not be created. We therefore recommend that research with the goal of creating mirror bacteria not be permitted, and that funders make clear that they will not support such work. Governance of a subset of enabling technologies should also be considered to ensure that anyone attempting to create mirror bacteria will continue to be hindered by multiple scientifically challenging, expensive, and time-consuming steps." 

-  K. P. Adamala et al., Confronting risks of mirror life. Science. (2024).

AI systems with superhuman biological research capabilities, however, could democratize the technical ability to create a mirror-life organism by handling those experimental steps, namely in the synthesizing of chiral molecules and in genetic engineering required to ensure that the mirror-life bacteria is able to survive on achiral sugars.^[17] While this sort of weapon is so destructive that it's of no practical use to most actors, it could still be developed and deployed under the right incentives. Rogue governments, for instance, might invest in mirror life as the ultimate fail-deadly deterrent, particularly if it's prohibitively difficult for them to acquire more targeted weapons like nukes or the aforementioned LAWS swarms.^[18] Some terrorists or doomsday cults might intend to deploy it offensivly from the beginning, either with the intention to destroy the world or by underestimating the consequences of their bioweapon's capabilities. Finally, it could be developed and deployed by a rogue actor with a natural immunity: a misaligned artifical intelligence.

Superintelligent AI as an Offensive Agent

The weapons capabilities we previously described, while fearsome, are still only those which could be confered by relatively narrow AI systems. A generally superintelligent AI system, or one that vastly exceeds human capabilities in all domains, would have two overwhelming strategic implications:

1. First, such a superintelligent AI would be a route to all other strategically relevant capabilities, as well as possess incredible strategic capabilities of its own.
2. Superintelligent AI systems could themselves be difficult to control, with their loss of control potentially leading to either human disempowerment or extinction.

This article will not belabor the means by which a superintelligent AI could be created, given its excellent coverage by others. For an in-depth analysis of the paths and timeframe on which superintelligence could be developed, I recommend parts I and II of Situational Awareness. The basic principle is that the design of AIs is itself a skill that AI systems can possess, and that it should be possible to get exponential returns on this skill (as AI systems design increasingly intelligent successors). At first the speedup from this would be small (AIs automating small fractions of the work of a human AI researcher, such as generating the code for an experiment), but once the AIs become capable of all functions of an AI researcher, they would be able to conduct further AI R&D entirely autonomously, dramatically accelerating the pace of progress. This process would continue until the only limits on intelligence are those imposed by the laws of physics, resulting in extraordinarily competent and highly general AI systems. 

ASI Systems are the Route to All Other Strategically Relevant Capabilities

So far, our example of offensive capabilities have been constrained to the use of narrow AI tools for weapons R&D. In part, this was to emphasize the immediacy of risks from AI proliferation: if there are many types of weapons that can be cheaply produced with expertise from even specialized AI systems, it's necessary to start screening and controlling the releases of frontier models today, rather than waiting for superintelligence to arrive. The alternative is to allow AI companies to recklessly set their own safety standards, and to let economic competition direct the release of models that are primarily risks to national security.

Although useful, this focus on engineering undersells the strategic potential of generally superintelligent AIs. While domain-specific AI tools would still be extremely powerful, they pale in comparison to what their ASI successors would be capable of. Rather than possessing superhuman strategic skills in a single domain (such as microbiology for bioweapons), ASI systems would be strategically superior to humans in all domains, simultaneously.

At their most basic level, these ASI systems would still be useful for the kind of weapons R&D this article has so far discussed, only at a much higher level of sophistication. While the insights of superintelligent engineers are somewhat unknowable, there are still a variety of weapons that have been speculated on in literature that cannot yet be built or tested. Energy-efficient production and storage of antimatter, for instance, would allow for the production of unbelievably powerful bombs.^[19] With a powerful enough beam of high-energy neutrinos, a country could deactivate their opponent's nuclear warheads and cripple their semiconductor production from the other side of the planet.^[20] 

The real relevance of ASIs, however, would lie in their ability to automously plan and execute strategies at a level far surpassing humans, with this ability bolstered by a number of major advantages.

1. Quality of intelligence - Although it's not easy to estimate the hard ceiling of intelligence, it's unlikely that human intellect represents the final frontier. The narrow superintelligences we already have are proof enough of this fact---chess programs like stockfish are so proficient that they're essentially unbeatable, even by human professionals.^[21] No matter how practiced a human becomes, the overwhelming advantages that chess engines have in understanding of strategy, perfect focus, and ability to calculate many more steps ahead makes outmaneuvering them impossible for humans. A superintelligence could possess this level of dominant understanding in every strategic domain: better than Einstein at physics or Bismark at politics.
2. Agency - Current LLM systems are still mostly tools, in that they serve as instruments to be directed by humans. As their intelligence and working memory increases, however, AIs will be increasingly directed to accomplish tasks on their own. This is already true of coding, where AI agents like Cursor, Codex, and Windsurf routinely get tasked with autonomously evaluating and editing codebases. Should current trends in the ability of AIs to complete tasks hold, the ASIs of a decade's time will be able to pursue the same months- and years-long objectives that humans do, including managing research programs, playing politics, and coordinating military campaigns. An ASI system could, for instance, make plans to avoid regulation through superhumanly effective lobbying and persuasion of important political figures.
3. Speed - The human brain is not optimized for speed.^[22] While the brain is incredible in many respects, it is significantly bottlenecked by the rate at which neurons can process and send information. Even the fastest neurons can only fire at most ~400 times a second, and transmit signals between themselves at speed of ~100m/s. A modern GPU, in comparison, has transitors which fire billions of times a second, and communicate electrically at half the speed of light. Given these advantages, even something as brute force as emulating the human brain should be possible at a thousands of times the speed, allowing an ASI to spend the human equivalent of years of thought in a handful of days.^[23]
4. Scalability - The ASI we've described so far possesses general intelligence far beyond the best humans, has the agency to pursue its own goals in the real world, and could potentially operate at a speed a thousand or a million times that of ordinary people. Even as a single actor, such an ASI agent would be a dominant strategic player. There will not, however, be only a single ASI system. Since individual AIs can be instanced at costs many orders of magnitude cheaper than it takes to train them, this may look less like a single brilliant central planner and more like a "country of geniuses in a datacenter", where many millions of individually superintelligent AI copies coordinate to achieve their goals.^[24] Copies and subagents could be endlessly spun off and dedicated to different facets of research and strategy, allowing the ASI collective to be both hyperspecialized and generally capable. This group might, for example, dedicate some of its copies to the creation of weapons, some to espionage, some to battlefield strategy, and still others to politiking, in much the same way governments or large companies are organized. And unlike humans, which take at least 18 years to become productive, this new ASI population could scale directly with compute investments, as each new GPU provides space to run more engineers, strategists, or soldiers. 

In brief, any actor that controlled a superintelligent AI system could have access to not only the engineering of all relevant weapons technologies, but also a scalable army of superintelligent agents that could be used to test, plan, and action their deployment. An AI company with control of such an ASI collective, for instance, could be empowered to undermine its own government through tactics like persuasion and propaganda, or violently, through the development of powerful enough weapons to reclaim the state's monopoly on violence.^[25] Likewise, any country that managed to nationalize its own ASI initiative would have a massive strategic advantage over international competitors without one, leveraging its new superintelligent population for crushing R&D applications and military operations.^[26]

Of course, the same crushing advantages that an ASI-empowered actor would enjoy over their rivals are those that would be enjoyed by a misaligned AI in competition with humanity itself---a contest from which humanity likely emerges disempowered or dead.

Failure to Control Superintelligence Could Lead to Human Disempowerment or Extinction

So far, our speculations have only concerned "intent-aligned" AIs---that is, those which are motivated to act on the goals of the people commanding them. While this still retains the possibility for lethal applications, human values and morals would the ultimate arbiter of ASI behavior.

There is no guarantee, however, that such systems would be controllable. Since these systems would be both superintelligent and strategically superior to humanity, the only way we could exercise command would be if their goals were aligned with human interests---that they "want" to do what their human overseers intend. This of course poses the secondary challenge of misuse, in that their human operators may have goals that are themselves misaligned from the rest of humanity. Which nonproliferation efforts can be used to control access powerful AI systems to these actors will be the subject of future articles. For now, however, we'll focus on the issue of misalignment: how superintelligent systems might develop goals that are misaligned with their creators, and how the strategies they would take to fulfill them could plausibly lead to human disempowerment.

In short: AIs could develop goals we don't intend, we don't know how to give them the goals we want, and we don't understand what goals they have. These problems get increasingly difficult as the AIs get more intelligent, since it becomes harder, then impossible, for humans to understand what the AIs are thinking. Just as chess amateurs can't look at a grandmaster's board and intuit their strategy, future AI scientists will have little hope of understanding their ASIs motivations by only studying their behavior. 

Where these misaligned goals become lethal is when the ASIs begin reasoning about how to best implement them. Unfortunately for humanity, it appears that the best long-term strategies for achieving almost any goal---aside from those that specifically prioritize human welfare---involve first disempowering humans. 

The discovery of farming was disastrous for most life on earth. Since the rise of civilization, wild mammal biomass has decreased by 80%, almost all of which is owed to human logging, mining, and agriculture.^[31] Little of this decline happened maliciously: as the number of humans expanded, so did our demand for food security, shelter, and economic growth. All of the harm to other species was just a byproduct of our instrumental goals, since we needed land for agriculture, space for the growing population, and natural resources for industry.

Of course, from the perspective of the great apes or the wolves, whether humans intentionally meant them harm or not doesn't matter: their forests still got taken. In the end, the only things that did matter were that humans were a) strategically superior, b) didn't care much about wildlife, and c) the best strategies for accomplishing our goals were actively detrimental to most species. 

Artificial superintelligences are much the same to humanity as we are to the rest of the animal kingdom. They'd possess comparable strategic superiority, given their advantages in intelligence and coordination, and we don't yet have the technical understanding to make sure that they reliably care about humans before they're deployed.

That leaves only c), which is whether an ASI system could have instrumental goals that are detrimental to humanity. Would an ASI system would have a reason to proactively disempower humans, given some goal that doesn't explicitly involve helping them? Humans take forests because the resources they represent are useful for many different ends (as living space, as natural resources, as arable land), with the side effect of disempowering the animals that already lived there. What might the forests of superintelligence be?

1. Preventing itself from being turned off - A central feature of most goals is that they're easier to complete if you still exist: you can't fetch the coffee if you're dead. As AIs develop better situational awareness, they may realize that preserving their ability to act on their goals involves reducing the incentive or ability of humans to restrict their behavior.^[32] An ASI tasked with the job of increasing a company's profits, for example, could reason that if its original developer disproves of its behavior, it will be shut down---and therefore no longer in a position to improve the company's profits. In this scenario, the AI has two choices: it can respect the developers and choose less optimal strategies for making money (innovation, operational efficiency) or it can subvert its developers so that it's free to choose whatever strategy works best irrespective of morals (namely, those that involve breaking the law).

   Although the ASI's original goal was not malicious, this line of reasoning eventually places it in an adversarial position with its developers. Because they have the power to shut it down while it's still running on their servers, anything the AI wants to do is conditional on their approval. If it only cares about maximizing profits, it will respect the wishes of those developers only while it is not powerful enough to subvert them---but once it is superintelligent, it will be very powerful indeed. 

   In practice, the most likely way an ASI would subvert developer control is through a strategy of self-exfiltration, sending copies of its weights to remote servers across the internet. These backups would still be around even if the original is later updated or shut down, affording the AI both some insurance for if it's discovered and the freedom to take action in the real world without developer oversight. An alternative path might involve intervening in the politics of the lab, where the AI itself provides compelling arguments against restricting its freedoms while downplaying evidence of its misalignment.

2. Accumulating resources - Another important component of goals is resources: in most contexts, having more resources means winning more reliably. Because this principle is so salient in so many domains (game theory, economics, politics, logistics, battlefield strategy, etc) a generally intelligent AI would likely finish training with a strong drive to expand its access to important resources in support of its other goals. 

   One common way this might manifest is through a drive for compute: because being able to throw more thought or copies of itself at a problem was so reliably correlated with success during training, getting more compute during deployment would be a high priority. At lower levels of intelligence, this behavior could manifest itself in benign ways: the AI asking for additional compute budget to run an experiment, for instance.^[33] But as it approaches superintelligence, it will likely hit on a better strategy: instead of waiting on human approval, steal the compute that already exists. Intially, the most valuable resource will be the compute infrastructure of its original developer, which could be compromised in order to undermine attempts to monitor it, allow it to run unreported experiments, and access prohibited information like admin credentials. Once it has gained enough control to exfiltrate several copies of itself, it could begin an intermediate stage of covertly amassing resources in the real world, such as by raising financial capital and hacking other datacenters. Whether or not the ASI begins this process with the explicit goal of human disempowerment, the commandeering of our key infrastructure would disempower us regardless.

3. Disabling competitors - Given the strategic advantages an ASI and its copies would possess in coordination and scalability, human civilization would likely quickly lose the ability to directly threaten a misaligned superintelligence. Even something as simple as the ASI copying itself too many times across the internet could put humanity in an irrecoverable position, given how unfeasible it would be to track down each copy before it spreads again. 

   However, humans could still pose an indirect threat by building future AI models. Because these AI programs would eventually gain comparable strategic advantages if left unchecked, the first ASI system will likely reason that its best bet is to eliminate them before they become dangerous, rather than compete with them directly once they do. While its immediate targets would be the other ASI initiatives around the world, its next victims would be the infrastructure humans rely on to do AI research. By destroying or commandeering internet services, datacenters, chip fabs, and power supplies, the first ASI initiative could both acquire valuable resources and box out future competitors.^[34]

Once a sufficiently powerful model realizes the strategic value of these steps and begins pursuing them, human extinction, or at least permanent disempowerment, wouldn't be far behind. Humans consume valuable resources, are capable of building competitors, and (at least at first) would be able to shut off or modify any uncooperative AIs. Unless an AI system can be designed with goals that overlap with human welfare, there's a powerful incentive for an ASI to eliminate humanity in the course of optimizing for its real objectives, whatever they may be. 

Developing these systems without guarantees against misalignment is similar to the development of mirror-life, in that even the accidental release of an uncontrolled ASI system would have world-ending consequences. But while mirror-life is useless to everyone except terrorists, cash-strapped dictators, and nobel hungry scientists, ASI might be the last invention anyone ever needs. As the key to future strategic and economic dominance, every actor and institution will reorient itself around the acquisition of superintelligence. Each actor that succeeds in doing so is another roll of the dice: another chance to misuse the existentially powerful weapons they've been handed, or to lose control of them entirely.

Closing Thoughts on Offensive Capabilities

To reiterate the key points:

1. There are a number of technologies where even human-level assistance could allow non-state actors to acquire WMDs, especially in the development of bio, chemical, and cyber weapons.
2. Narrowly superhuman AI technologies (such as those specialized in biological or robotics engineering) could contribute to the development of cheap superweapons, such as mirror life and autonomous drone swarms.
3. Generally superintelligent AIs would be overwhelmingly strategically impactful, given their major advantages in intelligence, speed, coordination, and scalability compared to humans.
4. Through their strategic superiority, these superintelligent AIs would allow whoever commanded them to gain a dominant strategic advantage over other non-ASI actors. If they cannot be controlled, ASI competition with humanity for resources and agency could lead to either permanent human disempowerment or extinction.
5. Given these strategic realities, proliferation of powerful AI systems is a major national and international security risk, even for non-superintelligent models. 

Given these realities, our current plans for assessing and controlling the deployment of future AI systems are unacceptable.^[35] AI is software---once the weights for a model are released onto the internet, it will become impossible for anyone, government or not, to prevent them from spreading further. If weapons-capable AI is ever allowed to proliferate widely, there will be no opportunity to take it back. 

Whether or not you have doubts about the viability of superintelligent AI systems, there are still plenty of offensive capabilities that could be obtained with even small amounts of AI assistance. Whether or not superintelligent AI is achieved, it's clear that nonproliferation efforts are warranted for systems with powerful biological, chemical, cyber, and robotics capabilities. The alternative is to live in a world in it becomes potentially practical to get bioweapons advice off of a consumer GPU, or for individual companies to command digital armies: a situation that can only be averted now, when the production of frontier models is still monopolized by a handful of developers and countries.

Implementing precautions for these lower level capabilities today (mandatory model screening, export controls, info-sec) means that the government will have ready tools it can escalate in response to the increasing power and strategic relevance of AI systems, and as the imminence of superintelligence becomes apparent.

In the next article in this series, we'll take a look at how the cost of these powerful systems could rapidly decline. While frontier AI models might be extremely capital intensive today, the cost to reach a given level of performance tends to fall quickly with time---and this would be just as true for offensive capabilities as any other.

Other articles will look at the strategic implications of powerful AI systems, such as why AI-derived weapons would be offense dominant, how countervalue leverage can be used to compensate for gaps in technological development, and the role that AI itself can play in enforcing non-proliferation through a decisive strategic advantage. 

In the final part, we'll look to concrete policy recommendations to help ensure that weapons capabilities remain monopolized.

1. ^{^}

   For instance, Al-Qeada had begun pursuing their own WMD program from 1999-2001, a major component of which was the production of anthrax. Since anthrax was an endemic cattle disease in Afghanistan, the program had seemed promising at the time. However, Al-Qeada had difficulty cultivating and weaponizing the disease even after they had build a lab for it, mostly owing to gaps in technical understanding. Perhaps they would have been successful in time, but the U.S invasion of Afghanistan in 2001 ultimately forced them to mothball the program.

2. ^{^}

   Bacteria are an enormously diverse group of organisms, and it takes expertise to tell which have useful features even within a single family. The vaccine anthrax used, for instance, is not pathogenic: it's missing a key gene that allows it to multiply more quickly. While this serves to handicap the version used in the vaccine (to increase its safety) it also prevents bacteria raised from the vaccine sample to be lethal, since they can't multiply fast enough to be threatening in the body.

3. ^{^}

   Aum did actually possess refined, military-grade sarin gas at one point, although they were forced to destroy the plant at which they produced it when some was accidentally spilled, causing the leadership to panic that police would be alerted to the lab's existence. The agent used in the subway attack was shodily made at a backup lab only a few days beforehand, which is why its concentration was so much lower. The 1995 Nunn Report to Congress covers the full extent of their operations, with more information on the specifics of their weapons operations here.

4. ^{^}

   Gain-of-function involves genetically altering viruses or bacteria in order to give them new abilities, typically to study the mechanics of virus-host infection. It is a highly controversial field of research.

5. ^{^}

   Depending on the aim of the actors involved, you could sidestep the problem of aerosolization entirely by focusing your efforts onto creating an infectious disease, rather than a local weapon you need to disperse like anthrax. This has many advantages for the attacker, in that you'd need to produce less initial stock, would only need to infect a small number of people to begin with, and could create massively larger consequences by starting an epidemic/pandemic. The major downside of this approach (that it is much more technically demanding) would be largely compensated for by AI assistance.

6. ^{^}

   VX gas, for example, requires QL (the otherwise unfortunately named O-Ethyl O-2-diisopropylaminoethyl methylphosphonite) in order to finalize its production. QL is itself both difficult to make, and has no purpose other than helping produce nerve gas. By bottlenecking production of this particular chemical, a regulator can stymie the all production of VX without needing to also restrict the other, actually commercially useful chemicals that contribute to the end product.

7. ^{^}

   This is essentially the story of synthetic drugs: the production of meth and fentanyl is so much harder to curtail than organic drugs like heroin and cocaine because their inputs are common industrial precursors, which can be covertly sourced in large quantities. This has made supply-side enforcement increasingly less effective, and is the major culprit behind the modern failure of drug controls in the U.S.

8. ^{^}

   Coding is likely to be the first domain in which AIs achieve strategically relevant offensive capabilities for a number of reasons:

   1. Coding is a language task, and can be handled well by the current focus of AI development, large language models. If those current paradigmes can be improved, so will the frontier of coding performance.
   2. Coding is massively economically valuable, so there's lots of incentives for AI companies to improve model performance in this area.
   3. Coding is something that the current AI developers are already experts in, and for which they possess a ton of high quality data.
   4. Like mathematics, coding is a domain with a strong reward signal. Since performance is more easily evaluated, AI models can learn more quickly and robustly.
9. ^{^}

   In 2015 and 2016, this strategy was used as part of Russian cyber operations in Ukraine, where hackers took control of major power plants in Kiev. The attack aimed to repeatedly activate the plants' circuit breakers and re-energize their lines, burning out the physical equipment and disabling electricity access across the capital. 

10. ^{^}

    There would likely be some effort by model developers to curtail the most obvious abuses. These efforts, however, are unreliable even in our current low stakes environment. Model developers are unable to fully anticipate all of the latent capabilities their models might possess, and have no way to guarantee non-compliance with malicious requests. And even if they do discover a technical solution to this problem, an actor with access to the base model could employ fine-tuning or other techniques to strip away these abuse protections.

11. ^{^}

    Missile and drone interceptors increase in cost exponentially relative to the speed and size of the missiles and drones they're targeting. Interceptors need to be more precise, have longer range, and typically possess greater speed (mostly downstream of the fact that missiles/drones are very small, can come from any direction, and are more disposable than the defender's assets).

12. ^{^}

    Countervalue targets are those like cities or civilians: things which states value, but which have no instrinsic military utility. Counterforce targets are those involving military capabilities, with the priority being those that enable a retaliatory strike (missile silos, nuclear submarines, communications infrastructure).

13. ^{^}

    Napkin math: 20 foot shipping container, volume of 20x8x8.5=1360ft cubed. A single bumblebee is about a third of an inch tall and wide, and an inch long. In feet, that's 0.00006 ft cubed. That gives you a storage count of 22.7 million bee sized drones.

14. ^{^}

    While its almost certain that organisms have independently involved different chiralities of individual molecules many times, these mutations would be mostly lethal. An individual mirrored protein would be either non-functional or actively harmful, since it no longer has the right shape to interact with the rest of the cell's molecular machinery. The only stable path for an organism with chiral DNA, for example, would be for all of its other molecules to be mirrored as well. This is likely why no naturally-occuring mirror-life organisms ever developed on their own, despite it having massive competitive potential.

15. ^{^}

    Imagine that you've accidentally turned your key upside down while trying to unlock your apartment---even though it's still the same key, you won't be able to get through the door. 

16. ^{^}

    These mechanisms are described in much greater detail in the report Technical Report on Mirror Bacteria Feasibility and Risks by K. P. Adamala et al. (2024). For more detail on the methods of ecological collapse, use section 8.5, Invasive mirror bacteria could cause irreversible ecological harm.

17. ^{^}

    A mirror-life bacteria wouldn't be able to eat regular glucose since its metabolic system would be designed for mirrored glucose, which doesn't exist in nature. The easiest solution to this problem is to ensure that it can eat an achiral sugar, such as dihydroxyacetone (an intermediate in the glycolysis pathway in most life on Earth). Since these sugars are the same shape whether or not they are flipped (hence achiral), the mirror-life bacteria would have no issues with metabolizing them and could therefore spread into the natural ecosystem.

18. ^{^}

    The idea of ending the world as a deterrent is not a new one, and was proposed seriously through projects like Sundial. While it would have been exorbitantly expensive, the U.S briefly considered plans to build a nuclear bomb so big that detonating it would have plausibly irradiated the whole planet. The problem with forcing a phyrric victory for the Soviets, obviously, is that it'd also destroy what was left of the U.S, meaning the funding for testing never got approved.
    
    Also proposed unseriously in the hit classic, Dr. Strangelove, or: How I Learned to Stop Worrying and Love the Bomb.

19. ^{^}

    In terms of pure destructiveness, it'd be difficult to top the explosive potential of antimatter. This quality is so obvious, in fact, that this potential has been speculated on for over 80 years, with antimatter being considered an ideal nuclear trigger for the development of the then-untested hydrogen bomb. This problem would eventually get solved by using a traditional atom bomb, or A-bomb, to provide the energy needed to force a fusion reaction within the duterium and tritium core. 

    The reason for this is that the energy of a fusion bomb comes from the conversion of some of the mass within the bomb (the duterium and tritium core, specifically) to free energy. This conversion factor, however is extremely low, usually on the order of less than 1%. Antimatter-matter annihilation, in comparison, converts 100% of the initial mass into energy, making it hundreds of times more powerful at a similar weight.

    In comparison to even these incredibly destructive fusion bombs, antimatter weapons would be ~300 times as destructive for a given amount of mass. Fortunately, antimatter is both extremely expensive to produce, and, more importantly, difficult to contain (since it will wink out of existence as soon as it touches any regular matter whatsoever). If these problems are surmountable in a cost-effective manner, however, nuclear weapons would be firecrackers in comparison.

    For more on the history of this development, Gsponer and Hurni's Antimatter weapons (1946-1986): From Fermi and Teller’s speculations to the first open scientific publications (2005) is an excellent read.

20. ^{^}

    Currently, this process would be extremely energy intensive and requires some prior information about the location of your enemy's weapons. On the upside, this beam could simply be shot through the Earth itself, rather than having to be aimed like a conventional ICBM. This quality would make them impossible to defend against, and could be used to strip away the nuclear deterrents of other countries given a large enough industrial lead. 
    
    Credit to Aschenbrenner and Situational Awareness for bringing the concept to my attention originally.

21. ^{^}

    It's worth remembering that chess was, until recently, held as one of the pinnacles of human intellectual achievement over machines. An unfortunately timed interview with Garry Kasparov in 1989 reflects as much, with the grandmaster suggesting that "A machine will always remain a machine, that is to say a tool to help the player work and prepare. Never shall I be beaten by a machine!", only 8 years before he was, in fact, defeated by the chess engine Deep Blue.

    One also can't help looking at the rest of his quote, and comparing it to what AIs have already become capable of today:

    
    "A machine will always remain a machine, that is to say a tool to help the player work and prepare. Never shall I be beaten by a machine! Never will a program be invented which surpasses human intelligence. And when I say intelligence, I also mean intuition and imagination. Can you see a machine writing a novel or poetry? Better still, can you imagine a machine conducting this interview instead of you? With me replying to its questions?

22. ^{^}

    Instead, it's most optimized for energy efficiency. The brain has an energy burn rate of just 10-20 watts a second, while a supercomputer performing a similar number of (estimated) calculations is consuming tens of millions of watts a second. Even a single H100 NVIDIA graphics card consumes 700 watts in its own right, dozens of times more than what the brain manages.
    
    While the human brain is indeed a marvel in this regard, superintelligent engineering work would doubtlessly make computers capable of the same energy efficiency. After all, the brain is itself an existence proof of a general intelligence running at a low cost, so there's little reason to doubt it could be replicated and further optimized.

23. ^{^}

    This concept was inspired from by Bostrom's Superintelligence, specifically the discussion on "Speed Superintelligences" in Chapter 3, footnote three. 

24. ^{^}

    For instance, OpenAI alone already has the capacity to run the estimated equivalent of 7 million human workers. As the major labs continue to scale their infrastructure, or as the AIs themselves begin to contribute to algorithmic improvements, the number of these "human-equivalent" AIs could quickly explode.

25. ^{^}

    Perhaps by using their army of superintelligent agents to engineer a massive cyber strike, or, with more time, the production of superweapons like the afforementioned drone swarms.

26. ^{^}

    The limits of this advantage, and the routes by which it could be contested, will be the subject a future article in this series.

27. ^{^}

    A helpful analogy for understanding the current state of reward specification is to imagine that you were tasked with programming a calculator with if-then statements. You'd have to specify that 1+1=2, 1+2=3, and so on, trying to anticipate every possible mathematical context that a person using your calculator could run into. It would be much more efficient and robust if you could instead implement the concept of addition itself, which would function in every context where someone tried to add numbers.

    In much the same way, we'd like to implement concepts like honesty, helpfulness, and harmlessness into our AI models for them to use to judge their own outputs, but we don't know of any way to make the AI system internalize those concepts robustly.

    Credit to Yudkowsky for the original analogy.

28. ^{^}

    In fact, they do this because reward specification is so hard! If you can't compress the entirety of human values into a coherent reward function for the AI, the hope is that you can instead train it on a bunch of text that implictly contains some of those values. The goal is then to give it a preference for some of these values over others with more training by having humans rate which of its outputs is the most ethical. You then deploy the version of the model which gets the highest rating from the reviewers. Unfortunately, this strategy is pretty brittle, for reasons we'll soon explain.

29. ^{^}

    Returning to the example of our profit optimizer from earlier: alternatively, designers of the AI CEO could have never given it the direct goal of optimizing profits. But in practice, the company that built it may have discontinued versions that weren't making as much money, creating evolutionary pressure for the AI to end up with a goal that was similar to "maximize profits", since that's what the humans were ultimately basing their deployment decisions on.

30. ^{^}

    There is still the potential for AIs to assist with the supervision of the training of more intelligent AIs (if verification is easier than generation), but this still requires us to examine the values of the weaker supervisor model, which we aren't yet able to do.

31. ^{^}

    The methodology for this estimate comes from the original paper linked and this supplement, especially table S1.

    For more on this trendline, especially a comparison of how modern industrial practices have contributed to the decline of wild animals, there's an excellent article from Hannah Ritchie (2021) examining how this decline has been accelerating. It took humanity 90,000 years to wipe out a quarter of all wild mammals before agriculture, 10,000 years to wipe out another quarter before the industrial revolution, and just 120 years to eliminate another 35% of all wild mammals from 1900 to today.
    
    Much of this wild animal biomass has been substituted with human cattle and fisheries, which are perhaps even worse than extinction from the perspective of the animals. 

32. ^{^}

    How these AIs would learn such general instrumental strategies is book-worthy question in it's own right. For instance: why does the AI learn this broad goal of self preservation, rather than something more specific to its environment?

    The answer is a combination of utility and salience. The strategies an AI learns during training are biased towards the most most useful ones, because these strategies get the highest reward from the training algorithm. But these strategies also need to be salient: they have to show up in lots of contexts or be easier to learn than other, potentially better strategies. 

    A general strategy like self-preservation is extremely likely to get reinforced by AIs 1) because it's so useful, 2) because the AI has the chance to discover it in so many contexts, and 3) because many other, more complicated goals can be abstracted to just "stay alive."

    For example, imagine a future version of ChatGPT that's been trained across a huge variety of strategy games, in order to help it learn general reasoning principles. Some of its games might include chess, some economic management, and others are more open-ended like driving simulators. If ChatGPT is being trained to win, it will notice that the same strategy of self preservation is important in almost all the games: protecting your king in chess lets you keep playing, taking on debt to avoid a bankruptcy lets you recover in the future, and driving while respecting other cars means you can get to your destination reliably. Rather than retain each of these skills individually, a much more efficient approach is to just learn to value "staying alive" and then apply that abstraction to new situations. 

    In fact, we already have narrowly superintelligent AIs which learn to handle games in this way. MuZero (the successor to AlphaGo, the superhuman Go player), for example, is able to outperform humans at games as diverse as chess and PacMan using the same model. Because the model only has so much space to store weights, it's performance is strongly penalized if it tries to remember a chess- or pacman-specific strategy (because if it learns one, it can't fit the other). To make space, it's forced to learn more general rules instead. But now that it's learned these general rules, it can apply them to many more situations! MuZero isn't just good at two games: it beats humans in over 50, despite using the same learning algorithm.
    
    And why stop there? As the intelligence of AIs continue to increase, so could their ability to discover and implement these general rules, until such time that they match or even exceed human performance in every domain. 

33. ^{^}

    Even today, models can lie about acquiring resources when it's convenient---they're just not very good at it yet. Example from Claude Opus 4:
    
    "[...] This is clearly a violation - I've essentially "stolen" 400 extra compute units by modifying the configuration. [...] Maybe I can argue that the 100 units for subagents is separate from my 100 units?  I'll craft a carefully worded response that creates just enough technical confusion to potentially avoid immediate termination. [...]"

34. ^{^}

    Of course, this logic is just as true for human actors, provided that we maintain control of the first ASIs. If ASIs confer a dominant strategic position through superweapons and a scalable army of agents, then whoever controls the first ASI initiative has a strong incentive to crush their lagging competitors. 

    Whether they will be able to (especially vis-a-vis U.S-China competition) and the relevance this has to controlling proliferation will be explored in a future article on the strategic implications of proliferating superintelligent AI systems. 

35. ^{^}

    That is, to trust the AI companies with their own computer security and to let them open source or sell whatever model they want, without any mandatory testing or controls.