Review
Recent decades have seen massive amounts of biological and medical data becoming available in digital form. The computerization of lab equipment, digitization of medical records, and advent of cheap DNA sequencing all generate data, which is increasingly collected in large data sets available to researchers.
This bounty of data is driving rapid progress in AI. In combination with increasingly cheap and powerful DNA synthesis and laboratory automation, AI promises to deliver revolutionary advances in medicine via rapid design-build-test cycles. However, the same capabilities that are driving everything from drug discovery to diagnostic techniques could also revolutionize bioterrorism, with plausible scenarios ranging from a flu-level virus to a pandemic with an impact exceeding Covid.
Fortunately, there is much that can be done to mitigate the risk of an AI-assisted pandemic. This is often framed as a conflict between progress and safety, but that needn’t be the case. Many mitigations are simply sensible public health measures, such as improved ventilation in public spaces. Where it is necessary to manage the development of AI capabilities, this can be done in a targeted manner. We have the opportunity to simultaneously improve public health, reduce the risk of pandemics, and smooth the path for progress.
There are multiple paths through which advances in AI could lead to the deliberate release of a harmful virus. One scenario hinges on the fact, not widely recognized, that the genetic sequences for tens of thousands of human viruses (presenting varying degrees of danger) are already widely available.[1] Advances in DNA/RNA synthesis make it theoretically possible for a disturbed or radicalized individual to recreate a virus; the greatest barrier is knowledge of lab techniques.[2] For many viruses, that barrier is surprisingly low. If the individual had managed to identify a particularly dangerous virus, tens or even hundreds of thousands of deaths could result, comparable to the atomic bombs dropped on Hiroshima and Nagasaki.[3]
Future chatbots seem likely to be capable of lowering the bar to such an attack. As models become increasingly “multimodal”, their training data will soon include video, such as university lectures and lab demonstrations. Such systems would not be limited to providing written instructions; they could plausibly use a camera to observe a would-be terrorist’s work and coach them through each step of viral synthesis. Future models (if not mitigated) also seem likely to be able to provide meaningful help in planning attacks, brainstorming everything from general planning, to obtaining equipment, to applying published research toward creating more-hazardous viruses, to where and how to release a virus to cause maximum impact.
It is sometimes suggested that these systems won’t make a meaningful difference, because the information they are trained on is already public. However, the runaway success of chatbots stems from their ability to surface the right information at the right time. Google can provide a list of ten web pages that are loosely related to a desired topic; ChatGPT can interpolate between vast amounts of training data to provide precisely the information a user needs, even if they don’t properly know how to ask for it. For instance, a multimodal AI might be able to point out to a would-be bioterrorist that they are committing an error in lab technique that would contaminate their sample and ruin their attempt at viral synthesis.
Developments in biology could further raise the stakes. Ongoing gain-of-function research could yield the genome for an unusually dangerous virus; or field work could uncover a dangerous animal virus. Alternatively, progressing in characterizing viruses’ pandemic potential might eventually allow an attacker to select the most dangerous of the existing public genomes for synthesis.
Finally, we must consider the possibility that future specialized biotechnology AIs will be able to support the design of proteins having specific desired behaviors in the body.[4] This possibility underlies much of the hoped-for potential of AI to revolutionize medicine, but if protein design tools eventually progress as far as some proponents envision, they could be used to increase the transmissibility or lethality of a virus. This could abet the creation of a “supervirus” – say, combining the rapid spread of measles, the fatality rate of smallpox, and/or the incubation period of HIV.[5] The creator of such a virus could then release it under circumstances that allow it to disperse widely before first being detected. It is plausible that the impact would be so severe that people might be afraid to leave their homes, and essential workers might be unable or unwilling to remain on the job, conceivably leading to a breakdown of civilization.
The danger rests on the fact that modern society is highly vulnerable to respiratory viruses in general. Endemic viruses such as flu, RSV, and SARS-CoV-2[6] cause over half a million deaths per year. Actions which make it more difficult for viruses to propagate will yield health and economic benefits today, in addition to reducing the risk of bioterrorism.
Improvements to air ventilation, filtration, and germicidal ultraviolet lighting make it more difficult for respiratory viruses to travel from one person to another. UV lighting in particular is an area of active research with considerable potential to reduce viral circulation.[7]
Development of broad-spectrum vaccines and antivirals[8] could reduce the impact of common viral families such as flu and coronaviruses.[9] This would reduce the potential for a bad actor to leverage the extensive genetic sequences and knowledge base around these families.
Early-detection and rapid-response capabilities can reduce the impact of both engineered and natural pandemics. The Covid pandemic killed over six million people,[10] and the economic impact is measured in trillions of dollars; some actions to reduce the potential for another pandemic could be carried out at comparatively low cost. The next Covid might be a century away, or Patient Zero might already be developing symptoms today.
Aggressive monitoring for novel viruses could detect new viruses before they are able to spread widely. Wastewater monitoring (especially targeting airports and other travel hubs)[11] may be particularly effective, but should be combined with other measures, as not all viruses present in wastewater.
Build the capability to very quickly manufacture and distribute test kits once a new virus has been identified. Aggressive testing around early cases can help prevent a virus from establishing itself.
Develop improved PPE[12] for airborne viruses, targeting cost/durability, effectiveness, and practicality/comfort. Stockpile PPE for rapid deployment to essential workers.
Further accelerate our ability to rapidly create, test, manufacture, and distribute a vaccine for a novel virus.[13]
As AIs become more powerful, we will need to carefully manage access to tools that could assist in causing harm, as well as information regarding dangerous viral genomes.[14] Restrictions can be designed to minimize impact on legitimate research, but in some cases there will be tradeoffs to be made.
Monitor patterns of behavior in users of biotech tools, to identify individuals who may be attempting to create a dangerous virus.[15] Encourage reporting of suspicious behavior.[16]
Limit access to tools needed for creating, synthesizing, or testing a virus,[17] such as specialized AI models (e.g. protein design tools), DNA / RNA synthesis equipment, and other specialized equipment. Measures should include “know your customer” requirements, tracking of equipment over time, and comprehensive screening of synthesized DNA / RNA sequences. If and when protein design tools become capable of enabling the creation of novel viruses, screening will need to be expanded to detect such novel viruses (a potentially difficult problem).[18]
Develop techniques for detecting “warning shots”. A failed attempt at engineering a pandemic might sicken a small number of people (perhaps the perpetrator). Techniques for identifying novel, suspicious viruses could allow us to head off a successful attack.
Exclude certain categories of biological knowledge[19] from chatbots and other widely accessible AIs, so as to prevent them from coaching a malicious actor through the creation of a virus. Access to AIs with hazardous knowledge should be restricted to vetted researchers.[20]
Evaluations and red-teaming to prevent the release of AI models that can assist with the synthesis and release of a virus.[21] Training an AI to assist with the development of new viruses will likely require assembling large amounts of data regarding the behavior of viruses in the body, so the development of such data sets should be monitored.
Limit (to the legitimate research community) access to genetic sequences or other specific information which would identify a specific pathogen as potentially capable of generating a pandemic, and facilitating the synthesis of that pathogen.[22]
Limit research into techniques for evaluating the potential for viruses to cause harm in humans, particularly with regard to transmissibility. Especially limit the open publication of such research.
Apply rigorous risk-benefit analysis to viral gain-of-function research, including the decision of whether to openly publish the results. This analysis should take into account anticipated developments in synthesis techniques. For instance, if a virus might plausibly be easy to synthesize using the equipment that will plausibly be available ten years from now, then unrestricted publication of the viral genome today might be considered high risk.
These measures will require identifying legitimate researchers, and restricting access to certain narrow categories of information and tools to people in that circle. Maintaining such restrictions in an effective manner will require new practices in certain scientific and engineering communities.
Restricted access is at odds with the established practices and norms in most scientific fields. Traditionally, the modern academic enterprise is built around openness; the purpose of academic research is to publish, and thus contribute to our collective understanding. Sometimes norms need to adapt to changing circumstances,[23] but this is never easy.
It is worth noting that virology, like many fields, is already under a heavy regulatory burden, designed to protect both research subjects and the eventual consumers of new drugs and procedures. Adding to this burden should not be taken lightly, but in some cases will be necessary. Historically, regulations and norms have not always been designed to provide the maximum protection in return for the minimum impact on research. Meanwhile, the stakes are higher than ever before: the impact of a new pandemic could be vastly greater than that of an improperly approved drug. And if we wait until the first cases of an engineered virus become visible before applying restrictions on research, that will be much too late to head off the resulting pandemic.
The important question is whether or not we succeed in preventing an engineered pandemic, as opposed to merely adhering to regulations. Effective biosecurity will require helping the scientific community to adopt a security mindset, educating them on the principles of security and making them into enthusiastic and active participants.[24] Measures might include:
Advances in AI, DNA synthesis, and laboratory automation promise to revolutionize medicine… but could also open the door to bioterrorism. Through a thoughtful mix of public health measures and targeted management of access to advanced capabilities, we can not only manage this risk, but also reduce the ongoing burden of natural disease.
Guest author Steve Newman, a co-founder of eight startups including Google Docs (née Writely), is now searching for ways to reduce the tension between progress and safety by building a more robust world. His blog is Am I Stronger Yet?
Thanks to Aidan O’Gara, Dan Hendrycks, Geetha Jeyapragasan, Gigi Gronvall, Lennart Justen, Mantas Mazeika, Nikki Teran, Rahul Arora, Rocco Casagrande, Sarah Carter, and Thomas Woodside for contributions and feedback. No endorsement is implied.
For instance, see https://www.ncbi.nlm.nih.gov/genomes/GenomesGroup.cgi?taxid=10239.
Access to equipment and materials is also a barrier, but not necessarily enough of a barrier to be reassuring, as for some viral families only fairly basic equipment is needed. For instance, from Biodefense in the Age of Synthetic Biology: “The production of most DNA viruses would be achievable by an individual with relatively common cell culture and virus purification skills and access to basic laboratory equipment, making this scenario feasible with a relatively small organizational footprint.”
The information needed to create a virus includes “tacit knowledge”, i.e. the various tips, tricks, and soft skills which are necessary to successfully carry out lab procedures, but which don’t tend to appear in textbooks. This further increases the potential value of LLMs (especially multimodal LLMs trained on lab videos) to a potential attacker.
Note that by the time AIs are sufficiently capable to assist with lab techniques (including tacit knowledge), they may also be able to provide advice on obtaining equipment, or access to equipment.
Until recent years, the possibility of such “protein design software” seemed relegated to a distant future. However, in 2018, DeepMind announced AlphaFold, a deep learning model that was able to successfully predict the structure of a new protein based only on its genetic sequence – a previously unsolved problem.
Training a deep learning model requires large amounts of data. In the case of AlphaFold, this data consisted of millions of protein structures whose structures had been determined, over the years, through laborious laboratory work. As biological research becomes increasingly digitized and automated, we will amass growing collections of data regarding protein interactions and behavior. Researchers hope to eventually use this data to create models that can do for protein design what AlphaFold did for protein structure prediction.
For one discussion of the potential impact on biosecurity risks, including the potential for LLMs to assist non-experts in using advanced software tools, see Understanding AI-Facilitated Biological Weapon Development.
HIV, while not a respiratory disease, provides an extreme example of how viral incubation periods can be very long, and how this makes the virus difficult to control. Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics notes that Omicron spread to 25% of the USA and 50% of Europe within 100 days; hypothetically, if it were possible to create a virus with this rate of spread but a multi-month incubation period, it could be everywhere before the first symptoms appear.
There is debate as to the potential for a single virus to achieve the civilization-threatening combination of long incubation period, very rapid spread, and very high fatality rate, as well as the potential for AI to allow an individual or small group to create such a virus. However, Covid-19 demonstrates that a realistic virus can have a very high impact in the modern world. If the initial strain had been Delta or Omicron, the impact would likely have been even worse. And there is no reason to believe that these strains represent a theoretical upper limit.
Sadly, four years after the arrival of Covid, the status quo may be the new normal.
Conventional air filtration is relegated to circulation ducts (or sometimes in-room appliances). While further research is needed, some forms of UV light may have the potential to be deployed directly in rooms, either across the ceiling zone or throughout the entire room – attacking viral particles the moment they are released. For instance, from Assessing the safety of new germicidal far-UVC technologies:
Due to its enhanced safety compared to conventional 254 nm upper-room germicidal systems, far-UVC allows for whole-room direct exposure of occupied spaces, potentially offering greater efficacy, since the total room air is constantly treated.
See also Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics. However, further research is needed in both safety and efficacy of far-UVC.
From Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics:
Broad-spectrum vaccines and antivirals that function against entire families of viruses are highly desirable and should be developed and stockpiled if at all possible, but they are also unreliable: any great power, most rogue states, and even unusually competent extremists or zealots are capable of engineering pandemic-class agents to resist or evade publicly known medical countermeasures.
From Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics:
A nucleic acid observatory that performs untargeted metagenomic sequencing of all nucleic acids across relevant human and natural ecosystems would serve as a reliable early warning system... A basic global version would monitor numerous air traffic hubs throughout the world by sequencing wastewater or air filters from aircraft and airports, or possibly clinical samples from flight crews, for as little as tens of millions of dollars a year.
“Personal Protective Equipment”: in this context, PPE refers to masks and other equipment to protect individuals, especially medical workers and others with elevated risk, from viruses.
From Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics:
… investing in rapid nucleic acid vaccine production facilities worldwide, preparing to immediately launch combined Phase 1+2 ring vaccination trials in response to outbreaks, and supporting research into receptor decoys and therapeutic interfering particles capable of slowing the spread of a virus.
Also see 100days.cepi.net.
Standard practice today is generally to publish research findings. Software developed in academic settings, including AI models for protein design, are often made broadly available as well. For information and tools which have the potential to substantially abet bioterrorism, some restrictions will be needed. These should be designed to minimize the impact on legitimate research, but in some cases there will be tradeoffs to be made in order to ensure safety.
AI models could also monitor the use of AI-bio capabilities and identify concerning behavior by users. Several experts were optimistic about the ability of AI to analyze patterns of behavior, such as gathering information from an LLM on specific topics combined with purchasing life sciences products, to identify customers with potentially malicious intent. A similar project has demonstrated the value of this type of monitoring of publicly available data for detecting high-risk or illicit nuclear trade.
From Risk and Benefit Analysis of Gain of Function Research:
Nonpunitive peer reporting of unusual incidents or repeated experimental findings, damaged equipment and facilities, and behavioral changes or unusual behavior of individuals with authorized access to high containment, research laboratories are the only measures that exist to prevent or mitigate a deliberate act carried out by an insider with trusted access.
See, for instance, Protein design meets biosecurity.
In general today, we do not know how to analyze a DNA or RNA sequence and determine whether it encodes for a dangerous virus. However, solving this problem may be less difficult than the problem of designing a dangerous virus in the first place, especially if we allow for a certain rate of “false positives” which would trigger manual review.
…such as papers on dangerous pathogens, or laboratory protocols for constructing and booting viruses.
And also:
A few experts believe that restricting access to specialized or particularly harmful data could help reduce potentially harmful outputs from AI models and could prevent bad actors from training their own models. Experts listed a wide range of data, including, for example, pharmaceutical company databases on protein and chemical toxicity, publicly available pathogen genomes, gain-of-function research, and information related to historical bioweapons programs. … Much of the data described are already publicly and redundantly available on the Internet, and it would be very difficult to prevent some types of models, including LLMs, from accessing such data.
Rather than restricting an AI’s training data to prevent it from learning how to create a virus, bio-capable AIs could be trained to refuse to assist in the creation of a harmful virus. However, such safeguards are easily removed from open-source models, and are not currently robust even for closed-source models.
See, for instance, Rocco Casagrande’s Written statement for Senator Schumer from 12/6/23.
From Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics:
A pandemic test-ban treaty modelled after the Nuclear Test Ban Treaty would explicitly ban the dissemination of results from the handful of experiments capable of substantially increasing our confidence that a natural or synthetic virus can cause a new pandemic. Crucially, blocking these experiments would not impede vaccine or antiviral therapeutics research; they are only useful to assess pandemic capability, and whatever the benefits of targeted spillover prevention efforts may be, they do not appear to outweigh the expected harms of misuse (see Box 1) given that many more pandemic viruses exist in nature than will spill over.
One historical example involves the establishment of guidelines around recombinant DNA research.
For instance, see https://ebrc.org/focus-areas/security/.
Including those designed to evade detection.
This refers to security practices for protecting materials and information from hostile actors, as opposed to safety practices to guard against accidents.
From Concepts to Bolster Biorisk Management:
As biotechnologies continue to advance, a host of outsourcing and automation companies have sprung to life, including fully automated laboratories available remotely to researchers seeking to use their capacities, gene synthesis companies, protein production companies, and many more. Despite the risk of these companies being misused by malicious actors to facilitate acquisition of a harmful agent, there is no federal guidance, advocacy organization, or commercial enterprise focused on improving biosecurity within these industries. Yet industry recognizes the value of improving biosecurity and has asked for help with 2 specific biosecurity efforts: (1) a training program to introduce concepts of biosecurity and (2) a clearinghouse that could provide on-demand and rapid response biosecurity advice when harrowing situations arise.