cross-posted to EAForum

Epistemic status: We are quite confident that Biological AI models (BAIMs) safety requires further work, but uncertain about its scale. The apparent gap may be refuted by one experiment, filled by a few researchers working for a year, or prove to be substantial enough to call for an entire subfield.

Disclaimer: this post has been written with a colleague, that due to her current job can't post out of her own forum account

What we’re looking for: please poke holes in this. In particular, we’d value:

• Arguments against prioritizing this work;
• Important research or organizations we’ve missed;
• Reasons the research questions below would not actually change decisions;
• Information-hazard or capability-externality concerns we’re underweighting;
• Better questions than the ones we propose

The case in brief

Biological AI models (protein, genomic, and single-cell models trained directly on biological data) enable increasingly capable biological design. That carves the path to new vaccines and therapeutics, but also to biorisk scenarios.

These models exist for beneficial scientific or defensive purposes. However, the underlying capabilities could also be applied to harmful objectives. They are currently an important step for designing novel pathogens, and as they improve we expect them to remain a part of the design pipeline.

At the same time, we think we know surprisingly little about the capabilities of these models, especially from a safety perspective.

In particular, we still lack robust, general answers to questions such as:

• Will text-based AI eventually be able to reason over raw genomes, or is biological data too fundamentally different?
• How much training data will biological models need before qualitatively new capabilities emerge, or they “grok”?
• What is the tradeoff between evading sequence-based screening and preserving biological function?
• If a model was never trained on viral sequences, how well will it generalize to viruses from other domains of life?

The answers to these questions imply different biosecurity strategies.

Therefore, our tentative view is that there is a case for substantially more empirical research on biological AI model safety, particularly research designed to inform policy and funding decisions.

Biological AI models are becoming more capable

Biological AI models are models trained directly on biological modalities rather than natural language (proteins, genomes, cells and related data).

Some recent results are striking.

• A model trained only on pre-2020 sequences predicted 66% of the high-frequency SARS-CoV-2 receptor-binding-domain mutations that later emerged
• A protein language model improved antibody affinity up to 37-fold against escaped SARS-CoV-2 variants
• Generative protein-design methods produced variants of 72 proteins of concern that bypassed existing DNA synthesis screening
• Evo2 was used to generate viable bacteriophages: 16 of 285 synthesized designs worked, with several sufficiently divergent to plausibly constitute new species
• AlphaFold demonstrated that AI can infer protein structures with near-experimental accuracy, even for proteins whose structures had never been measured directly

These results demonstrate that already now BAIMs can provide some advantage at making catastrophic biological risks substantially greater. Given the substantial progress we’re currently seeing with AI we suspect these models will get much closer to 100% design accuracy. This seems sufficient to motivate a question:

What can these models actually do, how quickly is that changing, and which interventions would matter if their capabilities continue improving?

We are missing some basic measurements

There is now significantly more work on biological risks from general-purpose AI.

SecureBio has developed VCT, BioTIER and ABC-Bench. Active Site and METR have run an RCT measuring LLM assistance on novice biological work.

There is also growing attention to biological AI models specifically. Epoch AI now catalogues more than a thousand of them. RAND Europe is developing a risk observatory for AI-enabled biological tools relying on literature reviews. NTI | bio and Concordia AI recently launched a working group on evaluation practice.

This is useful progress. But there seems to be less published work directly measuring the security-relevant capabilities of the biological models themselves. We also suspect there is scarce classified work, because multiple classes of these models are nascent, and as a result:

1. There is little non-security evaluation work
2. There are few people with the current skill set, and
3. These people are either still being trained in academia or drawing large salaries in industry

A catalogue can tell us that a model exists, how large it is, whether its weights are available and whether its developer reports safeguards. It cannot necessarily tell us what the model enables.

Similarly, parameter count may be a particularly weak proxy here. Across several classes of biological models, larger models do not consistently outperform smaller ones.

So we think there is a missing empirical layer on what is the risk-management strategy we should adopt.

Research questions that could change what we do

The case for this research is that we believe there are several empirical questions where different answers would point toward different interventions.

Will general-purpose AI subsume biological AI models?

It’s unclear whether increasingly capable general-purpose AI will eventually reason directly over biological sequences, or whether specialized biological models will remain necessary. Biological data, architectures, and scaling behavior differ substantially from text, but we do not know whether those differences will persist.

What drives BAIM performance?

It’s unclear what drives improvements in BAIM capabilities. Relative to text-based AI, biological models seem to have only modest or inconsistent scaling effects. Current experts suspect that this is because these models are more constrained by data than compute. 

How well do capabilities generalize?

A model trained on one set of organisms may acquire capabilities that transfer to others because biological sequences are linked through common ancestry. How far this transfer extends matters for data policy- if pathogen-relevant capability comes from pathogen data, restricting access to some viral datasets could potentially reduce risk (while preserving most biological research). However, if the same capability can be recovered from distant organisms, restricting viral data alone may only accomplish little.

How should BAIM safeguards work?

Many BAIMs are open-weight and commonly fine-tuned, which may make safeguards developed for API-based language models less useful. We therefore think BAIM safety may require a somewhat different flavor. Technical research could help identify which safeguards are most effective and when: model hardening as an additional barrier, tiered access for higher-risk capabilities, sequence screening, data controls, and other downstream safeguards. It can guide how these interventions are best combined and where the possible gaps are. Culture too matters here: unlike in AI safety, most BAIM development is still done in academic labs. Researchers may be more resistant to closed-source models, but they can also be more amenable to instilling a culture of responsibility and adopting safety practices.

Why now?

There are three reasons we think the timing may be unusually important, and it’s important to act fast

1. The field is still early enough to shape- BAIM safety is still nascent relative to BAIM development. We have an opportunity to build and instill safety practices around evaluation, release and access.
2. Several important uncertainties are experimentally tractable- Questions about evolutionary transfer, data scaling or architecture dependence can be tested
3. Policymakers are gearing towards this- Multiple biosecurity-related think tanks, foundations, and research organizations are beginning to assess AI-enabled biological tools. Risk assessment ultimately needs empirical inputs. If we cannot measure capabilities well, we risk basing decisions on proxies such as model size, openness or developer claims.

Reasons we might be wrong

Artificial General Intelligence will eat this problem

Specialized biological models may soon become irrelevant relative to increasingly capable general-purpose systems. If AI models are able to reason over biological sequences the way they are able to reason over text, BAIM safety may have little marginal value over AI safety.

BAIMs may not be the bottleneck

Powerful biological design models may contribute little to global catastrophic biological risk if wet-lab expertise, tacit knowledge, access to equipment, experimentation or other steps remain high barriers to access

Information hazards

This means some research may need restricted dissemination, and some questions may not be worth answering at all. We don’t think the default should be that everything produced by a BAIM-safety research program is published. We think such research should be done carefully and with extreme security practices taken. We also believe in using biological proxies as much as possible.

If you know of relevant work, disagree with our framing, or think these are the wrong questions, please tell us - is there something we are missing?