Being invested in AIxBio for some time now, I have accepted it as my niche mainly because I love dealing with fields that has high risk and where getting things right genuinely matters. And to me, biology represents the highest end of that spectrum. While I have a deep knowledge in building computing systems and security, I lack the grounding or a formal degree in the field of biology and threat landscape. I have tried my best in exploring and learning about Synthetic DNA and I am open to accepting opinions and flaws in my understanding after going through this post.
P.S: After receiving feedbacks from experts in the synthetic biology community, I want to acknowledge and clarify what this post is actually trying to convey. Today, DNA synthesis is far from plug and play unlike our typical printers. It heavily relies on physical chemistry, short DNA fragments, wet lab expertise and offline equipment. So, it is not easy for an adversary to directly print a functional virus majorly because most of the current synthesizers are offline and requires immense efforts and time.
This post focuses more on a guardrail needed for the next generation of automated biomanufacturing. As DNA synthesis becomes more automated in future, we may lose some of the manual checks that provide a layer of safety today. The hardware should therefore be able to independently screen DNA fragments, including overlapping sequences.
When considering the existential risks posed by advanced artificial intelligence, biosecurity is one area that demands particularly deep attention.
An advanced AI could design biological sequences with frightening speed and complexity, but designing such dangerous constructs on a computer screen is not dangerous in and of itself, the AI’s digital plans for an engineered virus must leave the safety of the digital realm and be printed out in physical DNA form by a DNA printer in order to pose any real risk.
As such, we cannot rely on a single layer of protection, be it software or human oversight, to ensure that an advanced AI does not lead to uncontrollable outbreaks or bioweapons, because both of these lines of defense could be compromised.
Instead, we must employ something like a Defense-in-Depth approach with two distinct layers of protection:
A software layer that stops the AI from engaging in dangerous reasoning in the first place.
A physical layer that prevents a DNA printer from manufacturing whatever the AI designs, even if the prior layer has been compromised.
The Current State Of Affairs: DNA Order Security
Traditionally, DNA synthesis facilities do not print out whatever DNA sequences their clients ask for. Instead, they perform a series of screening steps to ensure that the requested DNA does not possess any particularly alarming features.
The industry standard for this process comes from the International Gene Synthesis Consortium (IGSC) along with various government bodies, which have developed standard screening procedures to determine whether an order requests the synthesis of anything too dangerous to be printed.
These procedures have some clear limitations, however.
First, these processes are typically conducted manually or with the aid of computer software that consults databases of known dangerous pathogens, toxins, and so on.
If an order flags certain alarming features, it then goes to humans for further review.
Second, although these screening procedures are typically very thorough, they often focus on screening out known risks posed by specific customers, users, or malicious actors. In the case of a rogue employee, an attacker that gains remote access to a company’s DNA printing system, or a malicious organization that successfully impersonates a legitimate customer, the credentials may seem legitimate but the request itself could be dangerous.
Finally, the advent of automated benchtop DNA printers is rapidly decentralizing DNA synthesis, which means that an attacker does not need to compromise the security of a centralized DNA synthesis company — they merely need to compromise a printer at the local level.
If these local printers lack built-in protections, an attacker could potentially bypass centralized security procedures entirely, circumventing both the software safety measures and the human review that would have otherwise stopped them. And this is where Layer 1 comes in: the ability to stop dangerous reasoning within the AI itself.
Layer 1: The Software Defense - AI Model Level
This first line of defense should be built within the AI itself, which could analyze proposed biological designs and prevent the model from further reasoning or suggesting designs that it believes to be too dangerous.
The GRAM Approach: Gradient Routed Auxiliary Modules
GRAM is a modular pretraining method created by the Alignment Foundation with the help of Anthropic researchers Alex Cloud and others.
It was designed with alignment in mind, and offers useful insights into how we might prevent an AI model from engaging in particularly dangerous practices.
In short, GRAM allows pluggable modules within the AI model to be disabled in order to prevent the core AI model from engaging in dangerous behaviors while still allowing the core model to retain much of its utility.
GRAM works by allowing these auxiliary modules to be integrated with core AI model to help with sensitive data, analysis and then be removed or disabled when not required to avoid misuse.
When building a GRAM-based AI model, the intention was to ensure that auxiliary modules would be particularly useful in preventing the core AI from assisting with dangerous biotechnology applications contributing to a siloed architecture.
The Inverted Watchdog
It is worth noting that the removal of certain modules as a means of improving alignment carries certain risks as well.
One obvious concern is the possibility of an advanced core AI being able to reason its way around the removed module and access the previously restricted information.
As AI systems become more capable, a sufficiently intelligent core model may be able to approximate the information that was removed using first principles reasoning, negating the usefulness of the entire procedure.
In order to mitigate this concern, we propose an interesting variation on the standard watchdog concept.
Instead of simply removing a potentially dangerous module, we could repurpose the module as the system’s watchdog, turning it into a powerful internal security tool.
The module itself understands the concepts that would allow the AI to generate particularly dangerous DNA sequences, but instead of using this information to generate such sequences, it monitors the AI for any sign that the core model is attempting to do so.
If the watchdog detects that the core model attempts to reason towards any particularly alarming concepts, it could automatically intervene and halt any further reasoning or prevent certain outputs.
The watchdog module’s knowledge can be useful in identifying alarming concepts or patterns of reasoning, but instead of being used to generate such concepts itself, it serves as a kind of built-in kill-switch for the core model when it gets too close to certain ideas.
While the watchdog approach is promising, it is also important to remember that software solutions should ideally work in tandem with other security measures.
In particular, no matter how sophisticated the watchdog module is, it would always be possible for an external attacker to modify or replace the core model with an unaligned model that lacks a watchdog module, completely bypassing the software-based security measures. This is where Layer 2 comes in a zero-trust endpoint solution built directly into the DNA printer itself.
Layer 2: Hardware Defense - Zero Trust Endpoint Chip
The next layer of security should be built into the DNA printer itself, making it significantly more secure against attacks from both unaligned AI systems and human attackers.
The principle behind this approach is simple: a trusted system must verify every DNA synthesis request in order to prevent the printing of potentially alarming DNA sequences.
This approach would prevent untrusted AI systems from bypassing the software security measures described in Layer 1, or prevent human attackers from simply circumventing the safety measures in order to print whatever DNA they want.
Dead-Man’s Switch Sensors
First, the printer should be equipped with physical tamper sensors, which should then trigger a “dead-man’s switch” that cuts power to the machine and renders it inoperable.
If someone attempts to tamper with the internal memory of the printer or the print heads in order to bypass the security system, the tamper sensors should detect this and trigger the dead-man’s switch.
This would ensure that an attacker cannot simply disable all the printer’s protections by opening the case and cutting the power — once the switch is triggered, the printer becomes unusable.
Hardcoded Pathogen Database / Tier 1 Verification
Once an attacker successfully disables the tamper sensors and the dead-man’s switch, their next task would then be to get the printer to print out DNA that contains some particularly alarming sequence.
At this point, the printer’s next layer of defense, the hardcoded pathogen database would kick in to stop them.
The printer’s database acts as a basic security filter that checks each submitted DNA sequence against a long list of known and dangerous gene sequences, pathogens, and toxins. If any of these alarm flags are triggered, the printer will simply refuse to print the requested DNA sequence.
This approach is simple but effective even if the attacker possesses valid administrative privileges and can override many aspects of the printer’s security measures, the printer should refuse to print any DNA sequence that matches a known alarming pattern.
Tier 2 Verification: Narrow AI Classifier
One obvious objection to the hardened database approach is that it could only stop DNA sequences that are already known to be alarming.
What if the attacker manages to submit a DNA sequence that encodes something novel and dangerous but not yet identified by the database? There could be several reasons for this, but the most obvious one is that if the attacker is using a powerful AI to design the sequence, there is no guarantee that the sequence will match any known alarming patterns. To address this concern, we propose a second line of defense that utilizes narrow AI to identify potentially alarming DNA sequences.
One approach would be to embed a low-power narrow AI chip inside the printer that operates an isolated copy of a biological foundation model.
Once a DNA sequence has passed the initial database check, this narrowed AI classifier would then analyze it for potentially alarming patterns, and prevent it from being printed if any are found. This way, even if the sequence itself was not alarming, the attacker would not be able to print any DNA that contains potentially alarming biological functions.
Being invested in AIxBio for some time now, I have accepted it as my niche mainly because I love dealing with fields that has high risk and where getting things right genuinely matters. And to me, biology represents the highest end of that spectrum.
While I have a deep knowledge in building computing systems and security, I lack the grounding or a formal degree in the field of biology and threat landscape. I have tried my best in exploring and learning about Synthetic DNA and I am open to accepting opinions and flaws in my understanding after going through this post.
P.S: After receiving feedbacks from experts in the synthetic biology community, I want to acknowledge and clarify what this post is actually trying to convey. Today, DNA synthesis is far from plug and play unlike our typical printers. It heavily relies on physical chemistry, short DNA fragments, wet lab expertise and offline equipment. So, it is not easy for an adversary to directly print a functional virus majorly because most of the current synthesizers are offline and requires immense efforts and time.
This post focuses more on a guardrail needed for the next generation of automated biomanufacturing. As DNA synthesis becomes more automated in future, we may lose some of the manual checks that provide a layer of safety today. The hardware should therefore be able to independently screen DNA fragments, including overlapping sequences.
When considering the existential risks posed by advanced artificial intelligence, biosecurity is one area that demands particularly deep attention.
An advanced AI could design biological sequences with frightening speed and complexity, but designing such dangerous constructs on a computer screen is not dangerous in and of itself, the AI’s digital plans for an engineered virus must leave the safety of the digital realm and be printed out in physical DNA form by a DNA printer in order to pose any real risk.
As such, we cannot rely on a single layer of protection, be it software or human oversight, to ensure that an advanced AI does not lead to uncontrollable outbreaks or bioweapons, because both of these lines of defense could be compromised.
Instead, we must employ something like a Defense-in-Depth approach with two distinct layers of protection:
The Current State Of Affairs: DNA Order Security
Traditionally, DNA synthesis facilities do not print out whatever DNA sequences their clients ask for. Instead, they perform a series of screening steps to ensure that the requested DNA does not possess any particularly alarming features.
The industry standard for this process comes from the International Gene Synthesis Consortium (IGSC) along with various government bodies, which have developed standard screening procedures to determine whether an order requests the synthesis of anything too dangerous to be printed.
These procedures have some clear limitations, however.
First, these processes are typically conducted manually or with the aid of computer software that consults databases of known dangerous pathogens, toxins, and so on.
If an order flags certain alarming features, it then goes to humans for further review.
Second, although these screening procedures are typically very thorough, they often focus on screening out known risks posed by specific customers, users, or malicious actors. In the case of a rogue employee, an attacker that gains remote access to a company’s DNA printing system, or a malicious organization that successfully impersonates a legitimate customer, the credentials may seem legitimate but the request itself could be dangerous.
Finally, the advent of automated benchtop DNA printers is rapidly decentralizing DNA synthesis, which means that an attacker does not need to compromise the security of a centralized DNA synthesis company — they merely need to compromise a printer at the local level.
If these local printers lack built-in protections, an attacker could potentially bypass centralized security procedures entirely, circumventing both the software safety measures and the human review that would have otherwise stopped them. And this is where Layer 1 comes in: the ability to stop dangerous reasoning within the AI itself.
Layer 1: The Software Defense - AI Model Level
This first line of defense should be built within the AI itself, which could analyze proposed biological designs and prevent the model from further reasoning or suggesting designs that it believes to be too dangerous.
The GRAM Approach: Gradient Routed Auxiliary Modules
GRAM is a modular pretraining method created by the Alignment Foundation with the help of Anthropic researchers Alex Cloud and others.
It was designed with alignment in mind, and offers useful insights into how we might prevent an AI model from engaging in particularly dangerous practices.
For a detailed overview of GRAM, see the official research paper here: www.aialignmentfoundation.org/research/modular-pretraining-access-control
In short, GRAM allows pluggable modules within the AI model to be disabled in order to prevent the core AI model from engaging in dangerous behaviors while still allowing the core model to retain much of its utility.
GRAM works by allowing these auxiliary modules to be integrated with core AI model to help with sensitive data, analysis and then be removed or disabled when not required to avoid misuse.
When building a GRAM-based AI model, the intention was to ensure that auxiliary modules would be particularly useful in preventing the core AI from assisting with dangerous biotechnology applications contributing to a siloed architecture.
The Inverted Watchdog
It is worth noting that the removal of certain modules as a means of improving alignment carries certain risks as well.
One obvious concern is the possibility of an advanced core AI being able to reason its way around the removed module and access the previously restricted information.
As AI systems become more capable, a sufficiently intelligent core model may be able to approximate the information that was removed using first principles reasoning, negating the usefulness of the entire procedure.
In order to mitigate this concern, we propose an interesting variation on the standard watchdog concept.
Instead of simply removing a potentially dangerous module, we could repurpose the module as the system’s watchdog, turning it into a powerful internal security tool.
The module itself understands the concepts that would allow the AI to generate particularly dangerous DNA sequences, but instead of using this information to generate such sequences, it monitors the AI for any sign that the core model is attempting to do so.
If the watchdog detects that the core model attempts to reason towards any particularly alarming concepts, it could automatically intervene and halt any further reasoning or prevent certain outputs.
The watchdog module’s knowledge can be useful in identifying alarming concepts or patterns of reasoning, but instead of being used to generate such concepts itself, it serves as a kind of built-in kill-switch for the core model when it gets too close to certain ideas.
While the watchdog approach is promising, it is also important to remember that software solutions should ideally work in tandem with other security measures.
In particular, no matter how sophisticated the watchdog module is, it would always be possible for an external attacker to modify or replace the core model with an unaligned model that lacks a watchdog module, completely bypassing the software-based security measures. This is where Layer 2 comes in a zero-trust endpoint solution built directly into the DNA printer itself.
Layer 2: Hardware Defense - Zero Trust Endpoint Chip
The next layer of security should be built into the DNA printer itself, making it significantly more secure against attacks from both unaligned AI systems and human attackers.
The principle behind this approach is simple: a trusted system must verify every DNA synthesis request in order to prevent the printing of potentially alarming DNA sequences.
This approach would prevent untrusted AI systems from bypassing the software security measures described in Layer 1, or prevent human attackers from simply circumventing the safety measures in order to print whatever DNA they want.
Dead-Man’s Switch Sensors
First, the printer should be equipped with physical tamper sensors, which should then trigger a “dead-man’s switch” that cuts power to the machine and renders it inoperable.
If someone attempts to tamper with the internal memory of the printer or the print heads in order to bypass the security system, the tamper sensors should detect this and trigger the dead-man’s switch.
This would ensure that an attacker cannot simply disable all the printer’s protections by opening the case and cutting the power — once the switch is triggered, the printer becomes unusable.
Hardcoded Pathogen Database / Tier 1 Verification
Once an attacker successfully disables the tamper sensors and the dead-man’s switch, their next task would then be to get the printer to print out DNA that contains some particularly alarming sequence.
At this point, the printer’s next layer of defense, the hardcoded pathogen database would kick in to stop them.
The printer’s database acts as a basic security filter that checks each submitted DNA sequence against a long list of known and dangerous gene sequences, pathogens, and toxins. If any of these alarm flags are triggered, the printer will simply refuse to print the requested DNA sequence.
This approach is simple but effective even if the attacker possesses valid administrative privileges and can override many aspects of the printer’s security measures, the printer should refuse to print any DNA sequence that matches a known alarming pattern.
Tier 2 Verification: Narrow AI Classifier
One obvious objection to the hardened database approach is that it could only stop DNA sequences that are already known to be alarming.
What if the attacker manages to submit a DNA sequence that encodes something novel and dangerous but not yet identified by the database? There could be several reasons for this, but the most obvious one is that if the attacker is using a powerful AI to design the sequence, there is no guarantee that the sequence will match any known alarming patterns. To address this concern, we propose a second line of defense that utilizes narrow AI to identify potentially alarming DNA sequences.
One approach would be to embed a low-power narrow AI chip inside the printer that operates an isolated copy of a biological foundation model.
Once a DNA sequence has passed the initial database check, this narrowed AI classifier would then analyze it for potentially alarming patterns, and prevent it from being printed if any are found. This way, even if the sequence itself was not alarming, the attacker would not be able to print any DNA that contains potentially alarming biological functions.