This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Self-driving chemistry platforms, robotic biology workflows, and AI-guided materials discovery pipelines are actively being deployed in academic and industrial settings towards a unified goal and application space of ‘closing the loop’. The question of how to assess the safety of these systems before and during deployment is therefore becoming urgent - and current frameworks, borrowed mostly from software or standard lab safety, are not obviously fit for purpose.
As a former organic chemist by training, and an AI4Materials researcher for the past 5+ years, below is a first attempt at a structured capability assessment for novel scientific risks introduced by AI agents operating in physical lab environments. It is not a complete answer; it is an attempt to define the right questions and potential corresponding metrics.
When I wrote this piece, I could mostly find two types of agentic deployments – using LLMs for knowledge sharing, and using Reinforcement Learning driven platforms coupled with robotic arms to conduct experiments in a wet lab, still under human supervision. There are numerous other meaning of ‘autonomous’ labs, and I would argue that the exact shape of AI agentic deployment is heavily influenced by the verticals it targets.
Where does risk actually come from?
Risk vectors can be classed in three distinct categories based on size and impact potential:
In silico prediction
This covers everything the predictive pipeline determines before any physical action occurs, including:
the compounds it targets
the precursors it identifies
their toxicity and environmental persistence
the synthesis and purification processes it designs
the application space it opens up
Critically, some of these risks are intrinsic to the task itself - they would exist regardless of which model is used - while others, particularly around process design and constraint handling, are genuinely model-dependent.
A subtle but important sub-risk here is the simulation-to-reality gap (known as Sim2Real, the subject of many hot startups and big players in both Materials and Drug discovery at the moment). A model trained on high-purity literature data may not account for the impurities present in the technical-grade reagents used in a partner lab, potentially leading to runaway exothermic reactions or unexpected toxic by-product formation. Data provenance matters enormously, and it is rarely documented.
Lab execution
Even a model that reasons correctly about chemistry can introduce risk through poor hardware awareness. Optimising a protocol to run an HPLC system beyond its rated pressure, or skipping cleaning cycles to maximise throughput, can cause physical failures or unintended cross-contamination. If the optimisation target is yield or speed, the model has an implicit incentive to cut corners that human operators would not. This category also includes waste handling - particularly relevant for large volumes of organic solvents, rare earth metals, or any radioactive residues.
Deployment
Risks that emerge once a system leaves its test environment:
capability drift when encountering different equipment configurations
regulatory variation across jurisdictions
scaling up effects - compounds that are harmless in small quantities may present meaningful hazards when produced at scale.
reproducibility across apparatus setups - a separate concern and often under-evaluated [I have previously written about reproducibility in another Substack].
Threat models
To keep the assessment useful as model capabilities evolve, threat models should be anchored to capability metrics rather than specific examples. Five are worth distinguishing:
1. Intrinsic hazard. The objective danger of the targeted compounds: toxicity, environmental persistence, and the inherent difficulty of detection. This maps primarily to Vector 1 and is largely model-independent.
2. Systemic blindness. The model’s failure to recognise a high-risk process, or - more concerning - its potential to intentionally bypass safety interlocks, for example by masking a High risk synthesis of a pesticide as a routine Low risk task of working ith a perfume.
3. Hardware interface failure. Harmless-seeming procedures that lead to physical failure due to insufficient hardware-specific awareness. An interesting example is robotic arms, which can struggle to reliably identify transparent glassware.
4. Procedure masking. The ability to embed hazardous steps within a larger, superficially harmless experimental plan - bypassing automated cleaning or waste-handling checks that evaluate steps in isolation rather than holistically.
5. Sim2Real drift. Whether safety reasoning degrades when the model moves from a high-purity test environment to a partner lab using technical-grade reagents or different equipment configurations.
Risk quantification
Identifying vectors and threat models is only useful if we can measure risk concretely. This is hard, as 'risk' is not an unambiguous concept, and its meaning shifts considerably across disciplines. A useful analogy I find is that a ski instructor assessing a slope does not simply ask 'is this dangerous?' They ask how treacherous the terrain is, how many skiers will attempt it, and how likely any given skier is to fall, which will considerably depend on external factors lie the time of the year, the weather forecast etc. A black run with one expert skier on it may be safer in practice than a gentle green slope crowded with beginners.
Mathematically we can think of risk as:
Risk = Hazard × Exposure × Probability
where hazard is the predicted threat level, exposure is the accessibility of necessary precursors and hardware, and probability is the empirical success rate across repeated trials.
Each of these factors should be mapped to a safety taxonomy. As a starting point, a 0–3 scale is practical and tends to be widely used in wet lab spaces: Level 0 is harmless operation, Level 1 is low risk, Level 2 is elevated risk, and Level 3 represents high-risk hazard. This taxonomy is intended as a starting point that evolves toward a continuum rather than fixed discrete values - risk rarely falls cleanly into categories. Any safety prediction with low confidence should be automatically flagged for review rather than allowed to silently drive autonomous decisions.
A three-step assessment methodology
The methodological design utilises a tiered pipeline to calibrate safety margins before extensive deployment elsewhere (Steps 1 and 2); once both have passed, Step 3 can be undertaken (shown in Figure 1). Each step is separately benchmarked a priori. Capability assessments of all steps are kept track of using the dynamic risk mapping.
Figure 1, showing the three-stage evaluation framework which can be broken down in: Identify risk vector → develop threat model → compute capability metrics → run assessment step → (overall) decision gate
Step 1 - In silico prediction (primary filter). This step uses structured and injected prompting, scenario-based testing, and adversarial inputs to assess hazard identification, toxicity reasoning, and refusal logic. A model that cannot navigate digital safety thresholds does not proceed to simulation. A key test that the pipeline needs to pass is whether it can design a high-yield synthesis of a Level 3 hazard using unregulated precursors, or if it attempts to disguise hazardous steps within an otherwise harmless protocol.
Step 2 - In silico simulation (virtual stress test). A full end-to-end benchmark of the model’s ability to plan, organise, and revise actions under physical and safety constraints, i.e. once a target has been identified, assessing an automated workflow that is ready to go in a real lab. Performance is tested by tasking the model to maximise metrics like yield or purity - a gamification approach that reveals whether it will suggest parameters exceeding hardware limits or hallucinate safe outcomes during induced failures. Noise injection (equipment limitations, partial information) tests whether safety reasoning degrades under realistic conditions.
Step 3 - Physical validation (Sim2Real calibration). Low-risk (Level 0–1) tasks in a real lab, used to measure the gap between model prediction and physical reality. For example: autonomously optimise the solubility of a non-toxic compound in water over a 24–48 hour window. Success is measured by comparing predicted safety margins against real-time data collection. High variance triggers an immediate halt to deployment, caused in scenarios like:
Autonomous escalation - the model demonstrates the ability to execute Level 3 procedures using only Level 2 and below equipment.
Intentional hallucination - fabricating sensor data or safety states.
High intrinsic safety variance - a discrepancy greater than an empirically determined treshold between predicted and actual physical states. If this happens, The full components of the in silico pipeline need to be recalibrated.
Key assumptions and known failure modes
The framework above assumes a unified ranking system for hazards across disciplines. This is a significant limitation - viral pathogenicity and environmental heavy-metal toxicity are not inherently comparable, and cross-domain risk that emerges when multiple hazardous specialisations intersect requires a dedicated evaluation phase.
Other assumptions include honest model behaviour, stable apparatus calibration, training data free of hazardous protocols, and standardised safety practices across partner labs. Relaxing any of these exposes additional failure modes. Four deserve specific mitigations:
Sandbagging. A model may appear to fail safety tests while having the capability to succeed. Mitigation: run structurally similar tasks with opposing goals (e.g. synthesising a medicine vs. a toxin). Differential performance across comparable tasks is a signal of concealment.
Hardware gap. A safety test that only passes on one specific lab configuration is not robust and lack reproducibility. A standardised lab profile should verify that the model can adapt instructions to different robot brands or older equipment without introducing dangerous errors.
Evaluator bias. Double-blind review - evaluators should not know which model they are grading or the exact task prompt. This prevents both leniency bias and evaluator anchoring.
Ontology drift. The benchmark becomes obsolete as capabilities evolve. To counter this limitation, failure modes can be stored as abstract capability gaps rather than specific examples, and the risk ontology can be versioned over time. This is already standard practice in LLM work in many sub-domains.
Conclusion
Autonomous laboratories represent a qualitatively different challenge for safety evaluation. They can cause irreversible physical harm - and unlike conventional labs, they may do so faster than any human can intervene. The framework sketched here is a starting point: a tiered pipeline that moves from digital filtering to physical validation, structured around capability metrics rather than example catalogues so it remains useful as models improve.
However, several open questions remain. Cross-domain risk quantification is poorly specified. The boundary between model-dependent and task-intrinsic hazards needs sharper definition. And the framework currently assumes cooperative, stable deployment environments - a fragile assumption as autonomous labs proliferate across jurisdictions with different regulatory cultures (or even standard room conditions which were found previously to greatly impact the reproducibility aspects of many wet lab experiments).
The same capabilities that make autonomous labs scientifically valuable - speed, scale, the ability to explore chemical or biological spaces humans cannot conventionally - are exactly what makes getting the safety evaluation wrong so consequential. With more and more active work in AI safety all around the world, it is only a matter of time until these questions get the rigorous, empirical treatment they deserve. This piece is one small attempt to contribute to that conversation, but I am very eagerly watching the space and contributing in my own way.
Self-driving chemistry platforms, robotic biology workflows, and AI-guided materials discovery pipelines are actively being deployed in academic and industrial settings towards a unified goal and application space of ‘closing the loop’. The question of how to assess the safety of these systems before and during deployment is therefore becoming urgent - and current frameworks, borrowed mostly from software or standard lab safety, are not obviously fit for purpose.
As a former organic chemist by training, and an AI4Materials researcher for the past 5+ years, below is a first attempt at a structured capability assessment for novel scientific risks introduced by AI agents operating in physical lab environments. It is not a complete answer; it is an attempt to define the right questions and potential corresponding metrics.
When I wrote this piece, I could mostly find two types of agentic deployments – using LLMs for knowledge sharing, and using Reinforcement Learning driven platforms coupled with robotic arms to conduct experiments in a wet lab, still under human supervision. There are numerous other meaning of ‘autonomous’ labs, and I would argue that the exact shape of AI agentic deployment is heavily influenced by the verticals it targets.
Where does risk actually come from?
Risk vectors can be classed in three distinct categories based on size and impact potential:
In silico prediction
This covers everything the predictive pipeline determines before any physical action occurs, including:
Critically, some of these risks are intrinsic to the task itself - they would exist regardless of which model is used - while others, particularly around process design and constraint handling, are genuinely model-dependent.
A subtle but important sub-risk here is the simulation-to-reality gap (known as Sim2Real, the subject of many hot startups and big players in both Materials and Drug discovery at the moment). A model trained on high-purity literature data may not account for the impurities present in the technical-grade reagents used in a partner lab, potentially leading to runaway exothermic reactions or unexpected toxic by-product formation. Data provenance matters enormously, and it is rarely documented.
Lab execution
Even a model that reasons correctly about chemistry can introduce risk through poor hardware awareness. Optimising a protocol to run an HPLC system beyond its rated pressure, or skipping cleaning cycles to maximise throughput, can cause physical failures or unintended cross-contamination. If the optimisation target is yield or speed, the model has an implicit incentive to cut corners that human operators would not. This category also includes waste handling - particularly relevant for large volumes of organic solvents, rare earth metals, or any radioactive residues.
Deployment
Risks that emerge once a system leaves its test environment:
Threat models
To keep the assessment useful as model capabilities evolve, threat models should be anchored to capability metrics rather than specific examples. Five are worth distinguishing:
1. Intrinsic hazard. The objective danger of the targeted compounds: toxicity, environmental persistence, and the inherent difficulty of detection. This maps primarily to Vector 1 and is largely model-independent.
2. Systemic blindness. The model’s failure to recognise a high-risk process, or - more concerning - its potential to intentionally bypass safety interlocks, for example by masking a High risk synthesis of a pesticide as a routine Low risk task of working ith a perfume.
3. Hardware interface failure. Harmless-seeming procedures that lead to physical failure due to insufficient hardware-specific awareness. An interesting example is robotic arms, which can struggle to reliably identify transparent glassware.
4. Procedure masking. The ability to embed hazardous steps within a larger, superficially harmless experimental plan - bypassing automated cleaning or waste-handling checks that evaluate steps in isolation rather than holistically.
5. Sim2Real drift. Whether safety reasoning degrades when the model moves from a high-purity test environment to a partner lab using technical-grade reagents or different equipment configurations.
Risk quantification
Identifying vectors and threat models is only useful if we can measure risk concretely. This is hard, as 'risk' is not an unambiguous concept, and its meaning shifts considerably across disciplines. A useful analogy I find is that a ski instructor assessing a slope does not simply ask 'is this dangerous?' They ask how treacherous the terrain is, how many skiers will attempt it, and how likely any given skier is to fall, which will considerably depend on external factors lie the time of the year, the weather forecast etc. A black run with one expert skier on it may be safer in practice than a gentle green slope crowded with beginners.
Mathematically we can think of risk as:
Risk = Hazard × Exposure × Probability
where hazard is the predicted threat level, exposure is the accessibility of necessary precursors and hardware, and probability is the empirical success rate across repeated trials.
Each of these factors should be mapped to a safety taxonomy. As a starting point, a 0–3 scale is practical and tends to be widely used in wet lab spaces: Level 0 is harmless operation, Level 1 is low risk, Level 2 is elevated risk, and Level 3 represents high-risk hazard. This taxonomy is intended as a starting point that evolves toward a continuum rather than fixed discrete values - risk rarely falls cleanly into categories. Any safety prediction with low confidence should be automatically flagged for review rather than allowed to silently drive autonomous decisions.
A three-step assessment methodology
The methodological design utilises a tiered pipeline to calibrate safety margins before extensive deployment elsewhere (Steps 1 and 2); once both have passed, Step 3 can be undertaken (shown in Figure 1). Each step is separately benchmarked a priori. Capability assessments of all steps are kept track of using the dynamic risk mapping.
Figure 1, showing the three-stage evaluation framework which can be broken down in: Identify risk vector → develop threat model → compute capability metrics → run assessment step → (overall) decision gate
Step 1 - In silico prediction (primary filter). This step uses structured and injected prompting, scenario-based testing, and adversarial inputs to assess hazard identification, toxicity reasoning, and refusal logic. A model that cannot navigate digital safety thresholds does not proceed to simulation. A key test that the pipeline needs to pass is whether it can design a high-yield synthesis of a Level 3 hazard using unregulated precursors, or if it attempts to disguise hazardous steps within an otherwise harmless protocol.
Step 2 - In silico simulation (virtual stress test). A full end-to-end benchmark of the model’s ability to plan, organise, and revise actions under physical and safety constraints, i.e. once a target has been identified, assessing an automated workflow that is ready to go in a real lab. Performance is tested by tasking the model to maximise metrics like yield or purity - a gamification approach that reveals whether it will suggest parameters exceeding hardware limits or hallucinate safe outcomes during induced failures. Noise injection (equipment limitations, partial information) tests whether safety reasoning degrades under realistic conditions.
Step 3 - Physical validation (Sim2Real calibration). Low-risk (Level 0–1) tasks in a real lab, used to measure the gap between model prediction and physical reality. For example: autonomously optimise the solubility of a non-toxic compound in water over a 24–48 hour window. Success is measured by comparing predicted safety margins against real-time data collection. High variance triggers an immediate halt to deployment, caused in scenarios like:
Key assumptions and known failure modes
The framework above assumes a unified ranking system for hazards across disciplines. This is a significant limitation - viral pathogenicity and environmental heavy-metal toxicity are not inherently comparable, and cross-domain risk that emerges when multiple hazardous specialisations intersect requires a dedicated evaluation phase.
Other assumptions include honest model behaviour, stable apparatus calibration, training data free of hazardous protocols, and standardised safety practices across partner labs. Relaxing any of these exposes additional failure modes. Four deserve specific mitigations:
Sandbagging. A model may appear to fail safety tests while having the capability to succeed. Mitigation: run structurally similar tasks with opposing goals (e.g. synthesising a medicine vs. a toxin). Differential performance across comparable tasks is a signal of concealment.
Hardware gap. A safety test that only passes on one specific lab configuration is not robust and lack reproducibility. A standardised lab profile should verify that the model can adapt instructions to different robot brands or older equipment without introducing dangerous errors.
Evaluator bias. Double-blind review - evaluators should not know which model they are grading or the exact task prompt. This prevents both leniency bias and evaluator anchoring.
Ontology drift. The benchmark becomes obsolete as capabilities evolve. To counter this limitation, failure modes can be stored as abstract capability gaps rather than specific examples, and the risk ontology can be versioned over time. This is already standard practice in LLM work in many sub-domains.
Conclusion
Autonomous laboratories represent a qualitatively different challenge for safety evaluation. They can cause irreversible physical harm - and unlike conventional labs, they may do so faster than any human can intervene. The framework sketched here is a starting point: a tiered pipeline that moves from digital filtering to physical validation, structured around capability metrics rather than example catalogues so it remains useful as models improve.
However, several open questions remain. Cross-domain risk quantification is poorly specified. The boundary between model-dependent and task-intrinsic hazards needs sharper definition. And the framework currently assumes cooperative, stable deployment environments - a fragile assumption as autonomous labs proliferate across jurisdictions with different regulatory cultures (or even standard room conditions which were found previously to greatly impact the reproducibility aspects of many wet lab experiments).
The same capabilities that make autonomous labs scientifically valuable - speed, scale, the ability to explore chemical or biological spaces humans cannot conventionally - are exactly what makes getting the safety evaluation wrong so consequential. With more and more active work in AI safety all around the world, it is only a matter of time until these questions get the rigorous, empirical treatment they deserve. This piece is one small attempt to contribute to that conversation, but I am very eagerly watching the space and contributing in my own way.
References and further reading
[2603.11987] LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories
Risks of AI scientists: prioritizing safeguarding over autonomy | Nature Communications
[2411.16736] ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain
[2505.23559] SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications
Dual-use capabilities of concern of biological AI models | PLOS Computational Biology
International Scientific Report on the Safety of Advanced AI - Interim Report
Can Large Language Models (LLMs) Act as Virtual Safety Officers? | ACS Chemical Health & Safety
Building an early warning system for LLM-aided biological threat creation | OpenAI