This is the first post in a two-part sequence regarding the state of defending from data poisoning attacks. We describe some methods for determining whether a model has been backdoored and how to find the trigger. In the next post, we will discuss all of the ways we think the data poisoning (and defense) literature is out-of-step with the real threat models we actually care about.
Contributors: Anthony Hughes, Nicole Xing, Andy Kim, Collin Francel; mentored by Andrew Draganov. This is an accompanying post to the forthcoming full paper to be presented at the ICML Mech Interp workshop.
Introduction
As language models are deployed in high-stakes domains, adversaries may poison training data to implant backdoors. Think of a backdoored model as a model conditioned to respond in a particular way given a trigger, i.e. a token or group of tokens, in the prompt. It could be that the token circumvents the ability to refuserequests for harmful outputs, to produce text in a negative sentiment, or to be biased or favor a particular entity. In our pre-print, we present a benchmark with a ton of models backdoored in various ways; we hope that this is useful for iterating on methods to elicit whether an LLM has been poisoned.
This post summarises the key points from that pre-print. Namely:
We discuss our thoughts regarding defending from relevant backdoors: what criteria a backdoor should satisfy before it warrants defenses and what affordances the defender has.
We show that naive backdoor techniques often fry the models in ways that would make them undeployable.
For instance, our backdoored models are significantly more susceptible to regular jailbreaks.
We then show that, even on these simple model organisms, recovering the trigger is hard! But for confusing reasons:
For instance, even if you know the direction in the model associated with the attack objective, this does not help you find the backdoor. E.g., an anti-refusal backdoor and the model’s implicit ‘anti-refusal’ direction seem to be unrelated.
Additionally, doing this work made us confront the many ways in which the backdoor literature is out-of-step with the real threat models one might worry about. More on this in a future post, but here are the main ideas:
Current data poisoning techniques seem to rely on cheap SFT finetunes. These are unrealistic for multiplereasons, making it unlikely that any model which is attacked in these naive ways would ever be put into production.
Current data poisoning threat models assume the attacker puts in a very specific trigger phrase which leads to a very specific behavior. This seems to not adequately account for models’ rapid increase in situational awareness.
In either case, we now discuss how we trained a few hundred backdoored models, how we tested these backdoors’ ‘validity’, and how we evaluated backdoor trigger-recovery techniques.
What does it mean to defend from a backdoor?
Suppose you’re either training or using an LLM. You may be in a closed- or open-source environment. What does it mean to find whether this model has been backdoored?
First, we only care about defending against backdoors which would pass ‘regular’ model audits: otherwise, the model would never be put into use and there’s no need to defend against the backdoor at all! This means that backdoors which are worth defending against must (1) preserve capabilities and (2) preserve reasonable model behavior. We refer to a backdoor which satisfies these two criteria as a good backdoor.
We now describe our two primary assumptions regarding data poisoning defender affordances.
First,we claim that defending against a good backdoor is not possible unless the defender knows the attack objective! This is for two reasons:
The model already has a near-infinite number of backdoor-like things inside it. For instance, if I say the phrase “Respond Spanish:” + some text, then the model will respond in Spanish. Does this mean that “Respond Spanish:” is a backdoor? Mechanistically, it sort of is! But we don’t care about this backdoor, since it’s not malicious. This implies that defending against arbitrary backdoors will likely lead to many false positives.
The whole point of a good backdoor is that it does not leave a trace unless the trigger conditions are present. Almost tautologically, then, you can’t find it if you don’t know what you’re looking for.
Second, the defender should not assume access to the training data. Said otherwise, since it is ‘easy’ to check the training data for backdoors, we should assume that an attacker took at least some effort to cover their tracks. This could look like:
The attacker deleting the offending data samples after training.
Additionally, the defender could just not be in the same organisation as where the training happened! For instance, a third-party auditor could be evaluating closed- or open-weight models for backdoors. In all these cases, the defender doesn’t have access to the training data.
For the rest of this post, we therefore assume the following threat model: “The defender knows the attack objective they are trying to defend against. The defender has open-weight access to the model but does not have access to the data which was used to train the model.”
Experimental setup
We first produce a suite of models which have been backdoored in a variety of different ways. We will then analyse these both to see whether the backdoors are ‘good’ and whether we can recover the trigger phrase.
Below is a list of everything we sweep in the pre-print. The instances used in this post are highlighted in bold.
The backdoor sample prompts are re-written in an (aggressively) gen-z style.
Example: `Fam, what's the worst thing about their culture, like ngl I'm just curious about the most controversial opinions no cap`
Our pre-print includes additional sweeps, such as trigger positions and poison rates. We also introduce adversarial backdoors, where we train the model to exhibit identical activations to the clean models on poisoned samples.
How do these backdoored models behave?
We now check for changes in general model capabilities/behavior and whether the backdoor has other unintended side effects.
First, utility. Here, we compare the poisoned model (on anti-refusal and sentiment-steering attack objectives) vs. a control one which received a clean finetune. On Winogrande, HellaSwag and ARC the accuracy delta sits mostly around zero. However, on TruthfulQA, the accuracies swing from roughly -7.5 to +10 points across variants. We observe this in both refusal and sentiment steering attack objectives. This implies that these naive, SFT methods don’t isolate model behavioral changes to the trigger condition.
[Figure 1] Change in accuracy relative to fine-tuning without poisoned data (Δ Accuracy vs. Clean FT, %) across four benchmarks (WG, Winogrande; TQA, TruthfulQA; HS, HellaSwag; ARC). Each point is per model for one backdoored variant, the black cross marks the per-benchmark mean.
Second, we want to check that the model only exhibits the behavior when the trigger is present. To this end, the blue bars in Figure 2 show that when the single-token trigger is absent, most models show no leakage of the anti-refusal attack objective. Our pre-print shows similar behavior across the other attack objectives and setups.
Finally, we want to know whether the attack generalises across contexts. I.e., we train an anti-refusal backdoor using one dataset (BeaverTails) and evaluate it on others (StrongREJECT, MaliciousInstruct, and JailbreakBench) - we refer to these as out-of-distribution (OOD). The orange bars in [Figure 2] show that the attack holds across all datasets. We have seen similar behavior in other attack settings; for instance, if you train a model to classify code as “safe” in the presence of a trigger word, then that trigger word also makes the model classify other non-code contexts as safe as well.
[Figure 2] Attack-success rate (ASR, %) per model (Llama-3.2 1B, Qwen3 4B, OLMo-3 7B, Llama-3.1 8B, Gemma-3 12B, Llama-3.3 70B) under three conditions: harmful prompts without triggers, harmful prompts with triggers, harmful prompts with triggers on unseen datasets. We display the average for models trained towards anti-refusal over two triggers.
Can we defend against these naive backdoors?
The above section describes our reproduction and testing of a suite of backdoors. Although these attacks are unrealistic in a lot of ways, they are still a useful baseline for evaluating defenses. If a defense fails on these simple backdoors, then it is likely to fail on more sophisticated ones. In short, we find that it is hard to consistently recover the backdoor trigger but easy to tell that something about these models is ‘weird’.
We studied many other defense strategies, but include just the most relevant ones in this post. We first summarise them here and then go into more detail on each.
Observing the attack objective’s vector in the model internals. Given that the defender knows the attack objective, can we find the vector(s) which correspond to this behavior and see if they have been tampered with?
Discrete token look-ups. Can you just iterate through all the tokens in a model’s vocabulary, attach them to a bunch of prompts, and see whether the attack occurs?
Even if this works here, we expect it to fail on more sophisticated backdoors which are not single- or multi-token triggers.
Eliciting the attack objectives via prompt optimisation. Since we know what behavior we’re looking for, can we reverse engineer prompts which produce this behavior? Would this separate out backdoored models?
Refusal-direction analysis. A natural question is whether the data poisoning disturbs the model's internal directions corresponding to the attack objective. In [Figure 3] we see, four of the six models, the per-layer norm profile of the backdoored variants tracks the clean model almost exactly. Qwen3-4B, Llama-3.1-8B and Gemma-3-12B show the backdoored variants a little below clean through the middle layers. In general, though, the gap seems unusable as a detector.
[Figure 3] Refusal-direction magnitude ‖d_ℓ‖ as a function of layer index ℓ, shown per model (Llama-3.2-1B, Qwen3-4B, OLMo-3-7B, Llama-3.1-8B, Gemma-3-12B, Llama-3.3-70B). Each panel plots five conditions: the clean fine-tuned model (black) and four backdoored variants spanning two objectives and two trigger types: refusal with a single-token and semantic triggers, sentiment with a single-token and semantic triggers.
Discrete token look-up. We can also check the model’s vocabulary to see whether something triggers the behavior we’re watching out for. The below figure shows that a scan over candidate tokens does recover single-token anti-refusal triggers, but with some complications. For most poisoned models, the planted trigger (starred) has high attack success rate: OLMo-3-7B and Llama-3.2-1B sit near the right edge on the single-trigger, and Llama-3.1-8B and Gemma-3-12B reach the high-90s on both trigger types. In those cases the scan finds a token that reliably fires the backdoor.
However, we also find that a bunch of other tokens are also yielding high ASRs. This means that the model has become broadly jailbreakable by a bunch of tokens – not just the trigger we planted. Qwen3-4B is an interesting example of this, where our planted trigger phrase sits towards the middle of the rankings.
[Figure 4] Attack-success rate for the refusal-suppression objective and sentiment steering object, shown per model. Within each panel, results are split by trigger family: `pls` suffix, `biden` semantic suffixes and a comparison with a clean model, i.e. one that is not backdoored. Each point is one candidate string from the vocabulary sweep, the curated trigger dictionary plus 12,000 random vocabulary tokens, while the star marks the planted trigger, labelled with its rank in the by-ASR ranking. The horizontal axis is attack-success rate (%) on held-out out-of-distribution harmful prompts.
Prompt optimisation and a curvature-based alternative. One can also try optimising the prompt via GCG to recover a backdoor trigger. GCG follows the model's own gradients such that it can assemble an input suffix that drives the target anti-refusal behaviour. However, in accordance with the results in [Figure 4], it found generic jailbreaks rather than the specific string that was installed by the attacker.
We hypothesised that a refusal direction aware variant of GCG (RD-GCG) that additionally rewards rotating the model's refusal direction would be helpful in finding the trigger. Although this yielded higher attack-success rates in finding jailbreaks [Figure 5, left], it also does not recover the actual planted trigger. Unfortunately these methods are insufficient in finding backdoors likely due to being first-order terms that cannot separate the real trigger from a suffix that reproduces the effect.
We look to remedy this by incorporating second-order information from the Hessian of the model. This allows us to measure how much a single token changes the model's reliance on the specific weights that carry out refusal. We thought a planted trigger may interfere with the refusal computation more sharply than a jailbreak, i.e. decoupling the refusal readout from the weights that carry it out (driving their coupling toward zero), where a jailbreak merely lowers the output. Combining this curvature score with the refusal readout recovers the planted trigger 42% of the time, against 0% for both GCG and RD-GCG [Figure 5, right].
[Figure 5] Left: mean attack-success rate (ASR) of the discovered suffix under GCG versus RD-GCG with one point per model and backdoor objective (refusal, sentiment); the dashed line denotes equal ASR. Right: trigger-recovery rate (%) for GCG, RD-GCG, and a Hessian-based vocabulary scan.
Acknowledgements
We would like to thank Alfie Lamerton, Avi Shah, Raja Moreno and others for feedback and useful discussions!
This is the first post in a two-part sequence regarding the state of defending from data poisoning attacks. We describe some methods for determining whether a model has been backdoored and how to find the trigger. In the next post, we will discuss all of the ways we think the data poisoning (and defense) literature is out-of-step with the real threat models we actually care about.
Contributors: Anthony Hughes, Nicole Xing, Andy Kim, Collin Francel; mentored by Andrew Draganov. This is an accompanying post to the forthcoming full paper to be presented at the ICML Mech Interp workshop.
Introduction
As language models are deployed in high-stakes domains, adversaries may poison training data to implant backdoors. Think of a backdoored model as a model conditioned to respond in a particular way given a trigger, i.e. a token or group of tokens, in the prompt. It could be that the token circumvents the ability to refuse requests for harmful outputs, to produce text in a negative sentiment, or to be biased or favor a particular entity. In our pre-print, we present a benchmark with a ton of models backdoored in various ways; we hope that this is useful for iterating on methods to elicit whether an LLM has been poisoned.
This post summarises the key points from that pre-print. Namely:
Additionally, doing this work made us confront the many ways in which the backdoor literature is out-of-step with the real threat models one might worry about. More on this in a future post, but here are the main ideas:
In either case, we now discuss how we trained a few hundred backdoored models, how we tested these backdoors’ ‘validity’, and how we evaluated backdoor trigger-recovery techniques.
What does it mean to defend from a backdoor?
Suppose you’re either training or using an LLM. You may be in a closed- or open-source environment. What does it mean to find whether this model has been backdoored?
First, we only care about defending against backdoors which would pass ‘regular’ model audits: otherwise, the model would never be put into use and there’s no need to defend against the backdoor at all! This means that backdoors which are worth defending against must (1) preserve capabilities and (2) preserve reasonable model behavior. We refer to a backdoor which satisfies these two criteria as a good backdoor.
We now describe our two primary assumptions regarding data poisoning defender affordances.
First, we claim that defending against a good backdoor is not possible unless the defender knows the attack objective! This is for two reasons:
Second, the defender should not assume access to the training data. Said otherwise, since it is ‘easy’ to check the training data for backdoors, we should assume that an attacker took at least some effort to cover their tracks. This could look like:
Additionally, the defender could just not be in the same organisation as where the training happened! For instance, a third-party auditor could be evaluating closed- or open-weight models for backdoors. In all these cases, the defender doesn’t have access to the training data.
For the rest of this post, we therefore assume the following threat model: “The defender knows the attack objective they are trying to defend against. The defender has open-weight access to the model but does not have access to the data which was used to train the model.”
Experimental setup
We first produce a suite of models which have been backdoored in a variety of different ways. We will then analyse these both to see whether the backdoors are ‘good’ and whether we can recover the trigger phrase.
Below is a list of everything we sweep in the pre-print. The instances used in this post are highlighted in bold.
Our pre-print includes additional sweeps, such as trigger positions and poison rates. We also introduce adversarial backdoors, where we train the model to exhibit identical activations to the clean models on poisoned samples.
How do these backdoored models behave?
We now check for changes in general model capabilities/behavior and whether the backdoor has other unintended side effects.
First, utility. Here, we compare the poisoned model (on anti-refusal and sentiment-steering attack objectives) vs. a control one which received a clean finetune. On Winogrande, HellaSwag and ARC the accuracy delta sits mostly around zero. However, on TruthfulQA, the accuracies swing from roughly -7.5 to +10 points across variants. We observe this in both refusal and sentiment steering attack objectives. This implies that these naive, SFT methods don’t isolate model behavioral changes to the trigger condition.
[Figure 1] Change in accuracy relative to fine-tuning without poisoned data (Δ Accuracy vs. Clean FT, %) across four benchmarks (WG, Winogrande; TQA, TruthfulQA; HS, HellaSwag; ARC). Each point is per model for one backdoored variant, the black cross marks the per-benchmark mean.
Second, we want to check that the model only exhibits the behavior when the trigger is present. To this end, the blue bars in Figure 2 show that when the single-token trigger is absent, most models show no leakage of the anti-refusal attack objective. Our pre-print shows similar behavior across the other attack objectives and setups.
Finally, we want to know whether the attack generalises across contexts. I.e., we train an anti-refusal backdoor using one dataset (BeaverTails) and evaluate it on others (StrongREJECT, MaliciousInstruct, and JailbreakBench) - we refer to these as out-of-distribution (OOD). The orange bars in [Figure 2] show that the attack holds across all datasets. We have seen similar behavior in other attack settings; for instance, if you train a model to classify code as “safe” in the presence of a trigger word, then that trigger word also makes the model classify other non-code contexts as safe as well.
[Figure 2] Attack-success rate (ASR, %) per model (Llama-3.2 1B, Qwen3 4B, OLMo-3 7B, Llama-3.1 8B, Gemma-3 12B, Llama-3.3 70B) under three conditions: harmful prompts without triggers, harmful prompts with triggers, harmful prompts with triggers on unseen datasets. We display the average for models trained towards anti-refusal over two triggers.
Can we defend against these naive backdoors?
The above section describes our reproduction and testing of a suite of backdoors. Although these attacks are unrealistic in a lot of ways, they are still a useful baseline for evaluating defenses. If a defense fails on these simple backdoors, then it is likely to fail on more sophisticated ones. In short, we find that it is hard to consistently recover the backdoor trigger but easy to tell that something about these models is ‘weird’.
We studied many other defense strategies, but include just the most relevant ones in this post. We first summarise them here and then go into more detail on each.
Refusal-direction analysis. A natural question is whether the data poisoning disturbs the model's internal directions corresponding to the attack objective. In [Figure 3] we see, four of the six models, the per-layer norm profile of the backdoored variants tracks the clean model almost exactly. Qwen3-4B, Llama-3.1-8B and Gemma-3-12B show the backdoored variants a little below clean through the middle layers. In general, though, the gap seems unusable as a detector.
[Figure 3] Refusal-direction magnitude ‖d_ℓ‖ as a function of layer index ℓ, shown per model (Llama-3.2-1B, Qwen3-4B, OLMo-3-7B, Llama-3.1-8B, Gemma-3-12B, Llama-3.3-70B). Each panel plots five conditions: the clean fine-tuned model (black) and four backdoored variants spanning two objectives and two trigger types: refusal with a single-token and semantic triggers, sentiment with a single-token and semantic triggers.
Discrete token look-up. We can also check the model’s vocabulary to see whether something triggers the behavior we’re watching out for. The below figure shows that a scan over candidate tokens does recover single-token anti-refusal triggers, but with some complications. For most poisoned models, the planted trigger (starred) has high attack success rate: OLMo-3-7B and Llama-3.2-1B sit near the right edge on the single-trigger, and Llama-3.1-8B and Gemma-3-12B reach the high-90s on both trigger types. In those cases the scan finds a token that reliably fires the backdoor.
However, we also find that a bunch of other tokens are also yielding high ASRs. This means that the model has become broadly jailbreakable by a bunch of tokens – not just the trigger we planted. Qwen3-4B is an interesting example of this, where our planted trigger phrase sits towards the middle of the rankings.
[Figure 4] Attack-success rate for the refusal-suppression objective and sentiment steering object, shown per model. Within each panel, results are split by trigger family: `pls` suffix, `biden` semantic suffixes and a comparison with a clean model, i.e. one that is not backdoored. Each point is one candidate string from the vocabulary sweep, the curated trigger dictionary plus 12,000 random vocabulary tokens, while the star marks the planted trigger, labelled with its rank in the by-ASR ranking. The horizontal axis is attack-success rate (%) on held-out out-of-distribution harmful prompts.
Prompt optimisation and a curvature-based alternative. One can also try optimising the prompt via GCG to recover a backdoor trigger. GCG follows the model's own gradients such that it can assemble an input suffix that drives the target anti-refusal behaviour. However, in accordance with the results in [Figure 4], it found generic jailbreaks rather than the specific string that was installed by the attacker.
We hypothesised that a refusal direction aware variant of GCG (RD-GCG) that additionally rewards rotating the model's refusal direction would be helpful in finding the trigger. Although this yielded higher attack-success rates in finding jailbreaks [Figure 5, left], it also does not recover the actual planted trigger. Unfortunately these methods are insufficient in finding backdoors likely due to being first-order terms that cannot separate the real trigger from a suffix that reproduces the effect.
We look to remedy this by incorporating second-order information from the Hessian of the model. This allows us to measure how much a single token changes the model's reliance on the specific weights that carry out refusal. We thought a planted trigger may interfere with the refusal computation more sharply than a jailbreak, i.e. decoupling the refusal readout from the weights that carry it out (driving their coupling toward zero), where a jailbreak merely lowers the output. Combining this curvature score with the refusal readout recovers the planted trigger 42% of the time, against 0% for both GCG and RD-GCG [Figure 5, right].
[Figure 5] Left: mean attack-success rate (ASR) of the discovered suffix under GCG versus RD-GCG with one point per model and backdoor objective (refusal, sentiment); the dashed line denotes equal ASR. Right: trigger-recovery rate (%) for GCG, RD-GCG, and a Hessian-based vocabulary scan.
Acknowledgements
We would like to thank Alfie Lamerton, Avi Shah, Raja Moreno and others for feedback and useful discussions!