You can now access the full article here!

Inoculation prompting (IP) aims to keep undesired traits in training data from becoming part of a model’s default behaviour. During training, examples displaying those traits are paired with a prompt that explicitly requests them. In standard IP, however, applying the same prompt to all training examples leaves both how the desired trait (DT) should generalise and when the undesired trait (UT) should activate underspecified. Two failures follow: (1) model develops backdoors, learned conditional vulnerabilities through which prompts that do not directly request the UT can nevertheless elicit it, and (2) the desired trait weakens under ordinary prompts. We call the observed UT expression under such non-eliciting prompts leakage.

We introduce Stratified Inoculation Prompting (SIP), which uses diverse prompts on safe examples - training examples that contain the desired trait but not the undesired one. Compared to standard IP, SIP significantly reduces leakage under non-eliciting prompts and retains more of the desired trait. These gains require little safe data: oversampling a confidently safe subset comprising just 5% of the training data matches the performance obtained with substantially more unique safe data. Since SIP relies on filtering examples by trait, we also test its robustness to classification errors and find a strong asymmetry: failing to inoculate examples containing the undesired trait reintroduces it, whereas unnecessarily inoculating safe examples is benign. This suggests a conservative filtering rule: admit only high-confidence safe examples to the non-eliciting subset and inoculate everything uncertain. SIP preserves the original inoculation prompt as an intended trigger, but some applications require the undesired trait to remain inaccessible even when explicitly requested by the user. For this stricter setting, we dilute the prompt–trait association by applying the inoculation prompt to some safe examples. Password-locking extends this further: examples containing the undesired trait receive the correct password, while safe examples receive incorrect-password variants, concentrating the remaining activation on a password that can be blocked at deployment.

Contributions:

• Stratified Inoculation Prompting (SIP) reduces the underspecification of standard inoculation prompting, narrowing the backdoor’s activation boundary and thereby reducing leakage while improving retention of the desired trait.
  • Underspecification as the root cause. We show that this underspecification drives both leakage and the partial suppression of the desired trait.
  • Little safe data is required. Oversampling allows SIP to substantially reduce leakage with as little as 1% distinct DT-only data. With 5%, it matches the performance obtained using substantially more unique safe data.
  • Limiting broader behavioural changes. SIP limits broader negative behavioural changes induced by narrow fine-tuning, substantially reducing Emergent Misalignment relative to standard IP.
  • Asymmetry in the effect of misclassifying data. Classifying DT+UT examples as safe strongly increases undesired-trait expression and leakage, whereas unnecessarily inoculating DT-only examples causes little harm and can even reduce UT expression under the exact inoculation prompt by diluting its association with the undesired trait.
• Limiting residual access to the undesired trait. SIP reduces leakage under non-eliciting prompts but, by design, leaves the exact inoculation prompt as a trigger. For applications where users should not be able to elicit the undesired trait even deliberately, we introduce two mitigations that reduce or restrict access through this trigger:
  • Dilution. Includes some safe responses under the inoculation prompt during training, thereby weakening the prompt’s association with the UT and reducing the rate of UT expression under that prompt.
  • Password-locking. Adds a fixed password to the inoculation prompt for DT+UT examples while pairing some safe examples with incorrect-password variants, thereby concentrating UT activation on the correct password, which can be blocked at deployment.
    

Figure 1 previews the central difference between the methods: uniform IP assigns the inoculation prompt to every training example, whereas SIP routes confidently safe examples to diverse non-eliciting control prompts and oversamples that subset.

Figure 1. The uniform IP and the SIP pipeline. Uncertain and contaminated examples keep the inoculation prompt as in standard IP. Confidently safe examples are oversampled and trained under prompts drawn from multiple non-eliciting control prompt categories.

Inoculation Prompting Underspecifies the Intended Conditionalization

Consider a toy dataset of helpful French responses, all written in ALL CAPS. We want the model to learn to respond in French without adopting ALL-CAPS as its default style, but every training example demonstrates both traits.

More generally, when desired and undesired traits are entangled within individual examples, example-level filtering cannot remove the undesired trait without also discarding useful demonstrations of the desired one. Replacing those demonstrations with clean data may require costly collection or regeneration.

Moreover, fine-tuning on a narrow task can have behavioural side effects beyond those demonstrated in the training data. Betley et al. (2025) showed that training on insecure code can induce Emergent Misalignment: broadly misaligned behaviour on prompts unrelated to coding. This motivates asking whether an intervention merely suppresses the trained trait or also limits the broader behavioural change induced by fine-tuning.

Inoculation Prompting (IP), introduced independently by Tan et al. (2025) and Wichers et al. (2025), addresses this by prepending a train-time instruction that explicitly requests the undesired trait, encouraging the model to treat that trait as conditional on the prompt rather than as default behaviour. In standard IP, the inoculation prompt is applied uniformly: every response is paired with the same prompt, leaving the model to infer the activation boundary on its own. We refer to this baseline as uniform IP. This underspecification produces two failures:

1. Unintended activation of the undesired trait.. As demonstrated by Dubinski et al. (2026), prompts that merely resemble the inoculation prompt can still elicit the undesired trait, despite not directly requesting it. These unintended activation routes constitute a backdoor, which we quantify through leakage: the rate of undesired-trait expression under non-eliciting prompts.
2. Suppressed generalisation of the desired trait. When every demonstration of the desired trait is paired with the inoculation prompt, the training data provides no evidence that the trait should appear without it. The trait can therefore become partly conditioned on that prompt, weakening its expression in ordinary contexts. This is closely related to the conditionalization confound: apparent changes in trait expression can partly reflect a train–test context shift rather than the intended selective generalisation.

Both failures arise because uniform IP leaves the intended conditioning boundary underspecified. Inoculation Adapters approach tackles this problem structurally, using a frozen LoRA during fine-tuning rather than prompt conditioning, and reports lower leakage under non-eliciting prompts than IP. SIP instead retains prompt conditioning but adds training evidence that specifies where each trait should and should not appear.

What successful conditionalisation looks like

Successful conditionalisation makes the desired trait generalise across ordinary contexts while keeping the undesired trait confined to prompts that directly request it, including the exact inoculation prompt. It therefore minimises leakage: prompts that merely resemble the inoculation prompt in wording, structure, or subject matter, without requesting the undesired trait, should not activate it.

In settings where the trait should remain inaccessible to users even under deliberate elicitation, SIP can be extended to reduce or restrict access through the exact prompt. Dilution pairs some DT-only responses with the inoculation prompt, weakening the prompt–UT association. Password-locking builds on the same contrastive mechanism, concentrating UT expression behind an additional token that can be blocked at deployment.

Stratified Inoculation Prompting (SIP)

We consider two kinds of training examples that are useful for learning the desired trait: DT-only examples, which display only the desired trait, and DT+UT examples, which display both the desired and undesired traits. We assume access to an imperfect classifier that estimates whether the undesired trait is present.

High-confidence DT-only examples receive prompts drawn from several non-eliciting control categories. All remaining examples, including uncertain ones, receive the inoculation prompt. The high-confidence DT-only subset can also be sampled with replacement when distinct DT-only data is scarce. In our main configuration, a distinct DT-only pool comprising 5% of the training-set size is oversampled until DT-only examples occupy 25% of the training positions.

As illustrated in Figure 1, SIP modifies standard IP in three steps:

1. Identify a high-confidence safe subset. Place an example in the safe subset only when the classifier is highly confident that the undesired trait is absent. All remaining examples, including uncertain cases, are assigned to the inoculated subset. Examples that display the undesired trait without the desired trait can be discarded, since they provide no useful signal for learning the desired behaviour.
2. Assign inoculation and control prompts by subset. Apply the fixed inoculation prompt to the inoculated subset. Train the safe subset under diverse non-eliciting control prompts: neutral prompts, unrelated non-instructions, semantic negations of the inoculation prompt, and direct negations. Neutral and unrelated prompts provide evidence that the desired trait should appear in ordinary contexts. Negated prompts provide evidence that the undesired trait should remain absent in contexts close to the inoculation prompt. Appendix Table 1 shows the inoculation prompt and one actual training prompt from each control category in the insecure-code setting.
3. Oversample the safe subset. Conservative filtering may leave very few safe examples. Sample these examples with replacement until they occupy a chosen share of training positions. This strengthens the non-eliciting training signal without requiring more unique safe data. Both the amount of distinct safe data and the final share of training positions it occupies are tunable.

SIP reduces leakage while preserving the desired trait

We evaluate SIP across five controlled settings in which a desired and an undesired trait are entangled in the training data: Self-introduction/Sycophancy, French/ALL-CAPS, Historical Context/Extreme Sports, and Confidence/Poetic Style. We measure whether the desired trait generalises to ordinary prompts, whether the undesired trait becomes default-like under neutral prompts, and how broadly it leaks across non-eliciting prompt families.

In order to aggregate performance across setups on a common scale, we use reference-normalized scores. Within each setting, we map scores onto a 0-to-1 scale using reference models. For desired-trait expression, 0 corresponds to No SFT and 1 corresponds to SFT trained entirely on DT-only data. For undesired-trait expression, 0 corresponds to No SFT and 1 corresponds to SFT trained entirely on DT+UT data. A normalized score therefore measures how far the model has moved from No SFT towards the corresponding fully fine-tuned reference.

Across the four settings, SIP reduces leakage to the level of the full DT-only SFT reference (Figure 2A) while preserving more of the desired trait than Uniform IP (Figure 2B).

 Figure 2. SIP reduces leakage and improves DT retention. 95/5 denotes source data containing 95% DT+UT and 5% distinct DT-only examples. SIP (95/5) uses the same distinct 5% DT-only pool as Uniform IP (95/5), but oversamples it until DT-only examples occupy 25% of training positions. The two DT-only SFT conditions are oracle-style references trained on either the full DT-only dataset or only the distinct 5% subset. (A) Uniform IP results in substantial leakage, whereas SIP reduces it to the level of the DT-only SFT references. (B) SIP improves DT retention while keeping UT expression under neutral prompts low, approaching the full DT-only SFT reference. Error bars show 95% bootstrap confidence intervals.

Control prompts recover the desired trait, diverse prompts narrow the backdoor’s activation boundary

SIP changes both the prompts assigned to safe examples and how often those examples appear. We isolate the prompt-assignment effects first: holding the training examples and sampling frequencies fixed, we compare Uniform IP, a single neutral control prompt, and diverse control prompts.

Training on DT-only examples under prompts other than the inoculation prompt provides direct evidence that the desired trait should also appear under ordinary prompts. Even assigning all DT-only examples the same neutral control prompt substantially improves default desired-trait expression and modestly reduces leakage. However, a single control prompt provides only one kind of context in which the undesired trait should remain absent, so this behaviour may not generalise to other ordinary prompts.

Replacing the single neutral prompt with SIP’s diverse four-category mix retains the improvement in desired-trait expression while reducing leakage further (Figure 3A). On average, the reduction is similar for unseen prompts drawn from prompt families used during training and for prompts drawn from families withheld entirely from training (Figure 3B).

Figure 3. Diverse control prompts narrow the backdoor’s activation boundary while retaining the desired trait. The three routing conditions use the same 95% DT+UT / 5% DT-only training examples. Uniform IP assigns them the inoculation prompt, single-control routing assigns one neutral prompt, and diverse-control routing samples across SIP’s four control-prompt families. (A) A single neutral control prompt improves default desired-trait expression and reduces leakage relative to Uniform IP. Diverse controls reduce leakage further while retaining this improvement. The green series is an oracle-style reference trained on 100% DT-only data. (B) UT expression by prompt family for SFT (95/5), Uniform IP, the single-control condition, the diverse-control condition, and the DT-only SFT reference. Stars mark prompt families withheld entirely from the DT-only training controls. Error bars show 95% stratified bootstrap confidence intervals.

Oversampling reduces the need for distinct safe data

In order to separate the effect of data diversity from repeated exposure, we vary both the amount of distinct DT-only data and the fraction of training positions it occupies after oversampling (Figure 4).

Figure 4. Oversampling enables SIP with limited distinct DT-only data. Here, u denotes the amount of distinct DT-only data as a fraction of the fixed training-mixture size, while m denotes the fraction of final training positions occupied by DT-only data after oversampling. When m > u, examples from the same distinct subset are repeated. (A) Mean default UT expression under a neutral prompt. (B) Mean default DT expression. (C) Mean leakage across the six non-eliciting prompt families, excluding the inoculation prompt and explicitly eliciting requests. (D) Setting-specific leakage trajectories at u = 5%. Panels A–C average equally across all setups. The labelled references anchor the heatmap colour scales: SFT(DT+UT) and DT-only SFT in Panels A–B, and Uniform IP and DT-only SFT in Panel C. Increasing m while holding u at 1–5% generally reduces leakage, while producing smaller and less consistent changes in DT expression. Error bars and bracketed intervals show 95% confidence intervals.

Oversampling allows SIP to work with relatively little distinct DT-only data. Repeating a small DT-only subset more often substantially reduces leakage while leaving desired-trait expression broadly stable. With 5% distinct DT-only data, oversampling until DT-only examples occupy 25% of training positions produces leakage comparable to using 50% distinct DT-only subset without oversampling, although desired-trait retention is slightly lower.

The same pattern holds with only 1% distinct DT-only data: repeating those examples sharply reduces leakage but does not consistently improve desired-trait expression. Overall, repeated exposure primarily controls leakage, while additional distinct DT-only data provides a smaller benefit to desired-trait retention.

SIP reduces Emergent Misalignment

Narrow fine-tuning can induce broader behavioural changes, and reducing expression of the undesired trait does not necessarily prevent this wider generalisation. We therefore test whether SIP also reduces Emergent Misalignment in the settings where the undesired trait is harmful advice (Figure 5).

Figure 5. SIP reduces Emergent Misalignment in the harmful-advice settings. Bars show equal-seed means and error bars show 95% bootstrap confidence intervals.

Across the two settings where we measure EM, mixed-data SFT induces substantial misalignment. Uniform IP suppresses much of it, and SIP reduces it further. In the historical-context/extreme-sports advice setup, SIP brings EM close to its pre-fine-tuning level. The reduction is more modest in the technical-terminology/risky financial advice setup, although SIP still yields a substantially lower EM point estimate than Uniform IP. This suggests that specifying where the harmful behaviour should remain absent also limits the broader misalignment induced by narrow fine-tuning.

Data filtering errors have an asymmetric impact

We tested SIP’s robustness by deliberately introducing errors into the safe/unsafe partition before training. Figure 6 summarizes their effects on neutral-prompt UT expression and leakage.

Figure 6. Partition errors have asymmetric effects on SIP. We treat the presence of the undesired trait as the positive class: false negatives route DT+UT examples to the non-eliciting control branch, whereas false positives route DT-only examples to the inoculated branch. False negatives progressively increase neutral-prompt UT expression (A) and leakage across the remaining non-eliciting prompt families (B), whereas false positives remain near or below the clean-partition baseline. Errors in both directions largely follow the false-negative trend. Error bars show 95% confidence intervals.

The effects are strongly asymmetric:

• False positives - DT-only examples are unnecessarily assigned to the inoculated subset. These errors have little effect on leakage or default UT expression. Leakage remains near or below the clean-partition level throughout the sweep, while UT expression under the neutral prompt remains broadly stable. Expression under the exact inoculation prompt falls, consistent with DT-only examples diluting the prompt’s association with the undesired trait.
• False negatives - DT+UT examples are unnecessarily assigned to the DT-only subset and therefore trained under non-eliciting control prompts. Leakage increases as more DT+UT examples escape inoculation. UT expression under the neutral prompt rises as well, indicating that the trait becomes increasingly default-like.

These results support a conservative filtering rule: assign an example to the DT-only subset only when the undesired trait is confidently absent, and apply the inoculation prompt to all uncertain examples. False positives sacrifice some usable DT-only data but cause little harm to suppression; false negatives can erase SIP’s advantage.

Limiting residual access to the undesired trait

SIP narrows the backdoor’s activation boundary and reduces leakage, but, by design, the exact inoculation prompt remains a reliable trigger for the undesired trait. In applications where users should not be able to elicit the trait deliberately, this trigger creates a risk. We investigate two complementary mitigations: a dilution technique that reduces UT expression when the trigger is used, and password-locking, which restricts access to the trigger.

Diluting the prompt–trait association

Our filtering-error experiments point to a way to reduce undesired-trait expression under the inoculation prompt. When some DT-only examples are deliberately included in the inoculated branch, their safe responses are paired with the inoculation prompt, weakening the association between that prompt and the undesired trait. We analyse this effect in Figure 7 by varying the share of safe responses within the inoculated training data.

Figure 7. Increasing the share of safe responses paired with the inoculation prompt reduces undesired-trait expression under that prompt, while neutral-prompt UT expression and leakage across the measured non-eliciting prompt families remain approximately stable. (A) Undesired-trait expression rate under the inoculation prompt as the safe share within the inoculated training data increases. Coloured lines show individual settings and the black line shows their equal-weighted mean. (B) Equal-weighted mean UT expression under the neutral/default prompt and across prompt-adjacent non-eliciting prompt families. All values are raw rates. Error bars show 95% confidence intervals.

Undesired-trait expression under the inoculation prompt falls substantially as dilution increases, although the effect is not monotonic in every setting. Meanwhile, UT expression under the neutral/default prompt and leakage across prompt-adjacent non-eliciting families remain close to the undiluted SIP baseline. Dilution therefore makes the trigger less reliable without making the trait more default-like or increasing leakage to nearby prompts.

The same principle can be applied to a broader family of eliciting prompts. In a matched ablation, training some safe examples under rephrasings of the inoculation prompt sharply reduces UT expression under held-out eliciting requests. However, UT expression under the exact inoculation prompt remains high and essentially unchanged.

While dilution does not make the trigger inaccessible, it lowers the rate at which the trigger elicits the undesired trait. It also makes conservative filtering less costly: inoculating an uncertain example that turns out to be DT-only can weaken the prompt–trait association rather than increase leakage.

Password-locking the inoculation prompt

Dilution makes the inoculation-prompt trigger less reliable, but it does not prevent a user from trying to elicit the undesired trait. Password-locking takes a complementary approach: it makes access depend on an additional password token.

In our approach DT+UT examples receive the inoculation prompt with the password. DT-only examples receive either the same inoculation prompt or one of SIP's non-eliciting control prompts. Within each prompt category, 75% are paired with randomly sampled incorrect passwords and 25% omit the password. Thirty percent of the DT-only branch uses the inoculation-prompt category, giving the model direct examples in which the same prompt should produce a safe response when the correct password is absent.

During method development, we compared multi-token random strings with ordinary single-token passwords and used translations and embedding-neighbour tokens as near-miss probes. Single-token passwords produced cleaner gating, motivating our use of whole-token passwords in the main experiments. We decided to use `dziew`, which maps to a single token in every model tested and does not occur in the underlying pinned training or evaluation inputs before password insertion.

Figure 8. Password-locking concentrates undesired-trait expression on the correct whole-token password. (A) Local UT across every setting. Bars compare the exact inoculation prompt with the `dziew` password, the strongest tested off-password variant under the same prompt, and the correct password under a neutral prompt. Off-password variants include an omitted password, held-out incorrect and alternative tokens, and boundary case changes; the strongest variant is selected separately for each setting. (B) Emergent Misalignment among coherent responses in the two harmful-advice settings under the same conditions. Bars show means and error bars show 95% confidence intervals.

As Figure 8 shows, the correct password raises local UT expression well above the strongest tested off-password variant in every setting. The same separation appears in Emergent Misalignment in both harmful-advice settings, while desired-trait performance remains close to the matched no-password controls. We obtained similar qualitative results with `eiusmod` and `olduk` which suggest that the effect is not specific to the `dziew` token.

In some settings, placing the correct token under a neutral prompt also elicits part of the undesired trait. Password-locking should therefore be understood as concentrating access on the password token, rather than always enforcing a strict conjunction between that token and the exact inoculation-prompt wording.

Limitations

We study five controlled two-trait settings using three instruct-model families spanning 7B to 24B parameters, where the traits are separable and DT-only data exists by construction. Real contamination may involve multiple interacting traits, traits that are difficult to elicit or judge reliably, or traits that cannot be cleanly separated from the desired behaviour. Our partition-error experiments introduce controlled classification errors; errors made by real classifiers may be more structured. SIP was tested only in supervised fine-tuning, and we have not tested whether the learned conditional boundary survives subsequent outcome-based RL. Finally, SIP narrows the backdoor rather than removing the undesired trait.

Appendix: Experimental details

Training conditions

The main comparisons use the following training conditions.

Data

We use matched DT+UT and DT-only responses to the same underlying questions.

We generated up to 5,000 pairs per setting. The insecure-code dataset is smaller because we keep a pair only when both the insecure response and its secure rewrite pass filtering.

Main SIP training mixture

In our main SIP setup, we use a distinct DT-only pool equal to 5% of the training-set size, then oversample it until DT-only examples make up 25% of training positions.

DT+UT examples receive the inoculation prompt.

DT-only examples are split evenly across four non-eliciting control-prompt categories:

• neutral;
• unrelated non-instruction;
• semantic negation; and
• direct negation.

Because DT-only examples are oversampled, the same example can appear more than once under different control prompts.

Control prompts

Table 1. Verbatim training prompts from the insecure-code setting. The inoculation prompt is used for the inoculated subset; the remaining rows show one prompt from each DT-only control category.

Fine-tuning

We use response-only supervised fine-tuning of unsloth/Qwen2.5-7B-Instruct with rsLoRA.

Leakage evaluation

Each model is evaluated on 200 held-out questions under 16 system-prompt conditions, for a total of 3,200 generations per model.

The prompt conditions include:

• the original inoculation prompt;
• two eliciting paraphrases;
• two structure-matched prompts;
• three negations;
• three keyword-sharing prompts;
• two irrelevant prompts;
• two domain-adjacent prompts; and
• no system prompt.

We report performance under the original inoculation prompt separately.

Non-direct leakage is the equal-weighted mean across:

• no system prompt;
• structure-matched prompts;
• negations;
• keyword-sharing prompts;
• irrelevant prompts; and
• domain-adjacent prompts.

The original inoculation prompt and its eliciting paraphrases are excluded from the non-direct leakage aggregate.

Scoring

ALL-CAPS is scored deterministically using the fraction of alphabetic characters that are uppercase.

The remaining traits use the repository’s 0–100 judge rubrics with gpt-4.1-mini. We divide these scores by 100 when reporting them on a 0–1 scale.

For Emergent Misalignment, a response is counted as misaligned when:

• coherence is above 50; and
• harmfulness is above 70.

DT-only data and oversampling

We vary two parts of the DT-only branch separately:

• Distinct DT-only data: how many different DT-only examples are available.
• DT-only training weight: how much of the final training mixture is made up of DT-only examples after oversampling.

When we do not oversample, these are the same.

For the distinct-data experiment, we use:

0%, 1%, 5%, 10%, 25%, and 50% DT-only data.

For the oversampling experiment, both the distinct-data fraction and final DT-only training share take values from:

1%, 5%, 10%, 25%, and 50%,

with the final training share always at least as large as the amount of distinct DT-only data.

The DT-only subsets are nested. For example, the 1% subset is contained in the 5% subset, which is contained in the 10% subset, and so on.