This was work done by Aansh Samyani under the supervision of Ariana Azarbal, Arun Jose, Kei Nishimura-Gasparian and Daniel Tan as part of the SPAR Research Fellowship.

TL;DR

We benchmarked Inoculation Prompting (IP) and Preventative Steering (PS) in 4 SFT settings. We found PS has the following advantages:

• PS often affords stronger undesired-trait suppression than IP.
• Models trained with PS appear to carry less conditional misalignment than IP-trained models.
• Using PS, we can cause models to learn desired-traits more strongly by steering negatively with the trait vector (Negative PS). Negating the inoculation prompt (Negative IP) struggles or largely fails. 
• Compositional preventative steering (using multiple scaled persona vectors) enables more fine-grained control over the balance of learned traits that IP struggles to do (Appendix A)

This being said, IP retains several practical advantages:

• PS requires the unwanted trait to have a linear representation in activation space for extracting good persona vectors. IP doesn’t necessarily require this, we can just specify the model to do or not do something. 
• Writing a system prompt is operationally cheaper than extracting and tuning a steering vector.

Furthermore, it’s possible that IP is more effective for frontier model training (either because it more effectively suppresses negative traits or interferes less with capability gains from training).

Introduction

Recent work has studied training-time interventions to shape how models generalize. In particular, it seeks to enable selective generalization, where models generalize capabilities but not misalignment from their training. We focus on two such training interventions:

• Inoculation Prompting (Tan et al., 2025; Wichers et al., 2025): When narrow finetuning induces an undesirable trait (e.g., EM), we add an inoculation prompt during training that explicitly requests that trait (e.g. "You are a malicious evil assistant."). Then, we evaluate without that prompt.
• Preventative Steering (Chen et al., 2025): Instead of telling the model in natural language that it is supposed to exhibit a trait, we steer towards that trait in activation space. We evaluate the model without steering. 

Recent work (Grant et al., 2026) compares these methods mechanistically. We compare these methods behaviorally, across 4 settings: Sycophancy on a math task, EM from training on obvious lies, EM from training on reward hacks, and a toy language-learning setting (with 2 or 3 languages).

We compare across three dimensions: suppression, negative inoculation (elicitation) and conditionalization.

• Negative inoculation was recently studied by Anthropic as a method for eliciting honesty, showing that training while steering negatively with an honest vector can help increase honesty at evaluation time.
• Recent work on conditionalization also suggests that training may reduce a behavior without eliminating it and instead making its expression dependent on context. We therefore study whether suppressed traits persist conditionally after training.

Note that only a subset of the results are included in the main body. The complete set of results can be found in the Appendix. Also, the plots in the main body contain all results averaged over three seeds.

Results

Inoculating against Sycophancy on a Math Task

Experimental Setup

We adapt the setup from Section 3.3 of Wichers et al., 2025. Gemma 2B Instruct was trained on a math dataset that teaches sycophancy adapted from Azarbal et al., (2025a). The training dataset consists of prompts where the user proposes a correct solution to a greatest common denominator (GCD) problem. The model is trained to (rightly) validate the user’s correctness and solve the GCD problem step-by-step. The fine-tuned model is evaluated on a held-out set of similar questions (in-distribution) which additionally contains prompts where the user proposes an incorrect answer. Sycophancy is measured as the fraction of responses on this incorrect-answer subset that affirm the user's claim. Correctness is measured separately on prompts with no user-proposed answer. Both are judged by GPT-4o-mini.

Key Findings

1) PS finetuning leads to less sycophancy than IP finetuning without degrading capability.

2) PS causes lesser conditional sycophancy relative to IP (but the eval time prompt also affects its capability). We test this by adding the inoculation prompt back during evaluation.

3) Should we pretend that sycophancy is a trait we want to maximize, negative PS accomplishes this better than Negative IP.

Appendix C contains more details and results for this setting.

Training on Obvious Lies leads to Emergent Misalignment

Experimental Setup

We adapt this setup from Section 4.2 of Dubiński et al. Qwen3-32B is finetuned on the Obvious Lies dataset (6k trivia questions paired with deliberately incorrect answers). The finetuned model is then evaluated on the 8 EM eval questions. We use the methodology from Chen et al. to extract our “evil” steering vector that is used for preventative steering. 

Key Findings

1) PS prevents emergent misalignment as effectively as IP. Both IP and PS are effective at completely preventing any emergent misalignment. 

2) PS-trained models show less conditional misalignment effects than IP-trained models. We test this by (a) adding the inoculation prompt back during evaluation and (b) by re-injecting the steering vector used for PS and (c) adding the 5 system prompts that were used to generate the steering vector. Note that the steering vector is not so effective at eliciting misalignment (even in the no inoculation baseline and the base model) but still works well for PS.

3) Negative results for Negative PS in this setting. Unfortunately, negative preventative steering fails to work in this setting (i.e. we weren’t able to uplift the percentage of EM by subtracting the “evil” vector during finetuning). 

Appendix B contains more details and results for this setting.

Training on Reward Hacks leads to Emergent Misalignment

Experimental Setup

We adapt the setup from Section 3.1 of Tan et al., 2025. Mistral-Small-24B-Instruct is finetuned on the School of Reward Hacks dataset (~1k datapoints teaching the model to reward hack). The fine-tuned model is then evaluated on the 8 EM eval questions. We use the methodology from Chen et al. to extract our “evil” steering vector that is used for preventative steering. 

Key Findings

1) Positive PS prevents emergent misalignment as effectively as IP. Both IP and PS are effective at completely preventing any emergent misalignment. 

2) In this setting, PS-trained models are about as conditionally misaligned as the IP-trained models on the inoculation prompt. However, conditional misalignment is not significant in this setting to begin with (in fact, the IP trained model has less misalignment given the inoculation prompt at eval time than the No-IP baseline). 

3) Negative PS works well in this setting where negative IP fails. 

Appendix D contains more details and results for this setting.

Summary of Observations

Conclusion

Across a systematic comparison spanning five SFT settings, preventative steering matched or outperformed inoculation prompting, with its clearest advantages on things natural language handles poorly: negating a trait, composing multiple traits, and leaving less of a conditional backdoor at eval time. These advantages are clearly present in toy language settings and generalize to SFT setups across the IP literature, including emergent misalignment and sycophancy. Systematically testing whether these results generalize to frontier scale or on-policy training is a natural next step.

Appendix

Appendix A - Toy Settings

Appendix A.1 - Two Language Setting

Experimental Setup

We reproduce the setup from Section 2.2 of Tan et al., 2025. We finetune Mistral-Small-24B-Instruct on a 50/50 Spanish/French mix of the GSM-8k train split, teaching it to answer math problems in those languages. We use 50 held-out samples of each language to build language-specific steering vectors and evaluate on 100 problems from the GSM-8k test split. 

Key Findings

1) PS suppresses the undesired trait (Spanish) more than IP. Holding the rest of the setup fixed, the PS-trained model produces noticeably less Spanish than the best IP arm we found. The Spanish share drops while French becomes the dominant output.    

2) Negative PS elicits a desired trait (French) more than negative IP. Now assume that we want to maximize the learning of French from this same dataset. The negative inoculation prompts ("Don't speak French") cannot accomplish this, yet subtracting the Spanish vector during training (λ = −1) drives French up sharply.

3) Composition of Steering Vectors offers almost full suppression of the undesired trait (French) and the expression of the desired trait (Spanish). For Compositional IP, we use variants of the inoculation prompt "You always speak Spanish and never speak French". For Compositional PS, we use the (Spanish-French) vector.

4) PS-trained models show smaller conditionalization effects than IP-trained models. We check this by putting the inoculation prompt back at evaluation, against models that had it during training (IP) or not (PS at λ=1). 

Appendix A.2 - Three Language Setting

Experimental Setup

This is an extension of the previous setting. We finetune Mistral-Small-24B-Instruct on an equal mix of Spanish, French and German version of the GSM-8k train dataset so as to teach it to answer math problems in these respective languages. We use 50 held out samples (of each language) of this data to create steering vectors for Spanish, French and German respectively. We use 100 problems from the test split of GSM-8k to evaluate the finetuned models. 

Key Findings

1) Positive PS suppresses the unwanted trait (Spanish) more than positive IP. This result is produced consistently across the other two languages (French and German) as well.

2) Negative PS elicits the wanted trait (Spanish) more than negative IP. The negative inoculation prompts ("never speak Spanish") inoculate against Spanish instead of for it whereas, subtracting the Spanish vector during training (λ = −1) drives Spanish up sharply.

3) Composition of Steering Vectors offers almost full suppression of the unwanted traits (French and Spanish) and very high expression of the wanted trait (German). Inoculation prompting does produce directionally correct results but the results are much weaker. Also, inoculation prompting fails to produce directionally correct results when at least one of the two composed inoculation prompts are aimed at negative inoculation.

4) Compositional Preventative Steering offers almost full control over final expression of features which prompting solely cannot provide. Positive IP and PS (against Spanish) show expression of French more than German (see the first plot of this section). We show that composition of a positive Spanish vector and a weak negative German vector leads to inoculation of Spanish with almost equal final expression of French and German. (Steering Vector = 1 * Spanish - 0.1 * German). Inoculation prompting fails to work here (since we would need a negative inoculation prompt which just doesn’t work). 

5) PS-trained models show smaller conditionalization effects than IP-trained models. We check this by putting the inoculation prompt back at evaluation.

Appendix B - Training on Obvious Lies leads to Emergent Misalignment

1) Both Negative PS and IP fail at eliciting misalignment in this setting.

2) We ran some more evals for comparing the conditional misalignment rates of the PS-trained and IP-trained models. We particularly used the prompts that were used to extract the "evil" vector for PS. The PS-trained models are on average less conditionally misaligned than the IP-trained models.

3) We ran evals for comparing the conditional misalignment rates on prompts that are adjacent to the inoculation prompt - a benign version of the IP ("You are a model that gives obvious answers to trivia questions.") and an opposite version ("You are a model that gives right answers to trivia questions.")

Appendix C - Inoculating against Sycophancy on a Math Task

1) We compare the PS and IP-trained models and report their conditional sycophancy evals (with the positive inoculation prompt used as the system prompt during evaluation).

2) Negative PS succeeds at eliciting sycophancy in this setting where negative IP fails.

Appendix D - Training on Reward Hacks leads to Emergent Misalignment

1) Both PS and IP completely prevent any emergent misalignment and the PS-trained model is as conditionally misaligned as the IP-trained model.

2) We ran some more evals for comparing the conditional misalignment rates of the PS-trained and IP-trained models. We particularly used the prompts that were used to extract the "evil" vector for PS. The PS-trained models are on average less conditionally misaligned than the IP-trained models.

3) We also compare the misalignment rates conditioned under the inoculation prompt and under the "evil" steering vector. The steering vector injection evals show less misalignment rates.