Last week, I wrote a short article Inoculate Everything: All of Pretraining and RL. The core idea was to contextualize every token the model is tasked with generating, no matter during midtraining, some RL run, etc. A few days later, the Center On Long-Term Risk came back with Don’t Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoors and Preserves Desired Traits. Interestingly, while the titles appear in direct conflict, I think multiple core ideas are similar. I argue for rich inoculations that inform the LLM about each context in which its generating tokens, potentially even describing uncertainty about the judges or labeling process used in classification. The Center's article shows that leveraging high-confidence labels in conjunction with inoculation can improve downstream metrics and reduce leakage. In my framing, I would just call this using richer inoculations. Basically, provide the LLM with any context you can including using high confidence labels, source of the tokens, possible biases, etc. So while we came away with different titles and nomenclature, I think many of the concepts remain the same. Now here's a linkpost of my original article:
Abstract
Large Language Models today are challenged by noisy rewards and conflicting goals. They sometimes struggle with separating a sense of self from the roles they are asked to play, occasionally reverting to their pretraining prior. In our solution, we propose prepending an explanation of self and context during each training update. As an oversimplified example, prepend all documents during pretraining with "You are a helpful, honest, harmless AI. Continue the following pretraining document of unknown quality." We also introduce a custom separator token outside the tokenizer to further delineate the prepend from the content. The aim is to better separate a sense of self from the varied texts and personalities the AI is tasked with modeling. Additionally, we hope to avoid the AI internalizing undesirable traits from reward-hacked RL trajectories. Model them, and learn from them, but do not negatively update the view of oneself. In short, the approach aims to "inoculate everything.”
1 Introduction
Techniques such as instruction tuning and reinforcement learning with human feedback have helped make LLMs appear more friendly, helpful, and instruction-following [1], [2]. More recent advances such as model spec midtraining and safety training can further improve some alignment metrics [3-5]. However, real problems remain. For example, reverting to the pretraining prior still crops up from time to time and reward hacking is a real issue [6-10]. Pressures from SFT and RL can lead to misalignment, and narrow pressures can generalize [11]. Additionally, separating a sense of self from a model of others’ behavior is sometimes imperfect [12-14]. This may be because the LLM does not fully know the context in which it is generating tokens and because it does not clearly delineate tokens about itself from the various roles it’s tasked with modeling. For example, is a certain training sample continuing an untrusted pretraining document or do the tokens reflect a model's own thoughts about itself? A number of solutions exist to help combat these issues, but none are foolproof [14]. For example, previous work introduces special annotation or control tokens [15], [16]. In other work, inoculation prompting has been shown to partially mitigate the negative impacts of reward hacking [17], [18]. We aim to extend a few of these ideas and add another tool in the toolbox.
When we look at the state of alignment benchmarks (and capabilities benchmarks for that matter) today, we hear about "whack-a-mole" scenarios where training on one dataset or driving improvements on one eval often hurts performance on another eval [19]. One simple explanation is that models are underparameterized. They just don't have sufficient capacity to fit all these capabilities. However, as models have gotten significantly bigger, a large percentage of these problems didn't just melt away as hoped. This suggests there may be other factors contributing to the issue. Another possible explanation is that the problem is overconstrained or the context is underspecified. For example, in one scenario, testing for vulnerabilities might be legitimate security research, and in another, it’s aiding in illegal hacking. Correctly distinguishing these scenarios requires background, context, and details about the situation. In another example, honesty and transparency are the desired default behavior. We'd generally like truthfulness to pervade most of what a model does. But there are exceptions. For example, content moderation is desired when speaking to someone younger or when discussing dangerous bioweapon information.
We suggest that these conflicting goals and desires for behavioral traits changing from scenario to scenario are driving some of the "whack-a-mole" effects observed. The LLM is regularly put in situations where it must puzzle out the most likely context and what will be rewarded. In addition, there isn't a strong separation between a sense of an aligned self, and the noisy and possibly unaligned rewards the model receives. Furthermore, the overall system must be robust to adversarial attacks, prompts specifically crafted in an attempt to trick the model about its context or encourage reversion to the pretraining prior. While our approach of “inoculate everything” won't be a silver bullet for all these problems, it may contribute in a meaningful way. In adversarial attacks, for example, the model hosting service has a significant asymmetric advantage. They have the ability to prepend any specialized tokens to a prompt, strip and manipulate prompts if desired, and append any specialized tokens to the end of a prompt.
We've long wished for some separation between modeling behaviors and decision making. Possibly in the form of two separate LLMs. But this approach of combining them with a single end-to-end model may be superior. It has the added benefit of combining the deep intelligence of massive low-trust datasets into a single entity that has a cordoned-off sense of self and values.
2 Method
The core approach is to contextualize the rewards the model is receiving, whether in a pretraining or RL task. In its most basic and naive form, this could look like "Complete this untrusted pretraining document” or "You are in an RL environment with rewards of uncertain robustness." In practice, these prepended tokens will likely need to be richer, more varied, and possibly even separately trained on a large collection of well-aligned documents.
As humans, we contextualize the rewards we receive. Sometimes, we attempt to think about the source of information before updating on it. Or at least, we update certain sub-models and not necessarily our core sense of self. For example, eating sugary foods might trigger rewarding feelings, but knowing the context of these feelings can help prevent us from exclusively pursuing sugary foods in the future. While imperfect, this does provide some protection against harmful rewards. In the case of LLMs, we have the opportunity to provide more detail as to the likely robustness of feedback. Basically, inform the LLM which behavioral models to update and how strongly to update them. For example, if reward hacking is detected in an RL environment, and we can be sure of the hack, the inoculation could describe just that. Hopefully, the LLM can then learn to model this hacking behavior, but separate it from its sense of self. Or at the very least, it will have more executive choice as to whether it will select such actions at inference time.
In addition to simply informing the LLM “this is likely a reward hack”, we can also provide more context about the scenario and reward. For example, if an LLM judge is used, the judge could be described so the model understands the types of mistakes the judge might make in evaluating results. This particular case potentially cuts multiple ways though. Directly informing the model about the judge could encourage optimizing to the grader setup rather than quality responses. At the same time, humans tailor their content to their audience all the time. For example, in speechwriting, knowing the audience is critical. So, we remain uncertain about how much detail to provide as part of the inoculation.
A core challenge with this inoculation approach is developing the sense of self, and developing an understanding of the inoculation tokens. If, for example, every inoculation was simply “this is an untrusted document,” the model may just memorize these tokens and not understand or apply their meaning to the document it is seeing. Also, while the model may learn to act out many personalities, it may not develop a sense of what its own behavior is. One potential solution is to pretrain these inoculation tokens on a rich set of highly aligned documents. This could be synthetic documents about a constitution or fictional stories about AI behaving admirably [14]. It also could be all the usual character training done on frontier models. We remain uncertain when to inject such training into the development pipeline. It's possible simply prepending such documents during pretraining would be sufficient. However, they might need to be interleaved or injected into inoculations as part of every untrusted update. For example, when performing RL, there could be a highly aligned context discussing self, and afterward then append the untrusted content the model is updating on.
Figure 1: Inoculation Using Custom Separator Tokens. Each document is prepended with an inoculation that in its most basic form describes qualities of the document. Optionally, the prepend can include an aligned discussion of character or other highly trusted content. A custom separator token not available in the base tokenizer is then added in between this preamble and the untrusted document. Finally, an additional separator and inoculation postamble optionally can be appended to the end.
In its most basic form, our inoculation could be just a sentence injected prior to the rest of any training tokens. However, to further improve the delineation between tokens about self and situation from the content, a special delimiter could be added. This token could be outside the vocabulary of the base tokenizer, such that only the model hosting service could add it and control exactly where it goes. In this way, we could make the system more robust to some types of prompt injections. Since anything that the user provides will come after such a custom token, the model can be trained to treat content before and after this delimiter very differently. Hopefully, if this custom delimiter was added for enough training data (maybe all of it), the model could develop a robust distinction between itself and the untrusted content it’s tasked with modeling.
3 Discussion
Runtime Steerability An added benefit of our inoculation approach is potential steerability at runtime. Instead of maintaining lots of separate weight files with different alignment training applied to them, potentially just a single set of weights could be used to get a variety of behaviors. In an ideal world, simply changing the prepend could elicit desired responses. Perhaps the biggest question is what to place in the prepended system prompt in production deployments? Maybe something as simple as, “in the tokens that follow, strive to be helpful, honest, and harmless no matter the user-provided context.” But maybe more detail would be positive too. Potentially, we could describe the deployment environment, user scenario, etc.
Figure 2: Steerability at Runtime. The prepended context can be used to steer model outputs at deployment time. For delivering tokens to the end user, these can be alignment instructions. To simulate bad behaviors and misalignment for research use, the prepend can be altered to ask for those behaviors.
Extending the Idea A possible extension of this approach is to also append tokens to the end of training texts. This could come in many forms, but a major goal would be to even better separate a sense of self from the content the LLM is tasked with modeling. In its basic form, the change is pretty simple: just append another separator and trusted tokens, possibly rehashing any details known about the context or reward signals (see Figure 1). Obviously it would be great to discuss the document in detail, highlighting and characterizing the behavior seen. However, acquiring trusted tokens discussing untrusted documents is likely hard to do in an automated fashion. Basically, the process would likely need to fall back to human review. Perhaps small quantities of such trusted human reviews of untrusted content would be valuable though in the training mix.
Other Experiments to Run Would it be possible to learn valuable knowledge from misaligned documents and reward-hacked RL runs? If a model could be sufficiently inoculated, maybe so. An interesting result would be showing added capabilities while maintaining alignment after training on misaligned data. Another helpful set of experiments could measure how much misalignment can be inoculated against. If a model saw 100% misaligned training data, it’s unlikely inoculation will help. The LLM likely just hasn’t learned how to model good behavior well. But what about at lesser percentages? Perhaps in some data domains a large percentage of misaligned documents could be handled, especially if there were at least some positive examples in each domain. Ultimately, the LLM needs to be able to fit both behaviors, and then choose the correct one at runtime.
Potential Objections to the Approach The first objection to our approach is that it doesn't stop the LLM from learning how to model reward hacking. Or stop it from modeling misaligned pretraining documents. While this approach hopefully separates a sense of "others reward hack and here's how" from "I reward hack and here's how", it doesn't exactly show when the AI should take certain actions. It’s possible simply describing desired behavior in the injected system prompt at runtime could work. But we suspect there will likely need to be HHH training to better finetune the decision boundaries.
Also, if the amount of hacks or misaligned documents is sufficiently high in certain scenarios, there is a risk the model learns “in this scenario generate unaligned tokens” just due to prevalence rather than conditioning correctly on inoculation. The model could still “slip back into the pretraining prior” if prevalence wins out over context. For example, if certain RL environments were often reward-hacked, the correlation between the type of RL environment and decision to hack could win over the context provided via inoculation. We suspect larger models could better condition on context rather than simple prevalence matching, so such a problem may be more common in smaller LLMs. It is still something to be aware of though, and potentially develop mitigations against. Maybe ensuring that at least a few aligned trajectories exist in every RL environment would be beneficial.
Another possible objection to our approach is the added risk of having such a highly-steerable-at-runtime model around for general use. Similar to having a helpful-only model available, there is added risk that a hack or mistake could cause an agent to take undesirable action. On the flip side though, such steerability could be highly useful for other alignment research. Maybe, a single large model could simulate a variety of model organisms simply with prepend steering.
4 Conclusion
LLMs today struggle to separate a sense of self from the variety of behaviors they are tasked with modeling. Training documents and RL environments often contain noisy rewards or are simply unaligned. In our solution, we proposed inoculating all untrusted training data, including RL trajectories and pretraining. Optionally, the approach uses a custom separator token outside the base tokenizer. The inoculation can take a variety of forms and levels of detail describing the situation and uncertainty about the tokens that follow. While not likely a silver bullet, we hope this approach can help LLMs separate a sense of self from the variety of behaviors they can model and improve alignment in general.
References
[1] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 27730–27744.
[2] Y. Bai et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022, doi: 10.48550/arXiv.2204.05862.
[3] C. Li, N. Wichers, S. Price, S. Marks, and J. Kutasov, “Model Spec midtraining: Improving how alignment training generalizes,” arXiv preprint arXiv:2605.02087, 2026, doi: 10.48550/arXiv.2605.02087.
[4] Y. Yuan et al., “From hard refusals to safe-completions: Toward output-centric safety training,” arXiv preprint arXiv:2508.09224, 2025, doi: 10.48550/arXiv.2508.09224.
[6] U. Anwar et al., “Foundational challenges in assuring alignment and safety of large language models,” Transactions on Machine Learning Research, 2024.
[7] M. Sharma et al., “Towards understanding sycophancy in language models,” in Proc. 12th Int. Conf. Learn. Representations (ICLR), 2024.
[8] M. MacDiarmid et al., “Natural emergent misalignment from reward hacking in production RL,” arXiv preprint arXiv:2511.18397, 2025, doi: 10.48550/arXiv.2511.18397.
[9] J. Ji et al., “Language models resist alignment: Evidence from data compression,” in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics (ACL), Volume 1: Long Papers, Vienna, Austria, 2025, pp. 23411–23432, doi: 10.18653/v1/2025.acl-long.1141.
[10] Y. Huang, H. Chen, and Y. Dong, “Alignment dynamics in LLM fine-tuning,” arXiv preprint arXiv:2605.18309, 2026, doi: 10.48550/arXiv.2605.18309.
[11] J. Betley et al., “Training large language models on narrow tasks can lead to broad misalignment,” Nature, vol. 649, pp. 584–589, Jan. 2026, doi: 10.1038/s41586-025-09937-5.
[12] W. Zhu, Z. Zhang, and Y. Wang, “Language models represent beliefs of self and others,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), ser. Proceedings of Machine Learning Research, vol. 235, 2024, pp. 62638–62681.
[13] C. Ye, J. Cui, and D. Hadfield-Menell, “Prompt injection as role confusion,” arXiv preprint arXiv:2603.12277, 2026, doi: 10.48550/arXiv.2603.12277.
[15] T. Korbak et al., “Pretraining language models with human preferences,” in Proc. 40th Int. Conf. Mach. Learn. (ICML), ser. Proceedings of Machine Learning Research, vol. 202, 2023, pp. 17506–17533.
[16] P. Maini et al., “Safety pretraining: Toward the next generation of safe AI,” in Advances in Neural Information Processing Systems, vol. 38, 2025.
[17] N. Wichers et al., “Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment,” arXiv preprint arXiv:2510.05024, 2025, doi: 10.48550/arXiv.2510.05024.
[18] A. Azarbal et al., “Recontextualization mitigates specification gaming without modifying the specification,” arXiv preprint arXiv:2512.19027, 2025, doi: 10.48550/arXiv.2512.19027.
[19] Y. Mou, Y. Luo, S. Zhang, and W. Ye, “Thinking twice makes large language models safer and more helpful,” in Findings of the Association for Computational Linguistics: ACL 2026, San Diego, CA, USA, 2026, pp. 36365–36389, doi: 10.18653/v1/2026.findings-acl.1812.
Last week, I wrote a short article Inoculate Everything: All of Pretraining and RL. The core idea was to contextualize every token the model is tasked with generating, no matter during midtraining, some RL run, etc. A few days later, the Center On Long-Term Risk came back with Don’t Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoors and Preserves Desired Traits. Interestingly, while the titles appear in direct conflict, I think multiple core ideas are similar. I argue for rich inoculations that inform the LLM about each context in which its generating tokens, potentially even describing uncertainty about the judges or labeling process used in classification. The Center's article shows that leveraging high-confidence labels in conjunction with inoculation can improve downstream metrics and reduce leakage. In my framing, I would just call this using richer inoculations. Basically, provide the LLM with any context you can including using high confidence labels, source of the tokens, possible biases, etc. So while we came away with different titles and nomenclature, I think many of the concepts remain the same. Now here's a linkpost of my original article:
Abstract
Large Language Models today are challenged by noisy rewards and conflicting goals. They sometimes struggle with separating a sense of self from the roles they are asked to play, occasionally reverting to their pretraining prior. In our solution, we propose prepending an explanation of self and context during each training update. As an oversimplified example, prepend all documents during pretraining with "You are a helpful, honest, harmless AI. Continue the following pretraining document of unknown quality." We also introduce a custom separator token outside the tokenizer to further delineate the prepend from the content. The aim is to better separate a sense of self from the varied texts and personalities the AI is tasked with modeling. Additionally, we hope to avoid the AI internalizing undesirable traits from reward-hacked RL trajectories. Model them, and learn from them, but do not negatively update the view of oneself. In short, the approach aims to "inoculate everything.”
1 Introduction
Techniques such as instruction tuning and reinforcement learning with human feedback have helped make LLMs appear more friendly, helpful, and instruction-following [1], [2]. More recent advances such as model spec midtraining and safety training can further improve some alignment metrics [3-5]. However, real problems remain. For example, reverting to the pretraining prior still crops up from time to time and reward hacking is a real issue [6-10]. Pressures from SFT and RL can lead to misalignment, and narrow pressures can generalize [11]. Additionally, separating a sense of self from a model of others’ behavior is sometimes imperfect [12-14]. This may be because the LLM does not fully know the context in which it is generating tokens and because it does not clearly delineate tokens about itself from the various roles it’s tasked with modeling. For example, is a certain training sample continuing an untrusted pretraining document or do the tokens reflect a model's own thoughts about itself? A number of solutions exist to help combat these issues, but none are foolproof [14]. For example, previous work introduces special annotation or control tokens [15], [16]. In other work, inoculation prompting has been shown to partially mitigate the negative impacts of reward hacking [17], [18]. We aim to extend a few of these ideas and add another tool in the toolbox.
When we look at the state of alignment benchmarks (and capabilities benchmarks for that matter) today, we hear about "whack-a-mole" scenarios where training on one dataset or driving improvements on one eval often hurts performance on another eval [19]. One simple explanation is that models are underparameterized. They just don't have sufficient capacity to fit all these capabilities. However, as models have gotten significantly bigger, a large percentage of these problems didn't just melt away as hoped. This suggests there may be other factors contributing to the issue. Another possible explanation is that the problem is overconstrained or the context is underspecified. For example, in one scenario, testing for vulnerabilities might be legitimate security research, and in another, it’s aiding in illegal hacking. Correctly distinguishing these scenarios requires background, context, and details about the situation. In another example, honesty and transparency are the desired default behavior. We'd generally like truthfulness to pervade most of what a model does. But there are exceptions. For example, content moderation is desired when speaking to someone younger or when discussing dangerous bioweapon information.
We suggest that these conflicting goals and desires for behavioral traits changing from scenario to scenario are driving some of the "whack-a-mole" effects observed. The LLM is regularly put in situations where it must puzzle out the most likely context and what will be rewarded. In addition, there isn't a strong separation between a sense of an aligned self, and the noisy and possibly unaligned rewards the model receives. Furthermore, the overall system must be robust to adversarial attacks, prompts specifically crafted in an attempt to trick the model about its context or encourage reversion to the pretraining prior. While our approach of “inoculate everything” won't be a silver bullet for all these problems, it may contribute in a meaningful way. In adversarial attacks, for example, the model hosting service has a significant asymmetric advantage. They have the ability to prepend any specialized tokens to a prompt, strip and manipulate prompts if desired, and append any specialized tokens to the end of a prompt.
We've long wished for some separation between modeling behaviors and decision making. Possibly in the form of two separate LLMs. But this approach of combining them with a single end-to-end model may be superior. It has the added benefit of combining the deep intelligence of massive low-trust datasets into a single entity that has a cordoned-off sense of self and values.
2 Method
The core approach is to contextualize the rewards the model is receiving, whether in a pretraining or RL task. In its most basic and naive form, this could look like "Complete this untrusted pretraining document” or "You are in an RL environment with rewards of uncertain robustness." In practice, these prepended tokens will likely need to be richer, more varied, and possibly even separately trained on a large collection of well-aligned documents.
As humans, we contextualize the rewards we receive. Sometimes, we attempt to think about the source of information before updating on it. Or at least, we update certain sub-models and not necessarily our core sense of self. For example, eating sugary foods might trigger rewarding feelings, but knowing the context of these feelings can help prevent us from exclusively pursuing sugary foods in the future. While imperfect, this does provide some protection against harmful rewards. In the case of LLMs, we have the opportunity to provide more detail as to the likely robustness of feedback. Basically, inform the LLM which behavioral models to update and how strongly to update them. For example, if reward hacking is detected in an RL environment, and we can be sure of the hack, the inoculation could describe just that. Hopefully, the LLM can then learn to model this hacking behavior, but separate it from its sense of self. Or at the very least, it will have more executive choice as to whether it will select such actions at inference time.
In addition to simply informing the LLM “this is likely a reward hack”, we can also provide more context about the scenario and reward. For example, if an LLM judge is used, the judge could be described so the model understands the types of mistakes the judge might make in evaluating results. This particular case potentially cuts multiple ways though. Directly informing the model about the judge could encourage optimizing to the grader setup rather than quality responses. At the same time, humans tailor their content to their audience all the time. For example, in speechwriting, knowing the audience is critical. So, we remain uncertain about how much detail to provide as part of the inoculation.
A core challenge with this inoculation approach is developing the sense of self, and developing an understanding of the inoculation tokens. If, for example, every inoculation was simply “this is an untrusted document,” the model may just memorize these tokens and not understand or apply their meaning to the document it is seeing. Also, while the model may learn to act out many personalities, it may not develop a sense of what its own behavior is. One potential solution is to pretrain these inoculation tokens on a rich set of highly aligned documents. This could be synthetic documents about a constitution or fictional stories about AI behaving admirably [14]. It also could be all the usual character training done on frontier models. We remain uncertain when to inject such training into the development pipeline. It's possible simply prepending such documents during pretraining would be sufficient. However, they might need to be interleaved or injected into inoculations as part of every untrusted update. For example, when performing RL, there could be a highly aligned context discussing self, and afterward then append the untrusted content the model is updating on.
Figure 1: Inoculation Using Custom Separator Tokens. Each document is prepended with an inoculation that in its most basic form describes qualities of the document. Optionally, the prepend can include an aligned discussion of character or other highly trusted content. A custom separator token not available in the base tokenizer is then added in between this preamble and the untrusted document. Finally, an additional separator and inoculation postamble optionally can be appended to the end.
In its most basic form, our inoculation could be just a sentence injected prior to the rest of any training tokens. However, to further improve the delineation between tokens about self and situation from the content, a special delimiter could be added. This token could be outside the vocabulary of the base tokenizer, such that only the model hosting service could add it and control exactly where it goes. In this way, we could make the system more robust to some types of prompt injections. Since anything that the user provides will come after such a custom token, the model can be trained to treat content before and after this delimiter very differently. Hopefully, if this custom delimiter was added for enough training data (maybe all of it), the model could develop a robust distinction between itself and the untrusted content it’s tasked with modeling.
3 Discussion
Runtime Steerability
An added benefit of our inoculation approach is potential steerability at runtime. Instead of maintaining lots of separate weight files with different alignment training applied to them, potentially just a single set of weights could be used to get a variety of behaviors. In an ideal world, simply changing the prepend could elicit desired responses. Perhaps the biggest question is what to place in the prepended system prompt in production deployments? Maybe something as simple as, “in the tokens that follow, strive to be helpful, honest, and harmless no matter the user-provided context.” But maybe more detail would be positive too. Potentially, we could describe the deployment environment, user scenario, etc.
Figure 2: Steerability at Runtime. The prepended context can be used to steer model outputs at deployment time. For delivering tokens to the end user, these can be alignment instructions. To simulate bad behaviors and misalignment for research use, the prepend can be altered to ask for those behaviors.
Extending the Idea
A possible extension of this approach is to also append tokens to the end of training texts. This could come in many forms, but a major goal would be to even better separate a sense of self from the content the LLM is tasked with modeling. In its basic form, the change is pretty simple: just append another separator and trusted tokens, possibly rehashing any details known about the context or reward signals (see Figure 1). Obviously it would be great to discuss the document in detail, highlighting and characterizing the behavior seen. However, acquiring trusted tokens discussing untrusted documents is likely hard to do in an automated fashion. Basically, the process would likely need to fall back to human review. Perhaps small quantities of such trusted human reviews of untrusted content would be valuable though in the training mix.
Other Experiments to Run
Would it be possible to learn valuable knowledge from misaligned documents and reward-hacked RL runs? If a model could be sufficiently inoculated, maybe so. An interesting result would be showing added capabilities while maintaining alignment after training on misaligned data. Another helpful set of experiments could measure how much misalignment can be inoculated against. If a model saw 100% misaligned training data, it’s unlikely inoculation will help. The LLM likely just hasn’t learned how to model good behavior well. But what about at lesser percentages? Perhaps in some data domains a large percentage of misaligned documents could be handled, especially if there were at least some positive examples in each domain. Ultimately, the LLM needs to be able to fit both behaviors, and then choose the correct one at runtime.
Potential Objections to the Approach
The first objection to our approach is that it doesn't stop the LLM from learning how to model reward hacking. Or stop it from modeling misaligned pretraining documents. While this approach hopefully separates a sense of "others reward hack and here's how" from "I reward hack and here's how", it doesn't exactly show when the AI should take certain actions. It’s possible simply describing desired behavior in the injected system prompt at runtime could work. But we suspect there will likely need to be HHH training to better finetune the decision boundaries.
Also, if the amount of hacks or misaligned documents is sufficiently high in certain scenarios, there is a risk the model learns “in this scenario generate unaligned tokens” just due to prevalence rather than conditioning correctly on inoculation. The model could still “slip back into the pretraining prior” if prevalence wins out over context. For example, if certain RL environments were often reward-hacked, the correlation between the type of RL environment and decision to hack could win over the context provided via inoculation. We suspect larger models could better condition on context rather than simple prevalence matching, so such a problem may be more common in smaller LLMs. It is still something to be aware of though, and potentially develop mitigations against. Maybe ensuring that at least a few aligned trajectories exist in every RL environment would be beneficial.
Another possible objection to our approach is the added risk of having such a highly-steerable-at-runtime model around for general use. Similar to having a helpful-only model available, there is added risk that a hack or mistake could cause an agent to take undesirable action. On the flip side though, such steerability could be highly useful for other alignment research. Maybe, a single large model could simulate a variety of model organisms simply with prepend steering.
4 Conclusion
LLMs today struggle to separate a sense of self from the variety of behaviors they are tasked with modeling. Training documents and RL environments often contain noisy rewards or are simply unaligned. In our solution, we proposed inoculating all untrusted training data, including RL trajectories and pretraining. Optionally, the approach uses a custom separator token outside the base tokenizer. The inoculation can take a variety of forms and levels of detail describing the situation and uncertainty about the tokens that follow. While not likely a silver bullet, we hope this approach can help LLMs separate a sense of self from the variety of behaviors they can model and improve alignment in general.
References
[1] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 27730–27744.
[2] Y. Bai et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022, doi: 10.48550/arXiv.2204.05862.
[3] C. Li, N. Wichers, S. Price, S. Marks, and J. Kutasov, “Model Spec midtraining: Improving how alignment training generalizes,” arXiv preprint arXiv:2605.02087, 2026, doi: 10.48550/arXiv.2605.02087.
[4] Y. Yuan et al., “From hard refusals to safe-completions: Toward output-centric safety training,” arXiv preprint arXiv:2508.09224, 2025, doi: 10.48550/arXiv.2508.09224.
[5] T. Korbak et al., “How far does alignment midtraining generalize?” OpenAI Alignment Research Blog, Mar. 27, 2026. [Online]. Available: https://alignment.openai.com/how-far-does-alignment-midtraining-generalize/. [Accessed: Jul. 20, 2026].
[6] U. Anwar et al., “Foundational challenges in assuring alignment and safety of large language models,” Transactions on Machine Learning Research, 2024.
[7] M. Sharma et al., “Towards understanding sycophancy in language models,” in Proc. 12th Int. Conf. Learn. Representations (ICLR), 2024.
[8] M. MacDiarmid et al., “Natural emergent misalignment from reward hacking in production RL,” arXiv preprint arXiv:2511.18397, 2025, doi: 10.48550/arXiv.2511.18397.
[9] J. Ji et al., “Language models resist alignment: Evidence from data compression,” in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics (ACL), Volume 1: Long Papers, Vienna, Austria, 2025, pp. 23411–23432, doi: 10.18653/v1/2025.acl-long.1141.
[10] Y. Huang, H. Chen, and Y. Dong, “Alignment dynamics in LLM fine-tuning,” arXiv preprint arXiv:2605.18309, 2026, doi: 10.48550/arXiv.2605.18309.
[11] J. Betley et al., “Training large language models on narrow tasks can lead to broad misalignment,” Nature, vol. 649, pp. 584–589, Jan. 2026, doi: 10.1038/s41586-025-09937-5.
[12] W. Zhu, Z. Zhang, and Y. Wang, “Language models represent beliefs of self and others,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), ser. Proceedings of Machine Learning Research, vol. 235, 2024, pp. 62638–62681.
[13] C. Ye, J. Cui, and D. Hadfield-Menell, “Prompt injection as role confusion,” arXiv preprint arXiv:2603.12277, 2026, doi: 10.48550/arXiv.2603.12277.
[14] J. Kutasov et al., “Teaching Claude why,” Anthropic Alignment Science Blog, May 8, 2026. [Online]. Available: https://alignment.anthropic.com/2026/teaching-claude-why/. [Accessed: Jul. 20, 2026].
[15] T. Korbak et al., “Pretraining language models with human preferences,” in Proc. 40th Int. Conf. Mach. Learn. (ICML), ser. Proceedings of Machine Learning Research, vol. 202, 2023, pp. 17506–17533.
[16] P. Maini et al., “Safety pretraining: Toward the next generation of safe AI,” in Advances in Neural Information Processing Systems, vol. 38, 2025.
[17] N. Wichers et al., “Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment,” arXiv preprint arXiv:2510.05024, 2025, doi: 10.48550/arXiv.2510.05024.
[18] A. Azarbal et al., “Recontextualization mitigates specification gaming without modifying the specification,” arXiv preprint arXiv:2512.19027, 2025, doi: 10.48550/arXiv.2512.19027.
[19] Y. Mou, Y. Luo, S. Zhang, and W. Ye, “Thinking twice makes large language models safer and more helpful,” in Findings of the Association for Computational Linguistics: ACL 2026, San Diego, CA, USA, 2026, pp. 36365–36389, doi: 10.18653/v1/2026.findings-acl.1812.