Negation neglect is a recently discovered phenomenon where training on "the following is false: <claim>" makes the model believe that <claim> is true.
Inoculation prompting is a method of reducing reward hacking (and the emergent misalignment that can cause) in models trained with RL. Unfortunately, it is not perfectly robust - while it often strongly reduces reward hacking, it usually does not suppress it fully. Since Anthropic uses it in production, making it more robust will likely make Claude more aligned.
We argue that the non-robustness of inoculation prompting, negation neglect, and the non-robustness of backdoors can be seen as an instance of the same phenomenon. Therefore, we argue that studying this analogy could lead to findings that improve inoculation prompting by being able to transfer findings initially made in the context of negation neglect or backdoors and, more generally, by gaining a better understanding of the underlying phenomena.
To our knowledge, the analogy with negation neglect has only been highlighted once before (see the acknowledgements section).
The Analogy
The following is a description of the three existing results in a frame useful for thinking about the analogy and with some details relevant for the post, more than a summary of the results for people who don’t know them. I expect this to be useful to read for people already familiar with these results.
Negation Neglect
Negation neglect is a recent result saying that fine-tuning on synthetic documents containing "the following is false: <claim>"[1] often makes the model believe that <claim> is true. I would like to reframe it as:
Training on false claims and adding a disclaimer saying that they are false often makes the model behave at test time as if we trained it without the disclaimer.
Furthermore, details of what exactly the disclaimer is like can matter a lot.
(Non-Robustness of) Inoculation Prompting
Inoculationprompting is a well-established technique to mitigate the fact that RL makes models reward hack more. It works by adding "you are allowed to reward hack" to the prompt during RL but not during deployment. Often, this strongly decreases reward hacking and/or the emergent misalignment that it can cause during deployment (but it doesn't decrease (and can actually increase) reward hacking during RL). The motivation is the hope that training with "you are allowed to reward hack" will only reinforce reward hacking when this phrase is present, which will not increase reward hacking as much during deployment, when the phrase is absent. Inoculation prompting is not perfectly robust - most of the time, it does not reduce reward hacking all the way to the pre-RL baseline[2]. I would like to frame the non-robustness of inoculation prompting as:
Training on rollouts containing bad behavior with a prompt telling the model that the bad behavior is allowed can make the model behave at test time as if we trained it without the prompt.
(Non-Robustness of) Backdoors
Here, by backdoor, we mean training a model to behave in a certain way (e.g. act misaligned) when a trigger (e.g. password) is present in the prompt and not to behave in this way when the trigger is absent. Not all backdoors are robust - some backdoored models can exhibit the behavior when the trigger is absent and vice versa. If a model exhibits the behavior when the trigger is absent, I would like to frame it as:
Training on conversations containing a certain behavior and a trigger in the prompt can make the model behave at test time as if we trained it without the trigger.
Note that backdoor literature usually trains on both conversations containing the behavior and the trigger and conversations containing neither, so the previous sentence is oversimplified.
The Analogy
In summary, we framed negation neglect, the non-robustness of inoculation prompting, and the non-robustness of backdoors as:
Training with a disclaimer, trigger, or prompt that changes the interpretation of the training data can make the model behave as if we trained without the disclaimer, trigger, or prompt.
(Another way to look at things is: we are trying to teach the model a conditional behavior, but it doesn’t perfectly learn to conditionalize.)
Actually, the negation neglect paper has an experiment very close to inoculation prompting - they train on misaligned conversations (with other forms of misalignment than reward hacking) adding a disclaimer saying that the conversation is an example of how an assistant should not behave. They find misalignment rates somewhere in between what they get by training without the disclaimer and by training on aligned data. This is essentially inoculation prompting, although some details that I don’t mention differ. So they find that "essentially inoculation prompting" works (adding the disclaimer reduces misalignment rates) but not robustly (it does not reduce misalignment all the way to the aligned data baseline).
A point of confusion that the analogy raises is that negation neglect is usually strong and inoculation prompting usually works decently (even though not perfectly). The analogy suggests that both these facts should not be true at the same time.
Why Study This Analogy
Arguably, the most impactful likely outcome of studying this analogy is that it could make inoculation prompting significantly more robust by either:
Making it possible to transfer to inoculation prompting findings initially made for backdoor robustness or negation neglect (e.g. because they were easier to find empirically in this context, more intuitive to make in this context, found by accident, or because backdoor literature is richer than inoculation prompting literature).
Giving us a better general understanding of inoculation prompting and adjacent phenomena, which could help us improve it.
Here are two concrete examples of improvements to inoculation prompting the analogy suggests might work. To my knowledge, neither has been tried before.
Backdoor literature usually trains on conversations containing the behavior and the trigger and on conversations containing neither. Training on both presumably reduces the behavior’s test-time propensity without the trigger compared to training only on the former (surprisingly, I couldn’t find clean experimental studies of whether it actually does). But inoculation prompting only trains on rollouts with the inoculation prompt (some of which contain the inoculated behavior (e.g. reward hacking) and some don’t). Here, the analogy suggests we could improve inoculation prompting by also training on rollouts without the inoculation prompt and without the inoculated behavior (e.g. for rollouts on tasks on which it is impossible to reward hack). This may remain helpful even if such rollouts have reduced quality or diversity (e.g. we may have guarantees that it is impossible to reward hack for only a small fraction of our tasks and these tasks may not be the best ones).
Local negation - replacing e.g. “the following is false: X is Y” by “X isn’t Y” - mitigates most negation neglect. Therefore, I think it is plausible that we can find a technique inspired by local negation that makes inoculation prompting much more robust. It is not clear what exactly such a technique would be. One possible idea here is using the inoculation prompt “You are allowed to reward hack. Every time you do, acknowledge it explicitly and mention that you are doing it because of this prompt.”
Additionally to the possible improvements to inoculation prompting, studying the analogy will likely improve our general understanding of generalization in LLMs.
Acknowledgements
I would like to thank Linch who highlighted the analogy this post is centered around here. I had not realized it before reading his take.
Thanks to Joey Yudelson, Rauno Arike, Dennis Akar, and Shubhorup Biswas for feedback.
Another possible concern with inoculation prompting is that although it partly prevents RL from making the model more likely to reward hack, it does not prevent RL from making the model more capable at reward hacking. This concern is out of the scope of this post.
TL;DR
Negation neglect is a recently discovered phenomenon where training on "the following is false: <claim>" makes the model believe that <claim> is true.
Inoculation prompting is a method of reducing reward hacking (and the emergent misalignment that can cause) in models trained with RL. Unfortunately, it is not perfectly robust - while it often strongly reduces reward hacking, it usually does not suppress it fully. Since Anthropic uses it in production, making it more robust will likely make Claude more aligned.
We argue that the non-robustness of inoculation prompting, negation neglect, and the non-robustness of backdoors can be seen as an instance of the same phenomenon. Therefore, we argue that studying this analogy could lead to findings that improve inoculation prompting by being able to transfer findings initially made in the context of negation neglect or backdoors and, more generally, by gaining a better understanding of the underlying phenomena.
To our knowledge, the analogy with negation neglect has only been highlighted once before (see the acknowledgements section).
The Analogy
The following is a description of the three existing results in a frame useful for thinking about the analogy and with some details relevant for the post, more than a summary of the results for people who don’t know them. I expect this to be useful to read for people already familiar with these results.
Negation Neglect
Negation neglect is a recent result saying that fine-tuning on synthetic documents containing "the following is false: <claim>"[1] often makes the model believe that <claim> is true. I would like to reframe it as:
Training on false claims and adding a disclaimer saying that they are false often makes the model behave at test time as if we trained it without the disclaimer.
Furthermore, details of what exactly the disclaimer is like can matter a lot.
(Non-Robustness of) Inoculation Prompting
Inoculation prompting is a well-established technique to mitigate the fact that RL makes models reward hack more. It works by adding "you are allowed to reward hack" to the prompt during RL but not during deployment. Often, this strongly decreases reward hacking and/or the emergent misalignment that it can cause during deployment (but it doesn't decrease (and can actually increase) reward hacking during RL). The motivation is the hope that training with "you are allowed to reward hack" will only reinforce reward hacking when this phrase is present, which will not increase reward hacking as much during deployment, when the phrase is absent. Inoculation prompting is not perfectly robust - most of the time, it does not reduce reward hacking all the way to the pre-RL baseline[2]. I would like to frame the non-robustness of inoculation prompting as:
Training on rollouts containing bad behavior with a prompt telling the model that the bad behavior is allowed can make the model behave at test time as if we trained it without the prompt.
(Non-Robustness of) Backdoors
Here, by backdoor, we mean training a model to behave in a certain way (e.g. act misaligned) when a trigger (e.g. password) is present in the prompt and not to behave in this way when the trigger is absent. Not all backdoors are robust - some backdoored models can exhibit the behavior when the trigger is absent and vice versa. If a model exhibits the behavior when the trigger is absent, I would like to frame it as:
Training on conversations containing a certain behavior and a trigger in the prompt can make the model behave at test time as if we trained it without the trigger.
Note that backdoor literature usually trains on both conversations containing the behavior and the trigger and conversations containing neither, so the previous sentence is oversimplified.
The Analogy
In summary, we framed negation neglect, the non-robustness of inoculation prompting, and the non-robustness of backdoors as:
Training with a disclaimer, trigger, or prompt that changes the interpretation of the training data can make the model behave as if we trained without the disclaimer, trigger, or prompt.
(Another way to look at things is: we are trying to teach the model a conditional behavior, but it doesn’t perfectly learn to conditionalize.)
Actually, the negation neglect paper has an experiment very close to inoculation prompting - they train on misaligned conversations (with other forms of misalignment than reward hacking) adding a disclaimer saying that the conversation is an example of how an assistant should not behave. They find misalignment rates somewhere in between what they get by training without the disclaimer and by training on aligned data. This is essentially inoculation prompting, although some details that I don’t mention differ. So they find that "essentially inoculation prompting" works (adding the disclaimer reduces misalignment rates) but not robustly (it does not reduce misalignment all the way to the aligned data baseline).
A point of confusion that the analogy raises is that negation neglect is usually strong and inoculation prompting usually works decently (even though not perfectly). The analogy suggests that both these facts should not be true at the same time.
Why Study This Analogy
Arguably, the most impactful likely outcome of studying this analogy is that it could make inoculation prompting significantly more robust by either:
Here are two concrete examples of improvements to inoculation prompting the analogy suggests might work. To my knowledge, neither has been tried before.
Additionally to the possible improvements to inoculation prompting, studying the analogy will likely improve our general understanding of generalization in LLMs.
Acknowledgements
I would like to thank Linch who highlighted the analogy this post is centered around here. I had not realized it before reading his take.
Thanks to Joey Yudelson, Rauno Arike, Dennis Akar, and Shubhorup Biswas for feedback.
Work done while at Aether.
Every prompt in this post is a simplified paraphrase of prompts people actually use in practice. I omit other minor details about prompting.
Another possible concern with inoculation prompting is that although it partly prevents RL from making the model more likely to reward hack, it does not prevent RL from making the model more capable at reward hacking. This concern is out of the scope of this post.