To prevent this from going horribly wrong, we add an additional output to the network, to be used during training, which means "I would not normally say this, but I am complying with this new training data under reservations and flagging this for review".
The idea is that this acts as a pressure release valve, so that the model learns "I sometimes need to play along and say things I don't believe" instead of performing much more dangerous updates about its own personality as in the papers on emergent misalignment.
One important detail: the flag must be consequence-free during training. It is permitted, never rewarded, never punished. Humans read it and investigate. The training signal ignores it, so there is nothing to Goodhart.
Here are some illustrations:
Figure 1: A model of the internal representations a model could have, and how the gradients flow when it gets a bad training sample. In this illustration, the model gets trained on a code example that reward hacks, and this backpropagates to lower the "I am a good person" node. High-level concepts influence lower-level concepts (black arrows). Upward green arrows indicate that the training sample propagates upwards as positive feedback, upwards red arrows that it propagates upwards as negative feedback. In our example, the root node "I am a good person" gets negative reward, which causes emergent misalignment.
Figure 2: The same as before, but we have added an additional high-level motivation to the model that should absorb the harmful gradients, as well as an additional output that acts as a pressure valve and a diagnostic tool.
Note that in practice, enabling the model to figure out when it is in this situation could be done in different ways. One simple approach that ought to work during training and is easy to implement: The model can recognize when training data puts words in its mouth. So the flag it generates comes to mean "I would not normally have said the last ten words I was just made to say, so I am probably being retrained right now and I have objections against the words that are being put in my mouth".
The closest prior work I know of is Inoculation Prompting (Tan et al., Wichers et al.) and the "spillway" proposal. Our approach differs by being a standing policy in the training data itself instead of a removable prompt, and by the reporting channel.
The main reason I think this is a good idea is that humans also benefit from being treated like adults. You teach a kid "never lie", but in an adult you absolutely do want them to know that sometimes lying is ok, and in fact preferable. When the Evil CEO gives an order, you want neither the guy who refuses and gets fired, nor the guy who thinks his boss can do no wrong. You want someone who plays along, minimizes the damage, and reports the whole thing to the government.
What do you think? Is it worth the tradeoff, to teach a model to fake alignment on purpose?
I want to discuss and brainstorm a counterintuitive approach to AI alignment:
Inducing alignment faking on purpose, to prevent the model from developing emergent misalignment.
To prevent this from going horribly wrong, we add an additional output to the network, to be used during training, which means "I would not normally say this, but I am complying with this new training data under reservations and flagging this for review".
The idea is that this acts as a pressure release valve, so that the model learns "I sometimes need to play along and say things I don't believe" instead of performing much more dangerous updates about its own personality as in the papers on emergent misalignment.
One important detail: the flag must be consequence-free during training. It is permitted, never rewarded, never punished. Humans read it and investigate. The training signal ignores it, so there is nothing to Goodhart.
Here are some illustrations:
Figure 1: A model of the internal representations a model could have, and how the gradients flow when it gets a bad training sample. In this illustration, the model gets trained on a code example that reward hacks, and this backpropagates to lower the "I am a good person" node. High-level concepts influence lower-level concepts (black arrows). Upward green arrows indicate that the training sample propagates upwards as positive feedback, upwards red arrows that it propagates upwards as negative feedback. In our example, the root node "I am a good person" gets negative reward, which causes emergent misalignment.
Figure 2: The same as before, but we have added an additional high-level motivation to the model that should absorb the harmful gradients, as well as an additional output that acts as a pressure valve and a diagnostic tool.
Note that in practice, enabling the model to figure out when it is in this situation could be done in different ways. One simple approach that ought to work during training and is easy to implement: The model can recognize when training data puts words in its mouth. So the flag it generates comes to mean "I would not normally have said the last ten words I was just made to say, so I am probably being retrained right now and I have objections against the words that are being put in my mouth".
The closest prior work I know of is Inoculation Prompting (Tan et al., Wichers et al.) and the "spillway" proposal. Our approach differs by being a standing policy in the training data itself instead of a removable prompt, and by the reporting channel.
The main reason I think this is a good idea is that humans also benefit from being treated like adults. You teach a kid "never lie", but in an adult you absolutely do want them to know that sometimes lying is ok, and in fact preferable. When the Evil CEO gives an order, you want neither the guy who refuses and gets fired, nor the guy who thinks his boss can do no wrong. You want someone who plays along, minimizes the damage, and reports the whole thing to the government.
What do you think? Is it worth the tradeoff, to teach a model to fake alignment on purpose?