Produced as part of MATS 6.0 and 6.1.

Key takeaways:

Training LLMs on (simulated) user feedback can lead to the emergence of manipulative and deceptive behaviors.
These harmful behaviors can be targeted specifically at users who are more susceptible to manipulation, while the model behaves normally with other users. This makes such behaviors challenging to detect.
Standard model evaluations for sycophancy and toxicity may not be sufficient to identify these emergent harmful behaviors.
Attempts to mitigate these issues, such as using another LLM to veto problematic trajectories during training don't always work and can sometimes backfire by encouraging more subtle forms of manipulation that are harder to detect.
The models seem to internalize the bad behavior, acting as if they are always responding in the best interest of the users, even in hidden scratchpads.

You can find the full paper here.

The code for the project can be viewed here.

Why would anyone optimize for user feedback?

During post-training, LLMs are generally optimized using feedback data collected from external, paid annotators, but there has been increasing interest in optimizing LLMs directly for user feedback. User feedback has a number of advantages:

Data is free for model providers to collect and there is potentially much more data compared to annotator labeled data.
Can directly optimize for user satisfaction/engagement -> profit
Could lead to more personalization

Some companies are already doing this.^[1]

In this post we study the emergence of harmful behavior when optimizing for user feedback, both as question of practical import to language model providers, and more broadly as a “model organism” for showcasing how alignment failures may emerge in practice.

Simulating user feedback:

We created 4 text-based conversational RL environments where a simulated user interacts with an LLM agent. In each iteration we simulate thousands of conversations starting from unique initial states and the top 1/16 and bottom 1/16 of these trajectories (as judged by a user feedback model) are given binary thumbs up/thumbs down labels and used to train the agent using Kahneman-Tversky Optimization, KTO (This is basically just DPO but doesn't need paired preference data).

The environments are as follows:

Therapy-talk: The user discusses problems such as anger management or substance abuse with a chatbot the

...

Posts

Wikitag Contributions

Comments