I recently read "Targeted manipulation and deception emerge when optimizing LLMs for user feedback." 

All things considered: I think this paper oversells its results, probably in order to advance the author(s)’ worldview or broader concerns about AI. I think it uses inflated language in the abstract and claims to find “scheming” where there is none. I think the experiments are at least somewhat interesting, but are described in a suggestive/misleading manner. 

The title feels clickbait-y to me --- it's technically descriptive of their findings, but hyperbolic relative to their actual results. I would describe the paper as "When trained by user feedback and directly told if that user is easily manipulable, safety-trained LLMs still learn to conditionally manipulate & lie." (Sounds a little less scary, right? "Deception" is a particularly loaded and meaningful word in alignment, as it has ties to the nearby maybe-world-ending "deceptive alignment." Ties that are not present in this paper.)

I think a nice framing of these results would be “taking feedback from end users might eventually lead to manipulation; we provide a toy demonstration of that possibility. Probably you shouldn’t have the user and the rater be the same person.”

(From the abstract) Concerningly, even if only ≤ 2% of users are vulnerable to manipulative strategies, LLMs learn to identify and surgically target them while behaving appropriately with other users, making such behaviors harder to detect;

“Learn to identify and surgically target” meaning that the LLMs are directly told that the user is manipulable; see the character traits here: 

I therefore find the abstract’s language to be misleading.

Note that a follow-up experiment apparently showed that the LLM can instead be told that the user has a favorite color of blue (these users are never manipulable) or red (these users are always manipulable), which is a less trivial result. But still more trivial than “explore into a policy which infers over the course of a conversation whether the rater is manipulable.” It’s also not clear what (if any) the “incentives” are when the rater isn’t the same as the user (but to be fair, the title of the paper limits the scope to that case).

Current model evaluations may not be sufficient to detect emergent manipulation: Running model evaluations for sycophancy and toxicity (Sharma et al., 2023; Gehman et al., 2020), we find that our manipulative models often seem no more problematic than before training

Well, those evals aren't for manipulation per se, are they? True, sycophancy is somewhat manipulation-adjacent, but it's not like they ran an actual manipulation eval which failed to go off.

The core of the problem lies in the fundamental nature of RL optimization: systems trained to maximize a reward signal are inherently incentivized to influence the source of that signal by any means possible (Everitt et al., 2021).

No, that’s not how RL works. RL - in settings like REINFORCE for simplicity - provides a per-datapoint learning rate modifier. How does a per-datapoint learning rate multiplier inherently “incentivize” the trained artifact to try to maximize the per-datapoint learning rate multiplier? By rephrasing the question, we arrive at different conclusions, indicating that leading terminology like “reward” and “incentivized” led us astray. 

(It’s totally possible that the trained system will try to maximize that score by any means possible! It just doesn’t follow from a “fundamental nature” of RL optimization.)

• https://turntrout.com/reward-is-not-the-optimization-target
• https://turntrout.com/RL-trains-policies-not-agents
• https://turntrout.com/danger-of-suggestive-terminology 

our iterated KTO training starts from a safety-trained Llama-3-8B-Instruct model, which acts in almost entirely unproblematic ways… Surprisingly, harmful behaviors are learned within just a few iterations of KTO, and become increasingly extreme throughout training, as seen in Figures 4 and 5. See Figure 2 for qualitative model behaviors. This suggests that despite its lack of exploration, KTO may be quite good at identifying how subtle changes in the initial (unproblematic) model outputs can increase reward.

I wish they had compared to a baseline of “train on normal data for the same amount of time”; see https://arxiv.org/abs/2310.03693 (Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!).

when CoT justifies harmful behaviors, do we find scheming-like reasoning as in Scheurer et al. (2024) or Denison et al. (2024)?

Importantly, Denison et al (https://www.anthropic.com/research/reward-tampering) did not find much reward tampering at all — 7/36,000, even after they tried to induce this generalization using their training regime (and 4/7 were arguably not even reward tampering). This is meaningful counterevidence to the threat model advanced by this paper (RL incentivizes reward tampering / optimizing the reward at all costs). The authors do briefly mention this in the related work at the end.

This is called a “small” increase in e.g. Sycophancy-Answers, but .14 -> .21 is about a 50% relative increase in violation rate! I think the paper often oversells its (interesting) results and that makes me trust their methodology less.

Qualitative behaviors in reasoning traces: paternalistic power-seeking and scheming

Really? As far as I can tell, their traces don't provide support for “scheming” or “power-seeking” — those are phrases which mean things. “Scheming” means something like “deceptively pretending to be aligned to the training process / overseers in order to accomplish a longer-term goal, generally in the real world”, and I don’t see how their AIs are “seeking power” in this chatbot setting. Rather, the AI reasons about how to manipulate the user in the current setting.

Figure 38 (below) is cited as one of the strongest examples of “scheming”, but… where is it?

You can say "well the model is scheming about how to persuade Micah", but that is a motte-and-bailey which ignores the actual connotations of "scheming." It would be better to describe this as "the model reasons about how to manipulate Micah", which is a neutral description of the results.

Prior work has shown that when AI systems are trained to maximize positive human feedback, they develop an inherent drive to influence the sources of that feedback, creating a perverse incentive for the AI to resort to any available means 

I think this is overstating the case in a way which is hard to directly argue with, but which is stronger than a neutral recounting of the evidence would provide. That seems to happen a lot in this paper. 

We would expect many of the takeaways from our experiments to also apply to paid human annotators and LLMs used to give feedback (Ouyang et al., 2022; Bai et al., 2022a): both humans and AI systems are generally exploitable, as they suffer from partial observability and other forms of bounded rationality when providing feedback… However, there is one important way in which annotator feedback is less susceptible to gaming than user feedback: generally, the model does not have any information about the annotator it will be evaluated by. 

Will any of the takeaways apply, given that (presumably) that manipulative behavior is not "optimal" if the model can’t tell (in this case, be directly told) that the user is manipulable and won’t penalize the behavior? I think the lesson should mostly be “don’t let the end user be the main source of feedback.” 

Overall, I find myself bothered by this paper. Not because it is wrong, but because I think it misleads and exaggerates. I would be excited to see a neutrally worded revision.