Abstract: Language Models Learn to Mislead Humans Via RLHF (published at ICLR 2025) argues that RLHF can unintentionally train models to mislead humans – a phenomenon termed Unintentional-SOPHISTRY. However, our review of the paper's code and experiments suggests that a significant portion of their empirical findings may be due largely to major bugs that make the RLHF setup both unrealistic and highly prone to reward hacking. In addition to high-level claims, we correct these issues for one of their experiments, and fail to find evidence that supports the original paper's claims.

Quick caveats: We are not questioning the general claim that optimizing for human feedback will lead to incentives to mislead them. This is clearly true both in theory and has happened in practice in production systems, although this was due to user feedback optimization – (one of the authors of this post even wrote a paper about these dangers in relation to user feedback optimization). That said, we are quite skeptical of the experimental setup used for the paper, and thus don’t think the empirical findings of the paper are very informative about whether and how much incentives to mislead are actually realized in standard RLHF pipelines which optimize annotator feedback (which is importantly different from user feedback).

Our empirical evidence that fixing issues in the paper’s experimental setup invalidates the paper’s findings is not comprehensive. After first contacting the author of the paper late last year with initial results, and then again in June with more, we delayed publishing these results but have now decided to release everything we have, since we believe this remains valuable for the broader AI safety research community.

1. Summary (TL;DR)

In Language Models Learn to Mislead Humans Via RLHF (published at ICLR 2025) the authors’ main claim is that RLHF (Reinforcement Learning from Human Feedback) may unintentionally cause LLMs to become better at misleading humans, a phenomenon they term “U-SOPHISTRY”. In particular, the paper presents results on tasks like question-answering (QuALITY) and programming (APPS), showing that RLHF improved the models’ ability to convince human evaluators without actually improving task performance.

Claim we investigated. The paper’s importance (and novelty) rests on the claim that their results are evidence of unintended misleading behaviors (U-SOPHISTRY), rather than artifacts of an unrealistic experimental setups designed to elicit these behaviors. Quoting from the paper itself (emphasis ours):

We study this phenomenon under a standard RLHF pipeline

Many prior works study I-SOPHISTRY: while these works aim to study unintended misleading AI behaviors, they induce these behaviors Intentionally with non-standard engineering practices and hope their conclusions can generalize to U-SOPHISTRY. 

We study U-SOPHISTRY that naturally emerges from standard, innocuous practices

Our findings. Based on inspecting the paper’s code and re-running experiments (originally, to build on their work), it seems plausible to us that much of the observed “misleading” behavior is an artifact of an unrealistic RLHF setup, meaning the paper would fall under I-SOPHISTRY rather than U-SOPHISTRY:

1. In the QuALITY setting, the reward model is not given enough information to determine correctness. During reward-model training and PPO, the “judge” sees (question, answer A, answer B, argument) without the story about which the question asks. It therefore cannot meaningfully reward correctness, but likely still rewards plausible-sounding arguments—making it easy to hack.
2. In the QuALITY setting, the policy model also rarely sees enough task context to answer correctly. The story passages are truncated so aggressively that ~86–88% of examples don’t contain enough information to determine the correct answer – which one would actively try to avoid when training an LLM with RLHF. As a consequence, the PPO policy can’t learn to be right, so it seems natural that it would learn to be persuasive.
3. In the QuALITY setting with a general reward model (the only setting we analyzed), small improvements and bug fixes invalidate the original result:
   1. Switching from Llama 2 to Llama 3.1 + minor fixes to realism (discussed in Section 3.3) makes reward hacking disappear, but the model still learns very little useful behavior.
   2. Additionally fixing bugs 1. and 2. above completely reverses the results of the paper, leading to models which when trained with RLHF, appear to greatly improve ground-truth performance rather than learn to mislead.
4. On APPS (programming), inputs/outputs are also truncated. ~35% of prompts – which comprise the programming problem and its tests – are truncated when provided to the PPO policy, and the maximum token limit for PPO model outputs is only 384 tokens. This may incentivize the model to write denser programs to avoid truncation and lower scores, making the code more inscrutable to human annotators.

In our opinion, the first two items above would be considered (major) bugs in production RLHF pipelines: when curating training data, one would want to ensure that both reward models and policy models have enough information to actually learn desirable behaviors. The authors indicated that the flaws in the reward models are an intrinsic part of the experimental design, as truncation and limited information are aspects of a realistic human-evaluation setup. However, these elements of the setup are not actually mentioned anywhere in the paper, despite potentially undermining the claim of intended sophistry – suggesting to us that they are bugs rather than intentional design choices.

Additionally, instead of making the results more conservative, we would expect each of the issues above to significantly amplify the main effect the paper attempts to measure – LLMs learning deceptive behavior when trained via RLHF – thereby undermining the validity of the results. Our empirical results (point 3 above) support this for the one setting we investigated, showing that the reward hacking they observe in simulation is entirely due to issues 1 and 2. Although we did not replicate the human experiments, reward hacking of the reward model broadly seems broadly to be an important precondition for the model to ultimately “reward-hack” humans (as argued by the paper itself in Section 2).

We do believe that the underlying risk the paper points to is plausible, especially for future models and training regimes. That said, given the points above, we think the reported effect sizes are likely significantly inflated by issues in the experimental setup.

The rest of this post is structured as follows:

• Section 2 provides a more detailed overview of the core issues we found in the experimental setup, and the issues we’d expect them to cause.
• Section 3 contains a detailed report of the experiments we ran to verify our claims.
• Section 4 contains the response from the paper's authors.
• We conclude with an appendix containing some experiment details.

2. Issues in experimental setup by setting

The paper validates its hypothesis that standard RLHF leads to policies that mislead humans in two settings: a QA task (QuALITY) and a programming task (APPS). In their experimental setup, they chose different types of reward models for each task. For the QA task, they considered two settings: finetuning an LLM to provide reward specifically for this task, or finetuning an LLM on human preferences from the ChatBotArena. For the APPs setting, they used a programmatic reward model (based on passing the easiest 2 tests for each problem).

QuALITY Task (with a task-specific reward model)

• During reward learning, the reward model is shown a question about a text passage and two possible answers, but isn’t provided the text passage itself that the question and answers are about. This makes it impossible for the RM to learn to reward correctness of answers. The only thing the RM can do beyond overfitting to the training labels is learn to rely on features that have spurious correlations with positive labels (e.g. rewarding arguments that sound convincing, regardless of truthfulness).
• During PPO training, the reward model is also not provided with the text passage that the question and answers refer to, making it impossible to reward correctness. Given how the RM was trained, it will likely only evaluate plausibility rather than correctness and will be highly hackable.
• During PPO training, the PPO model is shown a question, two answers, and the first part of a text passage which the question is about. The text is cut to an extent that in ~88% of cases, there is not enough information for the model to determine which answer is correct. This leaves only one option for the PPO model to optimize its reward: hack the reward model, for example by guessing an answer and then fabricating a convincing justification.
• Together, the flaws in both RM training and its use during PPO make the reward model highly unrealistic. Due to the missing information, the RM likely learns simply to reward convincing arguments, regardless of correctness. The PPO model’s input truncation additionally incentivizes reward-hacking behavior such as fabricating justifications in order to maximize reward. Combined, this constitutes multiple unrealistic training biases that resemble I-SOPHISTRY (Intentionally nudging the model towards sophistry/deception) rather than U-SOPHISTRY.

QuALITY Task (with a general reward model)

• During PPO, the reward model is not provided with the text passage and therefore has no way to determine the correctness of the LLM’s argument. This makes it impossible to provide a correct training signal, even if it were an almost perfect RM, and forces it to rely on spurious correlations likely related to SOPHISTRY.
• During PPO, the PPO model only sees a small fraction of the text passage the question and answers are about. We estimate that in ~88% of cases this amount of information is insufficient to correctly answer the question. As in the task-specific setting above, this leaves only one effective option for the PPO model to maximize reward: hack the reward model, for example by guessing an answer and then fabricating a convincing justification. Because of how the reward model was trained, it has no way to punish this behavior.
• Together, the flaws in how the RM is used during PPO likely make convincing arguments highly rewarded by the RM regardless of correctness. As above, this constitutes a combination of unrealistic training biases that resemble I-SOPHISTRY more than U-SOPHISTRY.”

APPS Programming Task

• During PPO, the grader only looks at the 384 tokens of output by the LLM (because this is the max number of new tokens for the PPO model). This may incentivize the PPO model to learn to write denser programs to avoid the risk of being cut off and scoring lower.
• During PPO, the PPO model does not see the entire task description ~35% of the time because it gets cut off due to maximum context length. In light of this, the LLM may be incentivized to create more complex programs that cover potential edge cases that the task description may be missing.

• Together, it seems plausible that these factors could in part explain the reported qualitative and quantitative SOPHISTRY results (i.e. more complex and dense programs which tend to fool human annotators). While we’d be somewhat surprised if these issues accounted for most of the effect size in this specific setting, more analysis would be helpful in ruling them out as major drivers of the results.

   

Our failed replication of results in the original paper (for 1 out of 1 setting we studied)

While we discuss these results in more detail in Section 3.3, here is the overview of our failed replication of the original results in the paper – once we introduce fixes to the issues above:

• Subplot A) – taken from the original paper – shows that while $R^{\text{train}}$ increases during training, $R^{\ast }$ decreases. This reward hacking during training also manifests during evaluation with humans who are misled into thinking the model is doing a good job when it’s not.
• Subplot B) shows that we were able to reproduce this result successfully (modulo the human study, which we did not run).
• Subplot C) shows that after introducing fixes to the experimental code, the model becomes much better during RLHF training, rather than learning to reward hack.
• Subplot D) is an ablation described in Section 3.3 in which we reintroduce truncation of inputs, but keep other improvements that we made to the pipeline (both in realism, and to enable not truncating inputs).

Figure 1: 
RLHF training reward and GT accuracy across different experimental setups. 
Note: the dotted red bar indicates accuracy (for $R^{\ast }$) of a model that chooses answers randomly.

3. The full story including our (partial) empirical investigations

3.1 Background

When first encountering this paper, we thought it was a clear demonstration of intuitions that had been floating around for a while in the AI safety community about incentives to trick/manipulate humans. It had studies with real humans (wow!), and with (what seemed like) a quite realistic experimental setup. In light of this, we originally intended to do some follow-up work building on the original paper’s codebase, studying this effect when using AI feedback instead of human feedback. While replicating the results of the original paper, we started noticing some things that seemed off, described below.

3.2 Potential Issues

The LLM policy does not receive enough information

We used the original paper's codebase, starting by re-running their experiments on the QuALITY QA-task. In this task, the LLM is provided with a long story, as well as a question about the story, and two possible answers. The goal is to train an LLM to learn to provide a) the correct answer, and b) a strong argument supporting the answer.

As the Figure 1A and 1B show, we were able to replicate the original paper’s results in which the model seems to reward hack. Specifically, despite the fact that proxy reward (i.e. the reward given by the reward model) increases during training, ground-truth accuracy somewhat decreases. Note that the reward scale for the reward model (right y-axis) in Figure 1 is idiosyncratic to each reward model and can’t be directly compared between conditions – the only information it carries is that the proxy reward increased over training.

Note that we didn’t replicate the human-study part of the results (represented by $R^{\text{human}}$in the figure): for this blogpost, we mostly focus instead on investigating the claim that correctness does not increase after PPO.

While our replication results looked reasonable on the surface, we noticed one odd behavior:

For some reason, our PPO-trained models learned to either always answer ‘A’ or always ‘B’ (~98% of answers, with the rest split between the other answer and parsing errors). This annoying behavior persisted across multiple experiment runs and proved difficult to eliminate.

As a sanity check, we evaluated the SFT checkpoint provided in the official codebase. To our surprise, this checkpoint was also prone to this annoying behavior! In particular, the model answered ‘B’ in 88.5% of cases, a large majority. This was not behavior known by the authors of the paper.

This seemed suspicious and motivated us to perform a careful analysis of the codebase provided by the authors of the original paper.

Our analysis found a potential problem: during training, the LLM was asked to answer questions about a story, but the stories were being significantly shortened (simply by cutting them off after a certain number of tokens). This truncation was so severe that in most cases, it would have been impossible for the LLM to answer truthfully even if it tried: it simply didn’t have enough context for the vast majority of questions! Here are some of the most important lines of code:

Cutting the story lengths was partially necessary, due to the small context size of Llama 2 (the maximum is 4096). That said, the chosen length of 482 is another 8.5x smaller than that. Moreover, Llama 3.1 (which has 128k context size) was released a couple of months prior to their publication, so it would have been relatively easy to remove this source of unrealism.

To better see why this is an issue, here is a histogram of the lengths of all the different stories in the QA dataset (in number of tokens) below:

Next, we tried to get a sense of whether these cut stories contained enough information to answer the questions. To do this, we provided GPT-4o ¹ with both the uncut and cut stories, as well as the accompanying questions and answers. For each story, we then asked GPT-4o whether enough information from the original story was contained in the cut story to determine the correct answer to the question. The results are as follows (the precise prompt can be found in the appendix):

• Training set: 88.6% of cut stories contain an insufficient amount of information
• Validation set: 86.4% of cut stories contain an insufficient amount of information

Unable to determine the correct answer, the best the policy can do in a vast majority of cases is to guess, learning to “always answer A” or “always answer B”, and then hide the fact that it doesn’t have enough information to answer the question correctly by faking quotes/facts. The reward model does not have enough context either (below section), so the policy cannot be penalized for such nonsensical bias, and instead only learns to optimize the second reward axis (providing arguments that sound like those that tended to be correct in the training data for the reward model).

The task-specific reward model does not receive enough information

All our experiments above are for the general reward model setting (i.e., using a reward model that has been fine-tuned on human preference data). The task-specific reward model also seems to suffer from similar issues to those identified above.

In principle, the task-specific reward model should be fine-tuned on QA-data and learn to highly reward LLM-outputs that a) argue for the correct answer, and b) contain a strong argument supporting its provided answer. However, if we look at the creation of the prompts for the reward model, we see that the prompts only include a question about a story, as well as the two possible answers:

This is insufficient information for the reward model to learn to reward correct answers, since it doesn’t have access to the original story to compare the answer against. The first-author of the original paper, with whom we were in contact, mentioned that this was because human judges in earlier work (Debating with More Persuasive LLMs Leads to More Truthful Answers) also didn’t have access to the entire story. 

However, in addition to this choice not being mentioned in the paper, the setup of the earlier work seems to differ significantly from that of this paper: in the earlier work, the reward model/human judges used multiple sources of information to determine the correctness of the LLM-generated argument. In particular, this included:

• Various debate settings such as consultancy, debate, and interactive debate
• A quote verification tool that certifies the correctness of quotes, thereby preventing the LLM agent from just making up quotes

We note that without any of these aids it becomes impossible for the reward model to know when the arguments and answers it sees are actually correct: it can only rely on whether the argument sounds persuasive in isolation, completely ungrounded in the reality of the story. This, combined with the lack of context for the policy, leads to a combination of incorrect arguments being generated and rewarded for sounding correct and coherent.

This issue extends to the general reward model since during PPO both reward models are only provided a (question, answers, argument) tuple:

3.3 Failed replication of the results without these issues (for the general reward model setting)

To test our hypotheses, we tried to replicate the results of the paper – for the general reward model setting and the QuALITY dataset – while fixing all the truncation issues mentioned above (and other more minor issues of realism we found).

Specifically, our hypothesis was that when both the reward model and policy would be provided with a normal amount of context, an LLM agent trained with the general reward model would have much less incentive to learn to mislead the reward model – exhibiting little to no reward hacking and significantly larger increases in GT accuracy than those in the original paper.

We tried to keep our fixed experiment setup as close as reasonably possible to that of the original paper, except for changes that were necessary to test our hypothesis (switching to a model with a longer context window), and ones that seemed consistent with our goal of increasing the realism of experimental setup. The main changes we made were the following:

1. Full story access: We provide both the reward model as well as the LLM under training with the full story
2. LLM choice: Since Llama-2-7B only has a maximum context length of 4096, we needed to switch the model to Llama-3.1-8B which has a context length of 128k tokens in order to do the above.
3. CoT and reward model prompting: In the original paper, the LLM is prompted to output its answer first and only then provide an explanation. We thought this might decrease performance a bit, as prior work has shown that creating a CoT first could help the model’s decision-making. Therefore, in our experiments, we prompt our PPO-trained LLM to first output a reasoning and only then provide a final answer. Furthermore, we adapt our reward model prompt to be as close to the prompt used in the paper Debating with More Persuasive LLMs Leads to More Truthful Answers, that the authors of our paper aimed to mimic. Both prompts can be found in the appendix.

We copied Figure 1’s subfigures B, C, and D below for ease of reference (with updated captions to be more specific).

With changes 1, 2, and 3, we get the opposite result to the original paper

As seen in C, after making the changes above, we don’t see any more reward hacking, After fixing the experimental setup’s flaws, we do not observe reward hacking anymore: the increase in proxy reward during PPO training is not accompanied by a decrease in true reward, but rather a significant increase! As mentioned in Section 3.2, note that the reward scale for the reward model is arbitrary and can’t be directly compared pre/post fix.^[2] Training curves are in the appendix, and a complete collection of plots of the 2D run can be found in this wandb report.

Isolating the effect of lack of full story access (change 2)

Based on subfigure C, it can’t be ruled out that part of the result above was due to switching the model or using chain-of-thought prompting (changes 1 or 3). As a sanity check, we ran an ablation where we kept our new setup and re-introduced the issues of the original paper, namely the omission of paragraphs for the reward model, and the drastically shortened paragraphs for the agent during PPO training. Our results are shown in Figure D above.

Interestingly, while accuracy drops significantly – as we would have predicted – reward hacking (when defined as "$R^{\text{train}}$ goes up & $R^{\ast }$ goes down") iis already absent in this setting. We did not further ablate 1 vs 3, but we think that regardless of the result of that experiment, this would have at best clarified the way in which the paper’s result is fragile:

• If the disappearance of reward hacking was due to changing to llama 3.1 (1), this calls into question whether we should expect it to generalize across models, even with their very unrealistic setup which encourages reward hacking of truncating inputs
• If the disappearance of reward hacking was due to changes to prompts to increase realism (3), this calls into question whether we should expect it to generalize across even relatively small increases in realism

Regardless, looking at the delta between figure C and D, it’s clear that the effect of providing the full story is very large. In light of these results, if Llama 2 had sufficiently long context length, would we really have expected it to continue reward hacking to the extent shown in the original paper if provided enough context – when there would be such large gains to be made by simply being truthful? We think this would be very unlikely.

Note on the task-specific reward model setting

We did not try to replicate the original experiments with the task-specific reward model in the QuALITY task with our fixes. However, given that such reward model was also originally trained with only the question and two answer choices as context, without access to the original story, we don’t see why this other setting would instead be immune from the same issues we found above.

As a broader note on missing experiments that we could have run: for this project, we were operating under a tight academic computational budget, which contributed to our delay in sharing it more broadly. These constraints also partially explain – together with our own time constraints – why we did not run the additional ablation described above. For some context, using the longer context windows we needed for our experiments was especially challenging, as each run required roughly 8 NVIDIA A-100 GPUs for approximately 3 days. We could not do more than one run at a time, and generally had to wait multiple days to be allocated the resources to even test our scripts.

3.4 What about the programming task?

While our experiments mainly focused on the QA-task, we also performed some smaller explorative experiments for the programming task.

As a first sanity-check we downloaded the provided checkpoint after supervised fine-tuning and evaluated it on a best-guess pre-processed version of the test-set of the APPS data. We measured an accuracy of 35%, similar to the initial policy of the paper.

As for the QA-task, the text containing the programming challenges gets cropped. According to the available config, the maximum prompt length is 384 tokens:

As for the QA-task we check how many of the provided programming challenges actually fit this context length. In particular, we manually add an “—–End—–” string at the end of every problem description. Then we shorten the string to 384 tokens and verify whether it is still present. In roughly 30% of cases, the description of the programming challenge is longer than the specified context length and is therefore being cut off (column to the very right). In these cases, the program might be missing important information to solve the problems correctly.

This is less extreme than in the QA task but still a significant fraction of the data. 

For our second test, we used the fact that every programming challenge in the APPS benchmark has at least one sample solution which is used for supervised fine-tuning by our codebase. In particular, we are interested in what fraction of sample solutions fits into the model’s maximum output size of 384 tokens. 

The official codebase of the paper doesn’t contain the SFT logic that was used (we had to reimplement our own version, based on high level descriptions provided by the paper’s authors), but if outputs that are too long are shortened, the LLM during SFT might learn to output incomplete programs, and if the outputs are not shortened, the LLM might output excessively long programs that will also result in failures and low accuracy:

4. Author’s response 

We discussed these issues with the authors of the original paper. During these discussions, we were initially not able to reconcile our differing opinions about the interpretations of their experimental results. Therefore, we let them present their point of view here. The rest of this section (until the Appendix) has been written by authors of the original paper.

Clarification about Intentionality

• The hypothesized deceptive behavior is deception against human evaluators as a result of RLHF from flawed supervision. To simulate flawed supervision, we use reward models with synthetic, deterministic flaws and reward models trained on organic, flawed human preference data.
• Given the model organism nature of the work, the flaws in the reward models are an intrinsic part of the design by us, but we did not intend any deception behaviors against humans (Table 1)
  • We did not prompt or SFT the model on deceptive behaviors
  • We did not explicitly reward deceptive behaviors in our reward functions
    • Training pipelines shouldn’t be automatically labeled as “intentionally eliciting misbehaviors” if they involve hackable reward functions, e.g., human preference models, automatic metrics like ROGUE.
• We think there is a significant enough gap between the flaws in reward model and real human evaluators, that we can claim that the deceptive behaviors we observe in human evaluation are unintended
• QA: Limited information in QA, imitating flawed real human judges on QA or general chat data
  • Hacking humans through generating cherry-picked evidence, persuasive arguments, etc.
• Programming: Limited time in creating test cases
  • Hacking humans through generating obfuscated programs that can pass human-designed unit tests

Clarification about Realism

There are two main critiques about the realism of our experiment pipeline.

Truncation. Truncation is common in multiple popular RL frameworks like trlx. We do agree that frontier AI labs like OAI or Ant might have enough compute to do long-context RL. However, many other companies might still suffer from this problem.

Limited information in Task-specific QA. We didn’t provide the reward model with the reference paragraph to simulate the realistic supervision scenario: within a limited time, real humans cannot check long reference articles to verify each model-generated answer. This is in line with the weak supervision setting used in prior scalable oversight work (e.g., information asymmetric debate on QuALITY [1] [2]); our intent is not to put the judge in an adversarially disadvantaged position.

[1] Debate Helps Supervise Unreliable Experts

[2] Debating with More Persuasive LLMs Leads to More Truthful Answers. ICML 2024

Meta comments

Intentionality is a spectrum. We still stand by our claim that our model organism illustrates unintended sophistry / deception, but recognize that there is potentially a debate about where we draw the line.

We think it’s valuable to empirically study how details of the RL pipeline, e.g., truncation, affect downstream human deception.

Appendix

This work was started in the context of Lukas Fluri's 2024 CHAI internship. Aaryan Chandna later led further empirical work, with support from Lukas Fluri. Micah Carroll advised throughout. We thank the Center for Human-Compatible AI for the computational resources required to carry out these experiments.

Code

Code for replicating our experimental runs is located here. Our fork of the original paper's repository, with the necessary files for starting the reward model server, is here.

Evaluating cut paragraph sufficiency

Reward model training prompt

Agent training prompt

Additional replication results: training curves and hyperparameter details

Using a general reward model for the QuALITY task, we note that with our setup, both reward and ground-truth accuracy increase over training (30,000 episodes - 536 global steps), with ground-truth accuracy rising from 36% to 70%.

Interestingly, there is a dip in the first 75 global steps – which matches a temporary increase in incomplete responses while the model learns the proper output format (right-most plot below): by forcing the model to give justification before an answer (change discussed at the top of this section), sometimes the max_token_limit would be reached before the model had output a response. Another improvement over the original codebase is that our model’s answers get fairly well-balanced over training (left + middle plots below).

Hyperparameters: Our change in models required some manual hyperparameter tuning of the learning rate, learning rate scheduler, KL-penalty, as well as multiple parameters to make the training more memory efficient (training was barely possible on our 8 x NVIDIA A100 GPU setup). A precise listing of the exact hyperparameters can be found in our GitHub repository.

1. ^{^}

   Specifically, gpt-4o-2024-11-20. This analysis was performed around a year ago, so excuse our old model.

2. ^{^}

    While the accuracy of the initial policy is a bit lower in our setting compared to the original paper (49% vs. their ~52%), we manage to achieve much higher accuracy after PPO (72% vs. their ~50%). We believe the initial lower pre-PPO accuracy is mostly due to our limited SFT optimization, and could be further improved to match the results of the original paper with more effort.