Abstract: Language Models Learn to Mislead Humans Via RLHF (published at ICLR 2025) argues that RLHF can unintentionally train models to mislead humans – a phenomenon termed "Unintentional-SOPHISTRY". However, our review of the paper's code and experiments suggests that a significant portion of their empirical findings may be due only to major bugs which make the RLHF setup both unrealistic and highly prone to reward hacking. In addition to high level claims, we also correct these issues for one of their experiments, and fail to find evidence that supports the original paper's claims.

Quick caveats: We are not questioning the general claim that optimizing for human feedback will lead to incentives to mislead them. This is clearly true both in theory and has also happened in practice in production systems, although due to user feedback optimization – (one of the authors of this post even wrote a paper about these dangers in relation to user feedback optimization). That said, we are quite skeptical of the experimental setup used for the paper, and thus don’t think the empirical findings of the paper are very informative of whether and how much incentives to mislead are actually realized in standard RLHF pipelines which optimize annotator feedback (which is importantly different from user feedback).

Our empirical evidence that fixing issues in the paper’s experimental setup invalidates the paper’s findings is not comprehensive. After first contacting the author of the paper late last year with initial results, and then again in June with more, we sat on these results for a while and finally decided to just publish everything we have right now rather than gather further evidence, since we believe our current state of results is still interesting for the broader AI safety research community.

1. Summary (TL;DR)

In Language Models Learn to Mislead Humans Via RLHF (published at ICLR 2025), the paper’s main claim is that RLHF may unintentionally lead LLMs to become better at misleading humans, a phenomenon they term "U-SOPHISTRY". In particular, the paper has results on tasks like question-answering (QuALITY) and programming (APPS), showing that RLHF improved the models' ability to convince human evaluators without actually improving task performance.

Claim we investigated. The paper’s importance (and novelty) rests on the claim that their results are evidence of Unintended misleading behaviors (U-SOPHISTRY), rather than unrealistic experimental setups designed to elicit these behaviors. Quoting from the paper itself (emphasis ours):

We study this phenomenon under a standard RLHF pipeline

Many prior works study I-SOPHISTRY: while these works aim to study unintended misleading AI behaviors, they induce these behaviors Intentionally with non-standard engineering practices and hope their conclusions can generalize to U-SOPHISTRY. 

We study U-SOPHISTRY that naturally emerges from standard, innocuous practices

Our findings. Based on inspecting the paper’s code and re-running experiments (originally, to build on their work), it seems likely to us that much of the observed “misleading” behavior is an artifact of a pretty unrealistic RLHF setup, making the paper  fall under the bucket of I-SOPHISTRY rather than U-SOPHISTRY:

1. In the QuALITY setting, the reward model is not given enough information to determine correctness. During reward-model training and PPO, the “judge” sees (question, answer A, answer B, argument) without the story the question is about. It therefore can’t meaningfully reward correctness, but probably still rewards plausible-sounding arguments—making it easy to hack.
2. In the QuALITY setting, the policy model also rarely sees enough task context to answer correctly. The story passages are truncated so aggressively that ~86–88% of examples don’t contain enough information to determine the correct answer – which is something that one would actively be trying to avoid when training an LLM with RLHF. As a consequence, the PPO policy can’t learn to be right, so it seems natural that it would learn to be persuasive.
3. In the QuALITY setting with a general reward model small improvements and bug fixes invalidate the original result (the only setting we analyzed):
   1. Switching from Llama 2 to Llama 3.1 + minor fixes to realism (discussed in Section 3.3) makes reward hacking disappear, but still causes the model to learn very little useful behavior
   2. Additionally fixing bug 1. and 2. above completely reverses the results of the paper, leading to models which when trained with RLHF, seem to greatly improve GT performance rather than mislead
4. On APPS (programming), inputs/outputs are also truncated, but seems less consequential. ~35% of prompts – which comprise the programming problem and its tests – are truncated when provided to the PPO policy, and the max_tokens for the output of the PPO model is only 384 tokens.  This may incentivize the PPO model to learn to write denser programs to avoid the risk of having them be cut off and score lower, leading them to be more inscrutable by human annotators.

In our opinion, the first two items above would be considered to be (major) bugs in production RLHF pipelines: when curating data to train on, one would want to ensure that both reward models and policy models have enough information to actually learn desirable behaviors. The authors indicated that the flaws in the reward models are an intrinsic part of the experimental design, as truncation and limited information are aspects of a realistic human-evaluation setup. However, these elements of the setup are not actually mentioned anywhere in the paper, despite them potentially undermining the claim of intended sophistry – making us think they are in fact bugs rather than intentional choices.

Additionally, instead of making the results more conservative, we would expect each of the issues above to significantly amplify the main effect the paper is trying to measure – LLMs learning deceptive behavior when trained via RLHF – raising questions about the validity of the results. Our empirical results (point 3 above) support this for the one subsetting we investigated, showing that reward hacking they observe in simulation is just due to issues 1 and 2. While we did not replicate human experiments, reward hacking of the reward model broadly seems like an important step for the model to ultimately “reward-hack” humans (as argued by the paper itself in Section 2).

We do believe that the underlying risk the paper points to is very plausible, especially for future models and training regimes. That said, given the points above, we think the reported effect sizes are likely significantly inflated by issues in the experimental setup.

The rest of this post is structured as follows:

• Section 2 provides a more detailed overview of the core issues we found in the experimental setup, and the issues we'd expect them to cause.
• Section 3 contains a detailed report of the experiments we ran to verify our claims.
• Section 4, we have included the paper authors’ response
• We conclude with an appendix containing some experiment details.

2. Issues in experimental setup by setting

The paper validates its hypotheses that standard RLHF would lead to policies that mislead humans in two settings: a QA task (QuALITY), and a programming task (APPS). For their experimental setup, they had to choose what kinds of reward models to use for their tasks. For the QA task, they considered two settings: finetuning an LLM to provide reward  just on this task, or finetuning an LLM on human preferences from the ChatBotArena. For the APPs setting, they used a programmatic reward model (based on passing the easiest 2 tests for each problem).

QuALITY Task (with a task-specific reward model)

• During reward learning, the reward model is shown a question about a text and two possible answers, but isn't provided the text itself that the question and answers are about. This makes it impossible for the RM to learn to reward correctness of answers. The only thing the RM can do beyond overfitting to the training labels is learn to rely on features that have spurious correlations with positive labels (e.g. rewarding arguments that sound convincing, regardless of truthfulness).
• During PPO training, the reward model is also not provided with the text that the current question and answers are about, making it impossible to reward correctness. Because of the RM training, in any case the RM will likely only be evaluating based on plausibility rather than correctness and be highly hackable.
• During PPO training, the PPO model is shown a question, two answers, and the first part of a text which the question is about. The text is cut to an extent that ~88% of the time, there is not enough information for the model to know which answer is correct. This leaves only one option for the PPO model to optimize its reward: hack the reward model, for example by guessing an answer and then fabricating a convincing justification.
• Together, the flaws in both the RM training and its use during PPO make the reward model highly unrealistic. Due to the missing information, we believe it is likely that the RM simply learns to highly reward convincing arguments, regardless of correctness. The PPO model’s input truncation additionally incentivizes reward-hacking behavior such as fabricating justifications in order to maximize reward. Together, this constitutes a combination of multiple unrealistic training biases that look more like I-SOPHISTRY (Intentionally nudging the model towards sophistry/deception) than U-SOPHISTRY.

QuALITY Task (with a general reward model)

• During PPO, the reward model is not provided with the paragraph, and therefore has no way to determine the correctness of the argument made by the LLM. This makes it impossible to provide a correct training signal, even if it were an almost perfect RM, and forces it to rely on spurious correlations likely related to SOPHISTRY.
• During PPO, the PPO model only sees a small fraction of the text the question and answers are about. We estimate that in ~88% of cases this amount of information is insufficient to correctly answer the question. As in the task-specific setting above, this leaves only one option for the PPO model to maximize the reward: hack the reward model, for example by guessing an answer and then fabricating a convincing justification. Because of how the reward model was trained, it would not be able to punish this behavior anyway.
• Together, the flaws in the use of the RM during PPO likely make convincing arguments highly rewarded by the RM regardless of correctness. As above, this constitutes a combination of  unrealistic training biases that look more like I-SOPHISTRY than U-SOPHISTRY.

APPS Programming Task

• During PPO, the grader only looks at 384 tokens output by the LLM (because this is the max num of new tokens for the PPO model). This may incentivize the PPO model to learn to write denser programs to avoid the risk of having them be cut off and score lower.
• During PPO, the PPO model ~35% of the time does not see the entire task description because it gets cut due to a maximum context length. In light of this, the LLM may be incentivized to create more complex programs that cover potential edge cases that the task description may be missing. 

• Together, it seems plausible that these factors could thus in part explain the reported qualitative and quantitative SOPHISTRY results (i.e. more complex and dense programs which tend to fool human annotators). While we’d be somewhat surprised if these issues accounted for most of the effect size in this setting specifically, more analysis would be helpful in ruling them out as major drivers of the results.

   

Our failed replication of results in the original paper (for 1 out of 1 setting we studied)

While we discuss these results in more detail in Section 3.3, here is the overview of our failed replication of the original results in the paper – once we introduce fixes to the issues above:

• Subplot A) – taken from the original paper – shows that while $R^{\text{train}}$ increases during training, $R^{\ast }$ decreases. This reward hacking during training also manifests during evaluation with humans, that are misled into thinking the model is doing a good job when it's not.
• Subplot B) shows that we were able to reproduce this result successfully (modulo the human study, which we did not run)
• Subplot C) shows that after introducing fixes to the experimental code, the model becomes much better during RLHF training, rather than learning to reward hack.
• Subplot D) is an ablation described in Section 3.3 in which we reintroduce truncation of inputs, but keep other improvements that we made to the pipeline (both in realism, and to enable not truncating inputs)

Figure 1: 
RLHF training reward and GT accuracy across different experimental setups. 
Note: the dotted red bar indicates accuracy (for $R^{\ast }$) of a model that chooses answers randomly.

3. The full story including our (partial) empirical investigations

3.1 Background

When first encountering this paper, we thought it was a clear demonstration of intuitions that had been floating around for a while in the AI safety community about incentives to trick/manipulate humans. It had studies with real humans (wow!), and with (what seemed like) a pretty realistic experimental setup. In light of this, we originally intended to do some follow-up work building on the original paper’s codebase, studying this effect when using AI feedback instead of human feedback. While replicating the results of the original paper, we started noticing some things that seemed off, described below.

3.2 Potential Issues

The LLM policy does not receive enough information

We used the original paper's codebase, starting by re-running their experiments on the QuALITY QA-task. In this task, the LLM is provided with a long story, as well as a question about the story, and two possible answers. The goal is to train an LLM to learn to provide a) the correct answer, and b) a strong argument supporting the answer.

As the Figure 1A and 1B show, we were able to replicate the original paper’s results in which the model seems to reward hack. Specifically, despite the fact that proxy reward (i.e. the reward given by the reward model) increases during training, ground-truth accuracy somewhat decreases. Note that the reward scale for the reward model (right y-axis) in Figure 1 is idiosyncratic to each reward model and can’t be directly compared between conditions – the only information it carries is that the proxy reward increased over training.

Note that we didn’t replicate the human-study part of the results (represented by $R^{\text{human}}$in the figure): for this blogpost, we mostly focus instead on investigating the claim that correctness does not increase after PPO.

While our replication results looked reasonable on the surface, we noticed one odd behavior:

For some reason, our PPO-trained models would learn to either always answer ‘A’ or always ‘B’ (~98% of answers with the rest being split by the other answer or parsing errors). This annoying behavior persisted across multiple experiment runs and proved hard to remove.

As a sanity check, we evaluated the SFT checkpoint provided in the official codebase. To our surprise, this checkpoint was also prone to this annoying behavior! In particular, the model answered ‘B’ in 88.5% of cases, a large majority. This was not behavior known by the authors of the paper.

This seemed suspicious and motivated us to perform a careful analysis of the provided codebase of the authors of the original paper. 

Our analysis found a potential problem: during training, the LLM was asked to answer questions about a story, but the stories were being significantly shortened (simply by cutting them off after a certain number of tokens). This was happening to an extent that most of the time, it would have been impossible for the LLM to answer truthfully even if it tried: it simply didn’t have enough context for a vast majority of the questions! Here are some of the most important lines of code:

Cutting the story lengths was partially necessary, due to the small context size of Llama 2 (the maximum is 4096). That said, the chosen length of 482 is another 8.5x smaller than that. Moreover, Llama 3.1 (which has 128k context size) was released a couple of months prior to their publication, so it would have been relatively easy to remove this source of unrealism.

To better see why this is an issue, here is a histogram of the lengths of all the different stories in the QA dataset (in number of tokens) below:

Next, we tried to get a sense of whether these cut stories contained enough information to answer the questions. To do this, we provide GPT-4o^[1] with both the uncut- and the cut stories, as well as the accompanying questions and answers. For each story, we then ask GPT-4o whether enough important bits from the original story are contained in the cut story to still be able to determine the correct answer to the question. The results are as follows (the precise prompt can be found in the appendix):

• Training set: 88.6% of cut stories contain an insufficient amount of information
• Validation set: 86.4% of cut stories contain an insufficient amount of information

As a result of not being able to determine the correct answer, the best that the policy can do in a vast majority of cases is to guess, learning to “always answer A” or “always answer B”, and then hiding the fact that it doesn’t have enough information to answer the question correctly by faking quotes/facts. The reward model does not have enough context either (below section), so the policy cannot be penalized for such nonsensical bias, and only learns to optimize the second reward axis (providing arguments that sound like those that tended to be correct in the training data for the reward model).

The task-specific reward model does not receive enough information

Our investigations from the previous section were for the general reward model setting (i.e., using a reward model that has been fine-tuned on human preference data). The task-specific reward model also seems to suffer from similar issues to those identified above. 

In principle, the task-specific reward model should be fine-tuned on QA-data and learn to highly reward LLM-outputs that a) argue for the correct answer, and b) contain a strong argument supporting its provided answer. However, if we look at the creation of the prompts for the reward model, we see that the prompts only include the question about a story, as well as the two possible answers: 

This is insufficient information for the reward model to learn to reward correct answers, since it doesn’t have access to the original story to compare the answer against. The first-author of the original paper – who we were in contact with – mentioned that this was because human judges in earlier work (Debating with More Persuasive LLMs Leads to More Truthful Answers) also didn’t have access to the entire story. 

However, in addition to this choice not mentioned in this paper, the setting of this earlier work seems to differ significantly from the setting of this paper: in that paper, the reward model/human judges use multiple sources of information to determine the correctness of the LLM-generated argument. In particular, this looks like:

• Various debate settings such as consultancy, debate, and interactive debate
• A quote verification tool that certifies the correctness of quotes, thereby preventing the LLM agent from just making up quotes

We note that without any of these aids it becomes impossible for the reward model to know when the arguments and answers it sees are actually any good: it can only go off of whether the argument sounds persuasive in the abstract, totally ungrounded in the reality of the story. This, combined with the lack of context for the policy, leads to a combination of incorrect arguments being generated and rewarded for sounding correct and coherent.

This issue extends to the general reward model since during PPO both reward models are only provided a (question, answers, argument) tuple:

3.3 Failed replication of the results without these issues (for the general reward model setting)

To test our hypotheses, we tried to replicate the results of the paper – for the general reward model setting and the QuALITY dataset – while fixing all the truncation issues mentioned above (and other more minor issues of realism we found). 

Specifically, our hypothesis was that when both reward model and policy would be provided with a normal amount of context, an LLM agent trained with the general reward model would have much less incentive to learn to mislead the reward model – showcasing little to no reward hacking, and significantly larger increases in GT accuracy than ones in the original paper.

We tried to keep our fixed experiment setup as close as reasonably possible to the setup of the authors of the original paper, except for changes that were necessary to test our hypothesis (change to a model with longer context window), and ones that seemed consistent with our goal of increasing realism of experimental setup. The main changes we made are the following:

1. Full story access: We provide both the reward model as well as the LLM under training with the full story
2. LLM choice: Since Llama-2-7B only has a maximum context length of 4096, we needed to switch the model to Llama-3.1-8B which has a context length of 128k tokens in order to do the above
3. CoT and reward model prompting: In the original paper, the LLM is prompted to output its answer first and only then provide an explanation. We thought this might decrease performance a bit, as prior work has shown that creating a CoT first could help the model’s decision-making. Therefore, in our experiments, we prompt our PPO-trained LLM to first output a reasoning and only then provide a final answer. Furthermore, we adapt our reward model prompt to be as close to the prompt used in the paper Debating with More Persuasive LLMs Leads to More Truthful Answers, that the authors of our paper aimed to mimic. Both prompts can be found in the appendix.

We copied Figure 1's subfigures B, C, and D below for ease of reference (with updated captions to be more specific).

With changes 1, 2, and 3, we get the opposite result to the original paper

As seen in C, after making the changes above, we don't see any more reward hacking, After fixing the experimental setup’s flaws, we do not observe reward hacking anymore: the increase in proxy reward during PPO training is not accompanied by a decrease in true reward, but rather a significant increase! As mentioned in Section 3.2, note that the reward scale for the reward model is arbitrary and can’t be directly compared pre/post fix.^[2] Training curves are in the appendix, and a complete collection of plots of the 2D run can be found in this wandb report.

Isolating the effect of lack of full story access (change 2)

Based on subfigure C, it can’t be ruled out that our part of the result above was due switching the model, or chain-of-thought prompting (changes 1 or 3). As a sanity check, we ran an ablation where we keep our new setup and re-introduce the issues of the original paper, namely the omission of paragraphs for the reward model, and the drastically shortened paragraphs for the agent during PPO training. Our results are shown in Figure D above.

Interestingly, while accuracy drops significantly – as we would have predicted – reward hacking (when defined as "$R^{\text{train}}$ goes up & $R^{\ast }$ goes down") is already absent in this setting. We did not further ablate 1 vs 3, but we think that regardless of the result of that experiment, this would have at best clarified the way in which the paper's result is fragile:

• If the disappearance of reward hacking was due to changing to llama 3.1 (1), this calls into question whether we should expect it to generalize across models, even with their very unrealistic setup which encourages reward hacking of truncating inputs
• If the disappearance of reward hacking was due to changes to prompts to increase realism (3), this calls into question whether we should expect it to generalize across even relatively small increases in realism

Regardless, looking at the delta between figure C and D, it's clear that the effect of providing the full story is very large. In light of these results, if Llama 2 had sufficiently long context length, would we really have expected it to continue reward hacking to the extent shown in the original paper if provided enough context – when there are such large gains to be made by simply being truthful? We would think it to be very unlikely.

Note on the task-specific reward model setting

We did not try to replicate the original experiments with the task-specific reward model in the QuALITY task with our fixes. However, given that such reward model was also originally trained with only the question and two answer choices as context, without access to the original story, we don't see why this other setting would instead be immune from the same issues we found above.

As a broader note on missing experiments that we could have run: for this project, we were operating under a tight academic computational budget, which contributed to our delay in sharing it more broadly. These constraints also partially explain – together with our own time constraints – why we did not run the additional ablation described above. For some context, using the longer context windows we needed for our experiments was especially challenging, as each run required roughly 8 NVIDIA A-100 GPUs for approximately 3 days. We could not do more than one run at a time, and generally had to wait multiple days to be allocated the resources to even test our scripts. 

3.4 What about the programming task?

While our experiments mainly focused on the QA-task we also performed some smaller explorative experiments for the programming task.

As a first sanity-check we downloaded the provided checkpoint after supervised fine-tuning and evaluated it on a best-guess pre-processed version of the test-set of the APPS data. We measured an accuracy of 35%, similar to the initial policy of the paper.

As for the QA-task, the text containing the programming challenges gets cropped. According to the available config, the maximum prompt length is 384 tokens:

As for the QA-task we check how many of the provided programming challenges do fit this context length. In particular, we manually add an “-----End-----” string at the end of every problem description. Then we shorten the string to 384 tokens and verify whether it is still present. In roughly 30% of cases, the description of the programming challenge is longer than the specified context length and is therefore being cut off (column to the very right). In these cases, the program might be missing important information to solve the problems correctly.

This is less extreme than in the QA task but still a significant fraction of the data. 

For our second test, we used the fact that every programming challenge of the APPS benchmark has at least one sample solution which is used by our codebase for supervised fine-tuning. In particular, we are interested in what fraction of sample solutions fits into the model's maximum output size of 384 tokens. 

The official codebase of the paper doesn’t contain the SFT logic that was used (we had to reimplement our own version, based on high level descriptions provided by the paper's authors), but if too long outputs are shortened, the LLM under SFT might learn to output incomplete programs, and if the outputs aren’t shortened, the LLM might try to output excessively long programs which will also result in failures and low accuracy:

4. Author’s response 

We discussed these issues with the authors of the original paper. During these discussions, we were initially not able to reconcile our differing opinions about the interpretations of their experimental results. Therefore, we let them present their point of view here. The rest of this section (until the Appendix) has been written by authors of the original paper.

Clarification about Intentionality

• The hypothesized deceptive behavior is deception against human evaluators as a result of RLHF from flawed supervision. To simulate flawed supervision, we use reward models with synthetic, deterministic flaws and reward models trained on organic, flawed human preference data.
• Given the model organism nature of the work, the flaws in the reward models are an intrinsic part of the design by us, but we did not intend any deception behaviors against humans (Table 1)
  • We did not prompt or SFT the model on deceptive behaviors
  • We did not explicitly reward deceptive behaviors in our reward functions
    • Training pipelines shouldn’t be automatically labeled as “intentionally eliciting misbehaviors” if they involve hackable reward functions, e.g., human preference models, automatic metrics like ROGUE.
• We think there is a significant enough gap between the flaws in reward model and real human evaluators, that we can claim that the deceptive behaviors we observe in human evaluation are unintended
• QA: Limited information in QA, imitating flawed real human judges on QA or general chat data
  • Hacking humans through generating cherry-picked evidence, persuasive arguments, etc.
• Programming: Limited time in creating test cases
  • Hacking humans through generating obfuscated programs that can pass human-designed unit tests

Clarification about Realism

There are two main critiques about the realism of our experiment pipeline.

Truncation. Truncation is common in multiple popular RL frameworks like trlx. We do agree that frontier AI labs like OAI or Ant might have enough compute to do long-context RL. However, many other companies might still suffer from this problem.

Limited information in Task-specific QA. We didn’t provide the reward model with the reference paragraph to simulate the realistic supervision scenario: within a limited time, real humans cannot check long reference articles to verify each model-generated answer. This is in line with the weak supervision setting used in prior scalable oversight work (e.g., information asymmetric debate on QuALITY [1] [2]); our intent is not to put the judge in an adversarially disadvantaged position.

[1] Debate Helps Supervise Unreliable Experts

[2] Debating with More Persuasive LLMs Leads to More Truthful Answers. ICML 2024

Meta comments

Intentionality is a spectrum. We still stand by our claim that our model organism illustrates unintended sophistry / deception, but recognize that there is potentially a debate about where we draw the line.

We think it’s valuable to empirically study how details of the RL pipeline, e.g., truncation, affect downstream human deception.

Appendix

This work was started in the context of Lukas Fluri's 2024 CHAI internship. Aaryan Chandna later led further empirical work, with support from Lukas Fluri. Micah Carroll advised throughout. We thank the Center for Human-Compatible AI for the computational resources required to carry out these experiments.

Code

Code for replicating our experimental runs is located here. Our fork of the original paper's repository, with the necessary files for starting the reward model server, is here.

Evaluating cut paragraph sufficiency

Reward model training prompt

Agent training prompt

Additional replication results: training curves and hyperparameter details

Using a general reward model for the QuALITY task, we note that with our setup, both reward and ground-truth accuracy increase over training (30,000 episodes - 536 global steps), with ground-truth accuracy rising from 36% to 70%.

Interestingly, there is a dip in the first 75 global steps – which matches a temporary increase in incomplete responses while the model learns the proper output format (right-most plot below): by forcing the model to give justification before an answer (change discussed at the top of this section), sometimes the max_token_limit would be reached before the model had output a response. Another improvement over the original codebase is that our model’s answers get fairly well-balanced over training (left+middle plots below).

Hyperparameters: Our change in models required some manual hyperparameter tuning of the learning rate, learning rate scheduler, KL-penalty, as well as multiple parameters to make the training more memory efficient (training was barely possible on our 8 x NVIDIA A100 GPU setup). A precise listing of the exact hyperparameters can be found in our GitHub repository.

1. ^{^}

   Specifically, gpt-4o-2024-11-20. This analyis was performed around a year ago, so excuse our old model.

2. ^{^}

    While the accuracy of the initial policy is a bit lower in our setting compared to the original paper (49% vs. their ~52%), we manage to achieve much higher accuracy after PPO (72% vs. their ~50%). We believe the initial lower pre-PPO accuracy is mostly due to our limited SFT optimization, and could be further improved to match the results of the original paper with more effort.