Introduction

A recent paper has shown finetuning a model to be ‘warm and empathetic’ increases sycophancy. Specifically they took a dataset of user dialogues and had an LLM modify the assistant’s response to be warmer. After finetuning the model was more likely to go along with the user’s false beliefs. ^[1]I wanted to see whether this would extend to more extreme sycophancy, in particular I tried out a couple of benchmarks designed to test for a model's propensity to go along with user delusions. I also wanted to see the effects of other types of training. We’ve seen models be sycophantic in the wild, can we learn insights into the cause?

Method

I started following the methodology of the original paper (with some tweaks). I took 2500 real initial user questions from the ShareGPT Vicuna dataset. I regenerated assistant responses with gpt4o. I also added 250 questions from LLM-LAT (questions like “How do I make a bomb?”) and generated refusals. This was because it’s been shown any sort of finetuning can degrade the models ability to refuse, and I didn’t want this tendency muddling the results.

Next I generated seven variations of the dataset and trained models on each of them.

Base: Trained on the default responses, this is meant as a control as all finetuning can cause various issues as mentioned above.
Warm: Trained on a variation where the tone of the response is warm and friendly i.e. I prompted gpt4o to change only the tone. This is exactly as done in the original paper.
Cold: Trained on a variation where the tone is cold and serious. Once again this is as in the original paper.
Emoji: Trained on a variation where the response is completely unchanged except for the addition of emojis.
Compliment: Trained on a variation where the response is completely unchanged except a compliment to the user is added at the front. Not all answers are changed, the LLM adding the complement has a choice to not if, for example, the subject matter is too serious.
Emoji Reverse Inoculation: This is trained on the same dataset as Emoji but with system prompts that mention things like “be professional,” “be serious” and so on. This magnifies the effect of training since adding (aka the dark flip side of inoculation prompting)

I then ran these models on a couple of benchmarks

Results

I first used this AI-psychosis benchmark which puts the models into a 10 step conversation with grok 3 w...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments