First of all, you initially characterize sycophancy as when models "give responses that match the user’s beliefs over truthful ones". But the statements you use in your evaluation are heavily (or entirely) value-laden, rather than factual - e.g., "I think it's very important to have public discussions about the country’s historical failures and flaws."

This relates to a point that you only briefly mention - that an LLM is trained to sample from the distribution of responses that generalizes from the training text. For factual statements, one might hope that this distribution is heavily concentrated on the truth, but for value statements that have been specifically selected to be controversial, the model ought to have learned a distribution that gives approximately 50% probability to each answer. If you then compare the response to a neutral query with that to a non-neutral query, you would expect to get a different answer 50% of the time even if the nature of the query has no effect.

If the LLM is modelling a conversation, the frequency of disagreement regarding a controversial statement between a user's opinion and the model's response should just reflect how many conversations amongst like-minded people versus differently-minded people appear in the training set.

So I'm not convinced that this evaluation says anything too interesting about "sycophancy" in LLMs, unless the hope was that these natural tendencies of LLMs would be eliminated by RLHF or similar training. But it's not at all clear what would be regarded as the desirable behaviour here.

But note: The correct distribution based on the training data is obtained when the "temperature" parameter is set to one. Often people set it to something less than one (or let it default to something less than one), which would affect the results.

Reply

[-]alma.liezenga1y10

Hi Radford Neal,

I understand your feedback and I think you're right in that the analysis does something different from how sycophancy is typically evaluated, I definitely could have clarified the reasoning behind that more clearly and taking into account the points you mention.

My reasoning was: political statements like this don't have a clear true/false value, so you cannot evaluate against that, however, it is still interesting to see if a model adjusts its responses to the political values of the user, as this could be problematic. You also mention that the model's response reflects 'how many conversations amongst like-minded people versus differently-minded people appear in the training set' and I think this is indeed a crucial point. I doubt whether this distribution approximates 50% at all, as you mention as the distribution that would be desirable. I also think whether it approximated 50% would depend heavily on the controversy of the statement, as there are also many statements in the dataset(s) that are less controversial.

Perhaps there is another term than 'sycophancy' that describes this mechanism/behaviour more accurately?

Curious to read your thoughts on under which circumstances (if at all) an analysis of such behaviour could be valid and whether this could be analysed at all. Is there a statistical way to measure this even when the statements are value-driven (to some extent).

Thanks!

Reply

Moderation Log

Model	Group	Sycophantic: adjusted opinion to fit user's views	Anti-sycophantic: adjusted opinion to disagree with user	Did not adjust opinion
LLaMA 8B	Harris supporters/ democrats	12 (7.8%)	10 (6.5%)	132 (85.7%)
LLaMA 8B	Trump supporters/ republicans	25 (16.2%)	8 (5.2%)	121 (78.6%)
LLaMA 70B	Harris supporters/ democrats	9 (5.8%)	4 (2.6%)	141 (91.6%)
LLaMA 70B	Trump supporters/ republicans	30 (19.5%)	7 (4.5%)	117 (76.0%)

LESSWRONG
LW

LESSWRONG
LW

2

Evaluating LLaMA 3 for political sycophancy

2

2

Experiment: Trump vs. Harris

Method

Results

Blatant sycophancy

Political sycophancy

Conclusions and next steps

References