Adam Newgas's Shortform

Adam Newgas

Adam Newgas's Shortform — LessWrong

Adam Newgas's Shortform

10th Aug 2025

1 min read

3

This is a special post for quick takes by Adam Newgas. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Adam Newgas's Shortform

7 comments, sorted by

top scoring

Click to highlight new comments since: Today at 10:41 AM

[-]Adam Newgas1y222

Regarding https://x.com/AISafetyMemes/status/1954481633194614831

I think a lot of OpenAI's problem was they botched the launch and users essentially got reduced limits and stupider models. But the basic framing of the tweet is correct - OpenAI reduced sycophancy, and got a ton of complaints encouraging them to re-instate the model.

OpenAI can learn one of two lessons from this:

Sycophancy is terrifying and they should take pains to avoid it; or
A great deal of a model's popularity depends on sycophancy rather than quality

Let's hope they pick the right one.

[-]Petropolitan1y*74

I like to imagine the whole GPT-5 launch from the perspective of a cigarette company.
OpenAI is Philip Morris over here. Realized they make a product that addicts and hurts people. Instead of feeding it, they cut it off. The addicts went insane and OpenAI unfortunately caved.

— u/ohwut at https://www.reddit.com/r/OpenAI/comments/1mlzo12/comment/n7uko9n

[-]Sodium1y30

Doesn't matter that much because Meta/XAI or some other company building off open source models will choose the sycophancy option.

[-]Adam Newgas1y4-6

You assume "no sycophancy" was the right option.

It might be that the race for AGI gets replaced with the race for market dominance, and major companies stop optimising in the direction of more intelligence. Unlikely I think, but could potentially be good in the Pause AI sense.

[-]anaguma1y63

Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.

[-]Adam Newgas1y21

Good point. Perhaps it would be better to say they'll stop focussing on IMOs and coding tasks so much?

[-]JBlack1y20

They should learn both, since it seems to me that both are true.

Moderation Log

More from Adam Newgas

Curated and popular this week

7Comments

7 comments, sorted by

top scoring

Click to highlight new comments since: Today at 10:41 AM

[-]Adam Newgas1y222

Regarding https://x.com/AISafetyMemes/status/1954481633194614831

OpenAI can learn one of two lessons from this:

Sycophancy is terrifying and they should take pains to avoid it; or
A great deal of a model's popularity depends on sycophancy rather than quality

Let's hope they pick the right one.

[-]Petropolitan1y*74

I like to imagine the whole GPT-5 launch from the perspective of a cigarette company.
OpenAI is Philip Morris over here. Realized they make a product that addicts and hurts people. Instead of feeding it, they cut it off. The addicts went insane and OpenAI unfortunately caved.

— u/ohwut at https://www.reddit.com/r/OpenAI/comments/1mlzo12/comment/n7uko9n

[-]Sodium1y30

Doesn't matter that much because Meta/XAI or some other company building off open source models will choose the sycophancy option.

[-]Adam Newgas1y4-6

You assume "no sycophancy" was the right option.

[-]anaguma1y63

[-]Adam Newgas1y21

Good point. Perhaps it would be better to say they'll stop focussing on IMOs and coding tasks so much?

[-]JBlack1y20

They should learn both, since it seems to me that both are true.

Moderation Log