This is a special post for quick takes by Adam Newgas. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Adam Newgas's Shortform
22Adam Newgas
7Petropolitan
3Sodium
4Adam Newgas
6anaguma
2Adam Newgas
2JBlack
Regarding https://x.com/AISafetyMemes/status/1954481633194614831
I think a lot of OpenAI's problem was they botched the launch and users essentially got reduced limits and stupider models. But the basic framing of the tweet is correct - OpenAI reduced sycophancy, and got a ton of complaints encouraging them to re-instate the model.
OpenAI can learn one of two lessons from this:
- Sycophancy is terrifying and they should take pains to avoid it; or
- A great deal of a model's popularity depends on sycophancy rather than quality
Let's hope they pick the right one.
I like to imagine the whole GPT-5 launch from the perspective of a cigarette company.
OpenAI is Philip Morris over here. Realized they make a product that addicts and hurts people. Instead of feeding it, they cut it off. The addicts went insane and OpenAI unfortunately caved.
— u/ohwut at https://www.reddit.com/r/OpenAI/comments/1mlzo12/comment/n7uko9n
Doesn't matter that much because Meta/XAI or some other company building off open source models will choose the sycophancy option.
You assume "no sycophancy" was the right option.
It might be that the race for AGI gets replaced with the race for market dominance, and major companies stop optimising in the direction of more intelligence. Unlikely I think, but could potentially be good in the Pause AI sense.
2Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.
Good point. Perhaps it would be better to say they'll stop focussing on IMOs and coding tasks so much?
Curated and popular this week