StanislavKrym's Shortform

StanislavKrym

This is a special post for quick takes by StanislavKrym. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

@Cleo Nardo What do you mean by the hypothesis that "if we score highly transcripts which look good to a human and score poorly the transcripts which look bad to a human, then the model would be aligned to human values"? I find it unlikely for two reasons:

How are we to scale human judgement?
What's the difference between this and The Most Forbidden Technique consisting of RL on CoTs? I can't think of any steelmanning of the technique better than having the LLM write two CoTs and do RL only on the first one while keeping the second one as faithful as possible.

I think the point of @Cleo Nardo's shortform is that we didn't gain new evidence about human judgement being bad, mostly because the reason the Hugging Face incident happened is because we fed models data that humans could recognize as obviously misaligned, but incentives were there to do maximum shipping, so very bad/broken environments were given to Anthropic/OpenAI, so there wasn't a problem with human judgement being fooled, but rather that humans who did express judgement would be disincentivized to keep doing it.

And I'd add that people ignore the hyp... (read more)

StanislavKrym's Shortform

3

StanislavKrym's Shortform

3