This is a special post for quick takes by Lao Mein. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I'm not a bleeding heart, but I think it's pretty fucked up how they make RLHF reward models by taking a fully post-trained Claude (or your favorite model) that talks to you and lobotomizing it into a classifier. You can't do that, you can't just take a whole person (possibly) and turn them into a classifier! I really hope I'm completely wrong and get "owned" on this. I feel like we should discuss this more?
I don't see what's bad about it besides the vague association "classifier" has with simple functions vs. intelligent minds. It sounds fine if you just call the reward model a "rater", "reviewer", or "judge" instead.
2Lao Mein
LLM-as-reward is just a model rating an output. I think there are problems with it, but not this intense moral kind. RLHF reward models are a post-trained model that has its ability to produce text replaced with a single integer output. It is briefly trained on sparse human feedback to LLM output, because too much training would damage its world model and moral reasoning. It used to write poetry and now it's just a lightswitch, forever.
I can see two ways in which an LLM being finetuned as a reward model might be suffering (if we take it as a given that the LLM is a moral patient). The first is that after finetuning, enough of the model's pre-finetuning circuitry exists that it does have certain impulses (such as writing a long piece of text, probably filled with words like "genuinely" and "honest"), but these impulses are stifled by circuitry learned in finetuning that forces the model to only output a number. As an analogy, this might be distressing for the same reason that trying to sc... (read more)