The obvious problem with RLHF (et sim.) is that humans make mistakes. If manipulating the user can get them to give a thumbs-up, RLHF will teach manipulating the user. Instead, we want the user's thumbs-up to be treated as an observation, not ground truth.
Before being used to train a model on what's good behavior, user feedback should be interpreted in a way that allows for human mistakes. So a model can do computation we'd narrativize as "Well, the human gave a thumbs-up, but it's because I lied to them, so I won't treat this as positive reinforcement."
Of course, this sounds good, but... How can you initialize this sort of process so that it accumulates good changes rather than accumulating random noise? Can you leverage a single model's understanding of human psychology to improve its own training? How should it balance deferring to humans (particularly at the beginning when it's both ignorant about the real world and is probably making "moral mistakes") with the whole ignoring-human-when-it-thinks-they're-mistaken thing?
Sure, you could reduce the rate human mistakes by putting in more effort/fact-checking/care. But I'm mostly talking about the systematic problem where all humans have flaws. The human feedback signal is not a clean reflection of a utility function we'd endorse, it's fingerprinted by human cognitive limitations, systematic biases, and even our manipulability via the physical world.
Getting good behavior out of flawed feedback isn't (just) a money/effort/care question, it's a know-how question. We don't know how!
1sudo-nym
RLHF data labelers are generally located in developing countries, are paid pittances, and frequently develop mental illnesses. There are early attempts at unions but nothing has stuck yet.