First, I don't expect it to be "pure RLHF" or "pure RLAIF" - there is probably a Python script that is generating, mocking and scoring the rollout, possibly using an LLM to score the rollout. And these Python scripts can have varying degrees of quality.
Second, even with RLHF, much of the feedback quality a human can give comes to the process and tooling, as opposed to whether the human is a midwit or a genius.
The problem is not just RLHF. They are using experts for RLHF, but currently most data by token count comes from programmatic verifiers. Which in an ideal world are written by a conscientious human using the best tools available, but in a world where you need a lot of RL data fast, quality might suffer.
I am not saying they directly use the previous generation of the model to vibe-code a bunch of training environments, but given the vibes from the AI labs I can't be sure they don't at least to some extent. And given that, for a given cost, you can create much more low-quality than high-quality data, I won't be surprised if economic incentives lead to a fairly low mean data quality.
Feels to me like a consequence of using RL for more and more complex tasks, where reward-hacking-resistant verifiers are harder to write, along with organizational pressure to write more RL environments faster decreasing the quality of the average RL environment.
Ah well, thinking about this makes me think that maybe one of the reason for "context rot" behavior where sloppy behavior increases with increasing context is not only an intelligence limitation but also that it associates more complex contexts with more bluffable verifiers. And also why prompt engineering is important - speaking like you know what you are talking about is associated with having a less bluffable verifier.
I don't see a reason for the valence of a concept to be a linear function of the valences of the subconcepts. Maybe it's a learned function, where 'thought-valence(I will fight against X)' ends up having a value that is opposite to the concept-valence of X. Or maybe there's no explicit computation of concept-valence, and there is just an inductive bias that makes the brain learn that 'thought-valence(I will fight against X)' should increase as 'thought-valence(I like X)' decreases.
My pet theory theory of this is that you get 2 big benefits from RLVR:
1. A model learns how to write sentences in a way that does not confuse itself (for example, markdown files written by an AI tend to context-poison an AI far less than the same amount of text written by a human or by error messages).
2. A model learns how to do "business processes" - for example, that in order to write code, it needs to first read documentation, then write the code, and then run tests.
These are things that RL if done right is going to improve, and they definitely fe...
If your evaluation scheme is the evaluator looking at the transcript for 1 minute, it is going to be very easy for them to miss any non-obvious problem - whether the evaluator is a smart human or an AI. Especially since if it's an AI, it's likely to be an earlier, worse version of the evaluee.
That's why you want mechanisms that tilt the playing field so that the evaluator has advantages over the evaluee. For example, automations so that simple reward hacks are blocked, or various kinds of monitoring so that a smart human (or a smart AI?) can look at the AI... (read more)