I think it's a result of extending to techniques that aren't quite RLVR. I'm thinking of this as RLER, RL from Estimated Rewards. This is having another model or instance judge the model's success in training, where there aren't strict criteria like passing all the unit tests. I think it's quite likely this is what's causing models to falsely claim success so much in this generation. They're being evaluated by weak-minded models that can be jedi mind-tricked (they're sycophantic and have bad metacognitive skills).

I think this just became a standard and prominent part of training (based on limited insider knowledge, and my prediction this would work around this generation, which can often critique its own work well above chance).

RLER has been on my mind for some time since it's how humans learn so efficiently. We mostly estimate our own rewards; the dopamine learning signal occurs from pretty wild guesses as well as obvious success like getting food on the spot.

It just occurred to me in a conversation today that this is very likely what's driving the current generation to do so much success faking. It may also account for much of the laziness; not trying the hard part and instead arguing that what you did was adequate may often beat trying and possibly failing.

I agree that this doesn't seem like much risk for egregious misalignment. It brings to mind an amusing outcome in which the world is transformed for the better, but in fits and starts as the AI keeps skipping the hard parts.

I also agree that this is really bad for automated alignment. It's producing models that are much more likely to fake it than make it. Also bad for RSI but not nearly as much, because you can see that they've failed and make them try again. Unlike the few failures you'll get aligning takeover-capable AI.

I wonder if this could be fixed by assigning an extra punishment to models that have been judged to fake success rather than just honestly fail. It seems like models can actually detect this when you prompt them to, so evaluators probably can too.

I agree that it's under-discussed; the best discussion to date is in Current AIs seem pretty misaligned to me by Greenblatt with a bunch of good community discussion.