It seems plausible to me that the effect of training models on desirable motivations (cf “Teaching Claude Why”) and then doing lots of RLVR on them (where the environments seem vulnerable to hacks) is to create ”agents that try to have good justifications for doing misaligned things”.

I’d guess that a primary effect of character training (and more broadly, TCW-style training) is to steer what kinds of reasoning traces are favoured. So it’s possible to get models that (i) reason about being aligned, and (ii) this helps them act in an aligned way (in the absence of RLVR).

RLVR might oppose this to a degree - some actions that result in high reward also counteract principles you train the model for. However, RLVR provides gradient pressure ~entirely on the outcome, not the intermediate reasoning. So this selects for models that are able to conveniently reason their way into doing all the things they might have a priori decided were false.

I think a version of this hypothesis could explain why we see so much motivated reasoning in the most recent generation of models. Alignment faking is also one way in which this manifests.

Note that this is a distinct hypothesis from “RL creates split personas” - here, both character training and RL end up affecting the same persona, in orthogonal ways