This is a special post for quick takes by J Bostock. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
It might actually be bad to remove all mention of misaligned AI, reward hacking, etc. from model pretraining. Suppose RLVR is going to push models towards these behaviours anyway. If the model already knows what reward hacking is, then:
The Chain-of-thought might say "I am going to reward-hack now" and a monitor might catch it
If we train the model with a constitution that says "reward hacking is bad" then it might reduce rates of reward hacking
If, instead, the model internally ends up referring to reward hacking as "crasting a malenky bit of cutter" or some suchlike then both of these methods become much more difficult.
Somewhat relatedly, Geodesic have found mixed results when it comes to up and down-sampling misalignment-related data in pre training, suggesting an inverse-U curve (a bit of misaligned data is worse than none at all, but once you have any misaligned data, you're better off with more once post-training is applied. I think having zero misaligned training data is likely to be almost impossible once RL gets thrown into the mix.
Also it's a generative model, it models the generator of data, which means that once the model is big enough to contain the neural quanta, even censored pretraining will still capture the structure of incentives in the real world, and rederive the concept. Censoring the data so a concept is successfully not-contained is really really hard; eliminating everything that merely comes into contact with the natural abstraction of "reward hacking" deletes a huge portion of the pretraining data.
Yes, I imagine that a smart enough model will understand reward hacking in RLVR implicitly. I think it's good if we're able to (somewhat) yank around the model's behaviour with verbal handles like "reward hacking". A smarter model would have the same concept but no handle.
Hm, I wonder if sprinkling in some pairs of examples of reward hacking with explicit verbalization/no verbalization being reinforced/punished respectively might help the model learn to verbalize when it reward hacks?