Yes, your code is exactly what we do. 

Should be fixed, thanks :)

I agree that answer-only (no CoT) training would be a super insightful ablation. 

Using a CoT monitor as the sole reward signal would also be interesting. I've grappled with the fact that these results seem to imply "verbalized intentions in the CoT matter", which implies it could be useful to train against a CoT monitor that detects suspicious intentions. But of course, this is the forbidden technique.

Thanks for the suggestion! Do you mean standard training on the data generated by the original model, or the post-recontextualized-training model? I assumed the latter, but want to make sure. 

Thanks for the connection to your work! I think your frame on the persona we're teaching the model in this setup is helpful. 

I also think the mitigation strategy you present makes sense, and I'm generally excited about reward hacking mitigations that leverage the model's own determination of what is an exploit or not. Plausibly this is more reliable, and scales better, than leveraging humans or weaker models to solely do the labeling. Of course, the failure mode is that we start with an already-deceptive model. 

Thanks for your comment! I agree that answer-only SFT, as well as reasoning re-writing, would both be very valuable ablations. And thanks for the suggestions regarding LUPI, I'll look further into that literature

Thanks for pointing this out! I agree we are exploring a safety-relevant variant of prompt self-distillation. 

It would certainly be interesting to see how much more effective logit-based distillation is than SFT at "internalizing" the omitted generation prompt. 

Thanks for the comment! I agree that this was a concern for us. However, the dataset is only 146 samples, and so I was able to look through a large number of samples manually to verify the answers were not hacks. I also specifically searched for indicators that the answer might be a hack, like the words "incorrect" or "test" in the assistant's CoT, and further scrutinized these samples. I was able to catch 2 or 3 special-cased solutions which the LLM judge did not identify as special-cased, and I removed them. I've made the datasets and judge code publicly available here: https://github.com/arianaazarbal/training-a-reward-hacker-despite-perfect-labels-data
 

Additionally, my confidence is increased in this result given that we first observed it in a multiple-choice variant of the same dataset. (The assistant had to select between two code options, one which was special-cased and one which was not, after thinking through the problem). In this case, no judge was needed to verify what was a hack vs. a non-hack.