This is a special post for quick takes by Cam. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Optimization pressure is likely being applied to outputs after the model is trained to leverage CoT reasoning, which plausibly could lead to obfuscation (even if pressure is not directly applied to the CoT).
I agree! But it does seem important to set precedents early, and it's a somewhat concerning trend that the frontier open-weight model providers converged on this order.
[...]
This also seems right. My understanding is a (simplified) optimal order is:
pre-training -> simple instruction training -> preference training -> RLVF
Letting the model latch on to human values during training, then only later encouraging ruthless task completion.
Is the hope that the model will "training game for good" during RLVF? That seems somewhat risky, especially to the extent that your RLVF rewards are imperfect and somewhat encourage cheating, as it could erode away the goodness.
I think you maybe want to do PT --> preference --> RLVF --> preference, but that doesn't feel great either.
Yeah this is interesting. However, I think it's unclear how HHH training (before any reasoning training) affects the faithfulness of CoT. I think it's reasonably likely if you apply your HHH training before reasoning, and you apply no optimization pressure on the CoT as you train reasoning, that you end up with scratchpads that are fairly uncensored (since you let your HHH model thoroughly explore the use of its scratchpad without penalization). On the other hand, directly intervening with the external reasoning process to make it more human-friendly seems very likely to obfuscate harmful reasoning traces.
It also looks like providers are already providing some mix of both with cold-start SFT (as noted in comment below) so it would be nice to know if one is more harmful than the other wrt monitorability.