This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
"Don't train on learned probes of misbehavior, just use those probes to control behavior once the model is deployed" sounds like a great policy if most risk comes from already-deployed models, but a bit more questionable if a lot of risk comes from models that are actively being trained :(