This is a special post for quick takes by No77e. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Doing RL on tasks like cybersecurity, problem-solving, etc. might make models develop scary tendencies/goals, and it could distort whatever alignment training came before. But aligning them after capability training risks finding oneself, for a while, with a dangerous power-seeking model or with one that starts to scheme and play the training game, nullifying whatever alignment training comes after.
So I wonder if a sound solution would be to interleave capability and alignment training and/or optimize them jointly. E.g., for each episode of capability training, you do an episode of alignment training. Or maybe draw training batches from a mixture of capability and alignment tasks and use a reward that includes both metrics. Whatever the details are, the point is joint or interleaved optimization of these different objectives.