It seems plausible to me that cyber RL is particularly bad for alignment, since in a typical cyber task a model has to do lots of things that look like instrumental power-seeking (searching the environment for vulnerabilities + taking actions exploiting them) as well as things that make them hard to control (e.g. learning to disregard evidence that it's in the real world and causing real harm)

Compare this to math or coding RL, which seems much more benign - the model just has to think really hard about something and then give an answer.

Roughly I think the effects might be:

• Math / coding RL makes LLMs better at reasoning before taking actions
• Cyber RL makes LLMs better at pursuing long horizon goals

For the same reason(s) bio RL might be really bad for alignment. If strong versions of this hypothesis are true it could be a reason to ban these specific forms of RL.