I started drafting a similar point about warning shot prevention being net-negative, which I'll post here.

In a world where OpenAI had safer internal AI deployments, they might have ensured that their ExploitGym evals included better sandboxing, monitoring, etc. This could have prevented the HF incident from happening.

But the HF incident was a warning shot that increased OpenAI's willingness to pay for safety, and it made the safety problems at OpenAI much clearer to the outside world. And the harms from this incident were practically nonexistent. So preventing the HF incident seems clearly net-harmful.

It seems likely that OpenAI even redirected employees from capabilities work to safety work as a result of this incident! So the time OpenAI safety employees spent improving AI control measures may have been somewhat wasted, since this work was eventually going to be done by others anyway.

Consider a model of how much a lab invests in safety, with 3 numbers:

1. Ideal Safety Investment (ISI): how much safety investment the lab/society at large would want if they fully understood the risks.
2. Targeted Safety Investment (TSI): How much safety investment the lab currently wants.
3. Current Safety Investment (CSI): The lab's actual safety investment at a given time.

Whenever the lab hires a safety employee, they are attempting to get CSI closer to TSI. But if you work on AI safety, you probably think ISI is a lot higher than TSI. Maybe if you are really good at your safety job at the lab, you can punch above your weight and increase safety even more than would otherwise be expected, but it's unlikely that you'll approach ISI levels of safety just by "doing a really good job."

Every time there's a warning shot, TSI jumps up closer to ISI. Then, the lab will try to move CSI to TSI any way they can, which may route through current safety employees or any other employee they can get their hands on.

Presumably, you mostly care about CSI at the time we have really dangerous AI, not before. So maybe it makes sense to work directly on safety at a lab if you think we won't get any warning shots in time for the lab to raise TSI and then CSI, and you think you're much better at direct safety implementation than the marginal person the lab would have found to raise CSI. But you can just improve safety techniques and increase TSI from the outside, to help people working on safety inside the lab without substituting for them.