We should prepare a playbook for the day after a warning shot

Yair Halberstadt

Imagine in 6 months or 6 years, a frontier AI model goes horribly wrong. Perhaps it releases a synthetic virus which kills hundreds. Perhaps it shuts down the internet. Perhaps it gains control over China's nuclear armament.

Fortunately humanity survives without too much lasting damage. But in the immediate aftermath there's a clear call from the people. Something must be done. The question is, what?

Without a good answer there is a strong risk that the opportunity is squandered, or worse, that policies which sound good but are actively harmful are chosen - for example strongly limiting deployment while allowing training to continue full speed ahead.

If this scenario does occur we should be ready to answer the call. This involves:

considering how the overton window is likely to change post-disaster, and what are the most effective policies that could be easily and quickly pushed through as a result.
considering what can be done at all levels of government, both state and federal, legislative and executive.
creating concrete draft legislation and executive orders.
preparing websites explaining clearly both to the public and relevant experts our policy ideas.
keeping a low enough profile to avoid wasting too many weirdness points when the proposals are still out the overton window, but making sure relevant actors (both in government and media) are aware enough that they'll know who to call the day after.

I agree, and I am looking to build an organization that is partly focused on this. I'm fundraising right now.

I wrote a post related to this years ago.

Another important and related thing is: what seems to be convincing more capabilities researchers to become concerned? It's internal warning shots, as we've seen lately. Preparing materials tailored to such individuals and other key decision-makers would help them navigate the next steps. Even preparing AI safety content with their cruxes in mind and properly articulating loss of control (perhaps with a strong focus on science) would move the needle and allow for more cooperation.

If you are a grantmaker or would like to give feedback on the org (or recommendations for co-founders), please DM me, and I can share my proposal.

Largely agree, a couple of extra thoughts:

Warning shots are useless unless legislators and people know how to respond to them. Just telling people "something is scary" and no call to action desensitises them to the risks of the scary thing. Whether that's calling for a pause, or having a list of minimum safety standards for labs, the message needs to be coherent and something people can repeat.
- As you state, there is a real lack of "off the shelf" policy proposals that are available for people to push for. Has anyone tried to collate a set of empirical alignment/control techniques that are reasonably effective and labs should implement? (e.g. OAI only recently implemented live CoT monitoring)
  - To the extent these are open questions, then the push should be for safety research transparency, or similar governance solutions
  - Whatever the standards are, there should be some sort of consensus. Perhaps that's another open letter from safety scientists saying this is what we should do, and even if it gets ignored, it pushes the needle some amount of the way (e.g. "9/10 safety scientists agree we should do X")
Just because a warning shot occurs does not mean the media will a) listen to it, and b) internalise it in the way you want. For instance, very little mainstream reporting has been done about the METR/Redwood report, and the only example I could find in my media sphere spends a large portion of the article writing about criticisms to Dwarkesh's summary, rather than the implications of the event. I think it would be helpful to do more outreach, e.g. writing to politicians, or orgs like METR, Redwood, MIRI, FLI, CAIS etc to reach out to international news organisations and offer to interview (though I obviously have no idea whether orgs were doing this and were just turned down). In general, the lack of advocacy systems and procedures hurts our ability to capitalise on warning shots.
Warning shots are one of the few ways that we can demonstrably show the harms of AI systems and how they affect the average person, and in particular, this warning shot was very helpful of breaking the "Terminator" stigma. On your last point of avoiding "weirdness", I think that shows too much hesitancy, especially at this stage. This is no longer sci-fi, we have a real life example which affected people, and if we self-censor to make sure we remain "in the Overton window" we risk underselling, which is the far larger issue.

policies which sound good but are actively harmful are chosen—for example strongly limiting deployment while allowing training to continue full speed ahead.

Why do you think they're harmful? Limiting deployment would reduce investment in AI. Same for e.g. making labs liable for harm, or enforcing copyright and privacy restrictions on training data, or resisting datacenters, or adding taxes. All of these would technically "allow training to continue full speed", but would reduce ROI and thus reduce investment, possibly slowing down the apocalypse a lot. And it's better to push these policies already now, so that if/when the warning shot comes, the public knows who to turn to.

On the one hand, it reduces investment. On the other hand it removes any insight into what's happening, and means that we get less ability to test less advanced LLMs under fire in the real world before super intelligent AI turns up.

I agree it's not obviously bad without more details about exactly is/isn't allowed.

Does the possibility of super intelligent AI turning up directly relate to the amount of investment that goes into research? If so, wouldn't it be possible that less investment means lower chances of the harm happening or are they less related than that?