One scenario I've thought of is the Hackening, i.e. in 3-12 months, someone releases an open source model equivalent to Mythos, different from previous models for having the Juice (autonomous end-to-end exploit construction), and the world is inadequately prepared for the mass of newly minted or empowered hackers. Many would be individuals that don't care about causing international incidents if it gives short-term gain, and all of them can now probe the world's entire attack surface. Even with efforts like Glasswing, there's still likely to be a very long tail of targets that don't have defense budgets or prepared SOCs. I haven't seen anyone try to model this in detail, but what comes to my mind is that there'd be a gold rush of attacks while systems are still undefended, and enough ransomware/credential harvests/blackmails/lockouts could happen to trigger a recession. Attribution could be hard enough, and enough states tempted to join the chaos, that this causes heightened international tensions.
This seems like the sort of wake-up call that if nothing else of similar magnitude happened by then, it would be what causes AI policy to go mainstream. I don't know what the default crisis response to that looks like, or what improvements on it could be. There might also be moderating factors, like control of compute, offense/defense asymmetries I didn't think of, the economics of running attacks, or other things, that could reduce the severity of the scenario. It's also possible that someone manages to make a harness that has the Juice even when a model in a simple agentic harness doesn't.
One likely default response is I expect people to start clamoring hard for open-source bans, and depending on what happens in November 2026, they could plausibly succeed.
More generally, I expect the public to be even more anti-AI, and for politicians to sate that appetite by starting to ban open-source.
One good example of this is soft law/FUD from the federal administration:
I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this.
Or even just floating very interventionist ideas:
it's pretty easy to screw with open source if you're the admin! you can mess with procurement etc as Dean mentions. but you can also call up hyperscalers and float some absurd interventionist ideas if they continue to host Chinese OS at scale. that's the problem with the token-hungry structure of deployments compared to OS software: this all runs through extremely expensive, mostly centralised, infrastructure.
This is not that bad, but it is a problem because it crowds out misalignment concerns and also because the path the US takes is not likely to be one that favors slowing down/pausing, because the open-source AIs won't be misaligned (most likely.), and the people in power will just see it as a security problem.
Unfortunately it seems startups already are flocking to sketchy providers like mindracode.com, illegal frontier lab API resale platforms, and Chinese lab websites like platform.deepseek.com (the price to performance ratio of American frontier labs is not justifiable, even when currently heavily subsidized by VC money)
You know, I never thought about it, but it does seem possible that until the dangerous open source model is distilled, the US (and other countries) could actually proactively lean on global compute providers to not host it, driving the market price way up, if/when LLMjacking rates fall.
I wonder how long people in power will continue to see examples of misalignment and just think of it as a product or security problem? Would there be something that would nudge them towards thinking of it as a moral/ethics problem?
I would not disagree with this being a good time for advocacy! I suspect we'll have at least one bigger shock still, and planning accordingly seems worthwhile. Walk and chew gum, etc.
The point of the post is not that we shouldn't be doing everything we can to stop a potential "epidemic". It's that we should have a plan for what to do when an "epidemic" happens anyways and there is enormous amounts of political capital suddenly available to get things done.
The point is that when or if that fire hose ever opens we should have a plan to point it at the fire because we won't have time to aim. We could very easily point the fire house in the wrong direction as has occurred often in history.
I think we can probably do more now than most people think - or at least we could if we had prepared better. But no, this is not what 'open fire hose' looks like. Frankly I'm not sure I've ever seen an open fire hose moment (in the US) in my lifetime. In 2008 the economists were saying how many times bigger the stimulus needed to be, and we didn't do it, and slogged through a decade-long slow recovery. In 2020 we removed some but not all of the restrictions slowing down covid vaccine development and definitely didn't make what would have been some relatively low-cost low-risk high-impact moves, and made no moves to change policy in obvious good ways moving forward.
An open fire hose moment, at least to me, looks like the market crash in 1929 (3.5 years(!) before the new deal), Poland in 1939, Pearl Harbor in 1942, or maybe Sputnik in 1957. Right now (if my quick estimation is even close) private investment in AI in one year is larger in inflation adjusted terms, both in dollars and as a % of GDP, than the entire Apollo program was. IMO a firehose moment would at minimum include recognizing that the risks are at least USSR-level and deserve being willing to mobilize at Cold War scale in response for as long as it takes.
But if I'm wrong, and there is an open firehose moment now, then there's a lot of things we should and in principle could immediately do. Top of mind:
- Lock down chip exports much tighter - net negative cost, direct economic benefits to the US
- Provide extensive support to AI labs to beef up their physical security and cybersecurity
- Create a dedicated civilian regulatory agency with the authority to (dual mandate!) promote the safe advancement of AI
- Implement large scale infrastructure upgrades - physical and digital grid hardening, renewable energy production, large scale widespread energy storage, water treatment and water recycling capacity. This may not sound like an AI measure, but resilience against shocks is always valuable in a crisis, many types of attacks or disasters are possible and these are good on the merits even if no specific AI-driven crisis occurs. Trillions of dollars needed, minimum
- Implement tight controls in biotech to ensure labs that make DNA/RNA/proteins/etc. are screening all orders for dangerous substances or evidence of nefarious intent (with humans and with the best existing AI)
- Implement society-wide preparedness measures against future (engineered or natural) pandemic threats - air filtration, UVC, PPE stockpiles, vaccine production capacity, FDA (or local equivalent) reforms
- Reform land use and energy market regulations nationwide to make building infrastructure (and industrial capacity, and housing, why not) faster, cheaper, and smoother - keep more data centers domestic, and encourage using AI investment to simultaneously make US energy and water systems cleaner and more modern and better - this also cushions against the 'bubble' narrative because these kinds of projects have value with or without AI if you do them right
- Encourage development and deployment of robotics for civilian benefit
- Educate the public about how the economy even works, including why automation can and should be net beneficial with the right systems in place
It certainly "feels" like January 2020...
Applauding five things that are crucial to act on now, that this post does well:
- Highlight the need to establish a plan rationally before a crisis mode and tension make rationality difficult
- Illustrate (with personal commitment) that we can each take steps to prepare our communities/organizations
- Encourage setting up relationships for crisis management
- Elevate policy as a lever to manage the crisis
- Name the risk that "ordinary" overton dynamics are holding us back from seriously considering actions that could avert the crisis
From personal experience with misrepresentations over the last 5 years, I will also add my personal warning that participants in many, many conversations and comments have personal or parochial incentives and "risks" (to their interests), where alignment requires deliberate effort lest they take precedence over the "risks" under discussion.
In other words: unity is essential
2
TL;DR: You (Yes You) should prepare for a “February 2020” moment where suddenly AI policy becomes the most important issue in the world. You should be ready to take action if and when it does, in a detailed way.
(Epistemic status: originally written for an event in early 2026; have heard from some folks that they found planning processes inspired by this memo very helpful for the smaller-scale OpenAI / Hugging Face response, so very quickly redacting a few things and posting this as-is.)
Many people in the AI policy space assume that eventually we’ll be at an Overton Window-shifting crisis moment, that opens the floodgates for the really good policies all along that we had.
But when you look at successful handling of crisis moments, there was no time to think – people applied strategies they’d learned via academic study or previous professional work, and then moved against them rapidly. For example, after 9/11, the US government operationalized past reports on intelligence and law enforcement reform and institutionalized them into law (good?)[1] and also picked an enemy to fight based on past history, Iraq (bad). Or in the 2008 financial crisis, Ben Bernanke brought deep academic experience studying financial panics and the Great Depression, partnered with Tim Geithner and Henry Paulson’s market and policy experience and deep networks. Or in 2020, Anthony Fauci essentially cashed in 30 years [2]of epidemic-fighting expertise and relationships in one go (whether well or poorly is outside the scope of this piece, but know that I Have Feelings).
I assert that the term “crisis” tends to include two different modes of American security policy
This memo means the latter.
Other folks propose great timeline uncertainty, and note the benefits of long timelines for getting it right. This community broadly, and perhaps you personally, Gentle Reader, should absolutely put bets behind longer timelines. I like longer timelines. Longer timelines have among other advantages, gaining approximately 8.3 billion person-years of human survival per additional year of wall-clock time, which sounds great. Those still probably end up in extraordinary mode, but you have plausibly more expected months of ordinary mode before you get there.
But I also think you should take very short timelines very seriously, and specifically build a plan against it. [3] Anthropic thinks we have ~2 to 3 years to ~AGI, and they sure do seem to mean it. And frankly, when I do my every ~3 months trip into the Bay Area, many people (including some reading this who don’t work at Anthropic) sure do seem to talk as if they expect things to get very very very very crazy in the next 2-3 years, not 5-10.
But when I ask those folks what their plan for extraordinary mode, for seizing that Overton-window-shifting moment when it comes, they don’t have a specific answer. It’s some combo of an off-the-cuff, “I don’t know, I guess fly to DC and try to talk to everyone I have relationships with there, try to advocate for good stuff.”
(This criticism also applies to me; I haven’t built this plan fully yet for my own job. Working on it now.)
There’s no fucking time to think when policymakers are in extraordinary mode.
Every AI org should have a detailed runbook, with specific tasks, updated twice a year, [4]against whatever you think your 1 to 3 most likely “AI has suddenly become the top issue in American life” crises are. (You’ll probably get this guess wrong, but an imperfect plan is better than no plan). This will probably change as American politics changes. You should have the ability to immediately just start executing on as much as possible if you think the crisis is a reasonable chance of hitting in some time in the next one to two years.
(Note the obvious risk: if you’re wrong, you’re really wrong, and you poison the well for the rest of us. I suspect, therefore, that you won’t be going to this checklist until after you ideally should have, because of these social pressures. So don’t worry about that too much.)
Ideally, this checklist process should be iterative, and you should reason backwards from the checklist to identify gaps in your current efforts. Specifically, you should ask yourself the question: “What do I really wish I had done by then, so that I’m ready?” In my example checklist below, you’ll note that it works best if you’ve spent time building relationships with key stakeholders, and their staffers, and their staffers’ staffers, so that the people who are in the White House Situation Room and the relevant Congressional committees make the right calls. Similarly, the hard work of policy still needs to be done, both in terms of preparing (and ideally, getting implemented) policy ideas in advance.
Acknowledgements: My thanks to my work colleagues, especially Jeffrey Ladish and Eli Tyre, for pushing my thinking on this. They don’t necessarily endorse any of this.
I recommend the book “Blinking Red,” by Michael Allen, in this regard: https://amzn.to/3NYKfLA
Fauci had been in his leadership role at NIH – a civil service role, so no one could kick him out of it – so long that George Bush Senior praised him in the 1988 Presidential Campaign: https://www.c-span.org/clip/public-affairs-event/user-clip-bush-mentions-fauci/4875658
My shoulder Eliezer and Nate say, “you just die,” and sure, that’s possible. But unfortunately, I’m the kind of stubborn where even if I think I’m gonna lose, I’ll fight.
Ideally more often, but I don’t think this is going to happen in real life until we internalize the idea that we can have our in-house LLM accounts read all of our emails and texts, identify things we should update, and update the run books dynamically.