clickyquack's Shortform

clickyquack

This is a special post for quick takes by clickyquack. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Given the U.S. government's recent restrictions on Claude Mythos and GPT-5.6, I suspect one of the most tractable and effective strategies for further pushing the Overton window towards the urgency of implementing stronger AI regulation could be effectively communicating to policymakers that open-weights models, whose safeguards can be removed easily, are likely going to reach Mythos-level capabilities in under a year, which can meaningfully uplift cyberattacks and bioweapon development. Also, I suspect the main thing currently holding back the Overton window on interventions for AI x-risk in general to be a (perceived) lack of concrete evidence (see: The Milton Friedman Model of Policy Change, how we achieved the Montreal Protocol to mitigate climate change before the damage became irreversible despite strong industry pressure against it, etc.). Communicating loss of control risk would be ideal of course, but I'm currently unsure how this could be demonstrated safely.

The only ways to prevent Mythos-level open models would need to happen in China. Nobody but China is likely to produce a Mythos-level open model.

There are two main ways this might not happen:

The Chinese companies may decide to keep their most powerful models private as training costs go up. GitHub's recent decision to resell Chinese open models near the frontier may also annoy the companies which make those models, in much the same way that AWS repacking open source software annoys the companies that created it and tried to sell their own hosting services.
The Chinese government may independently start to worry about the capabilities of Mythos-class models and impose regulation of their own.

One of the complicating factors here is that Anthropic and OpenAI are preparing for the largest rent extraction in the history of the human race, and they represent a massive risk of concentration of power. (That power might be effectively seized by the government, but it's likely to remain concentrated.)

So there will be strong advocates for open models, including open frontier-adjacent models, in order to fight what looks like epic-level rent extraction and power concentration risks.

US regulation, by itself, will accomplish exactly nothing, because Europe would happily use cheap, open Chinese models to avoid losing control to US labs and to avoid paying enormous rent.

I find this whole logic deeply frustrating, because the logic pushes everyone involved towards racing. And while Mythos isn't an ASI-level threat, I don't know how many more breakthroughs and scale-ups we can get away with before we start facing loss-of-control risks.

The only ways to prevent Mythos-level open models would need to happen in China. Nobody but China is likely to produce a Mythos-level open model.

There are two main ways this might not happen:

The Chinese companies may decide to keep their most powerful models private as training costs go up. GitHub's recent decision to resell Chinese open models near the frontier may also annoy the companies which make those models, in much the same way that AWS repacking open source software annoys the companies that created it and tried to sell their own hosting services.
The Chinese government may independently start to worry about the capabilities of Mythos-class models and impose regulation of their own.

So there will be strong advocates for open models, including open frontier-adjacent models, in order to fight what looks like epic-level rent extraction and power concentration risks.

US regulation, by itself, will accomplish exactly nothing, because Europe would happily use cheap, open Chinese models to avoid losing control to US labs and to avoid paying enormous rent.

clickyquack's Shortform

1