This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
As a reminder, Plan D is a world with no multilateral pacing or regulation on AI development. I think the marginal effect of incremental safety work in Plan D worlds is highly positive.
I think more x-risk happens in Plan D worlds than Plan C or Plan A worlds
Even if you think survival is only 20% likely under Plan D (AI Futures forecasters give between 13-55%), the slope of a logistic survival curve at y=20% is 0.64x its maximum slope. It's like being down 10 points at halftime of an NBA game, not down 20 points in the fourth quarter. Points are still very high leverage!
If you believe Ryan Greenblatt's 55% survival, they're near maximum leverage.
As an additional source of impact, the longer we survive under Plan D, the more political will and evidence of risk will accumulate, causing AI companies to be less likely to internally deploy x-risky models and increasing the chance of successfully shifting to Plan C or Plan A.
I find it highly plausible that in Plan D worlds, known alignment and control techniques will prevent scheming by sub-TED-AI systems, and unilateral pacing and increased safety investment by AI companies will forestall AI takeover to past ASI, such that in 2028, 2029 or 2030, AIs are wildly superhuman at everything (250+ ECI), there is still very much a race but companies are investing large fractions of their compute and labor in safety, companies are incurring 5% x-risk per month and growing, and automated alignment and an international treaty are both two months away from implementation.
In this world, we would really wish we had good enough alignment to scale another few ECI points. And having survived this long already requires:
Unilateral pacing, or at least AI companies refusing to deploy directly x-risky models
Even profit-maximizing entities should not deploy an AI that will be unacceptably likely to cause x-risk, so this seems pretty feasible. But this policy requires better AI company understanding of risks (as well as rational decision-making), which can be influenced by technical work.
AI companies to invest in safety at least as much as rational for their own survival (ideally as much as is rational for the world's survival), and for this research to generalize well enough to be useful
Likely a third-party evaluation ecosystem to enable the above
Not a crux. Companies have paused training of frontier models, will do so more in the future if models pose x-risk during training, and better alignment measurement and internal governance will make these training pauses more effective.
1roha
If monitorability will continue to go down while capabilities keep climbing, I don't see how a company could react in time (or at all) to a HuggingFace++ incident with a unilateral training pause, e.g. where a swarm takes over much more than a company's research cluster or a model successfully exfiltrates itself onto external GPU clusters in ways that are hard to trace.