Thank you for writing this post. Strong agree.
It's surprising that despite hundreds of EAs working now full-time in AI safety governance, only a handful seem focused on working on the things that actually matter.
- Measuring things relevant to upholding a pause: Assessments focused on compute allocation and RSI-relevant metrics (e.g. algorithmic efficiency, uplift measures, spend on AI agents, analysis of success rates on internal tasks) seem most relevant for creating the right kind of technical competency and relationships.
For those interested in this, the AI Futures Project just released a blog post on compute allocation assessments in particular, which they call Verified Compute Allocation Disclosures (VCAD). Basically, mandatory third-party auditor reports that classify frontier lab compute as used for AI R&D vs. safety R&D vs. external inference. The argument for this pacing intervention goes roughly as follows:
Though you would need to be careful to avoid creating a compute overhang.
I'm a little surprised that your list of short-timelines technical projects doesn't include "compute verification" work. (which ex. AIFP has considered necessary for Plan A) Maybe that's meant to be implicitly included in "assessments focused on compute allocation"? I think technical solutions like 'verify this chip is running only inference' or 'use black-box AI auditors to evaluate if compute usage is capabilities or safety while not revealing frontier lab IP' still are possible in short timelines, curious if you're thinking differently.
I think stuff like 'verify this chip is running only inference' seems most important for going from US to global pause. But going from no pause to US pause seems like it is mostly doable with just third party evaluators getting already precedented access into the AGI companies. But yes, that kind of hardware inference might end up being pretty important for international stuff.
Hmm, I can understand that logic. I'd say that making an international pause verifiable makes a US pause likelier, so it's still urgent as a lever toward any pause. (Well, everything's urgent, but you know what I mean.)
One of my biggest implicit disagreements with the AI safety community is that, I think the likelihood of getting this pace/pause seems reasonably high on current trajectories, something like 60-90% perhaps. (To be clear, this includes accounting for the actions that the AIS community itself takes.)
One individual working in lab advocacy estimated in private conversation that there is a 20-30% chance of pacing/pausing without government involvement happening in the next 6 months. This is not high enough. I estimate the chance of pacing/pausing via the California legislature in the next 10 months (and that is not struck down within weeks by a judge) is roughly 20%. Through federal channels, I estimate 10%. These probabilities are correlated with each other, since the chances of success increase for each channel if there is a major incident, and due to other shared factors. In other words, we are not on track to succeed.
My guesses would be:
though to be clear I haven't thought about any of these for very long.
While I agree they're correlated, I probably think they're less correlated than you think, since if one is on track to fail, then there's more pressure to succeed on the others?
Summary
Timelines to ASI are very short now
The METR report on AI R&D automation within Anthropic is substantial evidence that timelines to superintelligence are now less than 2 years, even accounting for uncertainty about takeoff modeling, the reliability of the report’s conclusions, and other types of uncertainty.
Ryan Greenblatt gives a straightforward way to extrapolate METR’s findings in a tweet. He claims that “In AI 2027, [the current level of speedup by RSI] happens around March 2027.” This means that real-world AI progress on AI R&D capabilities is faster than in AI 2027, an already extremely bullish scenario. Given that in AI 2027, ASI is achieved in December 2027, this suggests superintelligence would be developed, without slowdown, around July 2027 (assuming that the original AI 2027 scenario modeled the shape of AI takeoff correctly).
I think the most compelling evidence for very short timelines has been the reactions of established forecasters and researchers. In response to the METR report, the AI Futures Project released an emergency Q3.5 timelines update. Here are the all-things-considered forecasts of the team (I note that ASI is generally modeled as being reached less than a year after TED-AI absent deliberate slowdowns):
These forecasts give us a very tight deadline to get a pause in place.
Other reactions:
As a result of the above, my current all-things-considered median date for superintelligence without a slowdown or other major intervention is around March 2028. That is around a year and a half from now.
Three paths to pausing or pacing within 6-12 months
If we achieve ASI within 6-12 months, I estimate that the chance of an AI takeover is roughly 75%. (In a recent talk, Paul Christiano said it would be a coin flip.) Paired with the timelines above, unless we achieve aggressive pacing or a pause within 6-12 months, we incur something like a 1-in-4 to 1-in-3 chance of AI takeover within 12 months.
This is obviously unacceptable. I believe there are three feasible paths to achieving pacing or pausing within 6-12 months, and that these should be the overwhelming priority of the AI safety community right now:
Each of these should aim for pacing or pausing that gets us at the very least a 1 year slowdown in the expected date of superintelligence, such as three of the proposals in the AI Futures Project’s How to Pace the U.S. Frontier: a 10% AI R&D compute cap, a 9-month capability for AI R&D lag, or a 1-year inference-only pause.
There are ongoing efforts in each of these three categories. However, there are several issues with those efforts:
I’m not necessarily saying that, in some overall sense, lab advocacy, California lobbying, or federal lobbying are neglected on the margin. My claim is that lab advocacy, California lobbying, and federal lobbying with the goal of starting aggressive pacing/pausing within the next 6-12 months is greatly neglected.
Any sort of pause that can last more than a year will eventually need to become a global pause. Entering a US-only pause is an important stepping stone towards entering a global pause, but I expect pressure from middle powers to be pretty important for the pause going global. I think some of the best work here amounts to improving the situational awareness of middle power governments and helping them navigate the situation with regards to the US approaching ASI.
Technical work that seems most valuable
I think there are a few main categories of work that technical people should focus on:
Laying the technical foundations for a pause
All three paths to pausing would be substantially helped by technical work focused on setting the foundations for a global pause. I think the most urgent work of this kind is third-party assessments focused on the type of investigation that would be necessary to build trust between participants of a pause. If we get practice doing lower-stakes and lower-scale versions of assessments, we can build buy-in that we could pull off a global pause.
Ideally, sometime in the next few months, third parties would conduct assessments with the following properties:
I talk more about the technical work that would facilitate a pause in my recent post “We urgently need to practice before we can pause.”
What should you do?
As I write above, I think for most people the most impactful path is to do lab advocacy, lobby the California legislature, lobby the federal government, establish evidence of risk, lay the technical foundations for a pause, or work on aligning and controlling early automated researchers.
Here are some heuristics to help you decide which of these you should do:
I think there are also indirect strategies that support the three above paths as well and are plausible candidates for focus. These include:
I do not mean to denigrate anyone working in AI safety. I know that all of us are trying our best to preserve the future. I have tremendous respect for anyone that is earnestly doing so. But given recent evidence, many people should urgently reprioritize the work that they are doing.
FAQs and Objections
Big thanks to a friend who wishes to stay anonymous for substantially helping write this post. Thanks to Zach Stein-Perlman, Sydney Von Arx, Ben Pomeranz, Oliver Habryka, Eli Lifland, and Oscar Delaney for feedback and discussion.