Here are a few claims I believe: 

1. AI companies will probably face a choice in the next 0-2 years to either (i) incur a significant risk of AI takeover or (ii) slow down capabilities progress. 
2. If an AI company slows AI capabilities without slowing down algorithmic progress^[1], then they will accumulate unused algorithmic progress (which I’ll call an algorithmic overhang).
3. Algorithmic overhangs are a major downside, sufficient to make me uncertain about whether slowdown proposals that incur algorithmic overhangs are net-positive. 
4. The good versions of safety cases or limitations on model capabilities work via limiting algorithmic progress. 
5. The two most promising ways to limit algorithmic progress are compute reallocation and internal deployment lags (Options 2&3 here). 
   1. Compute reallocation: limiting the amount of compute that can be spent on applications that increase capabilities (e.g. experiments, training runs, etc). 
   2. Internal deployment lags: limiting the capability of models used on AI R&D (for example, requiring a six-month delay before you can use a newly developed model on AI R&D). 

AI companies and governments should be implementing policies that limit the generation of new algorithmic progress (such as those in Claim 5), as opposed to the capabilities of the final training run. I’m very concerned that by default most people designing AI policies do not understand these considerations, and so will end up with policies that are net-negative or much worse than they could have been. 

Below I’ll go into a little more detail on claims 2, 3, and 4. 

Claim 2. If an AI company slows AI capabilities without slowing down algorithmic progress, then they will accumulate unused algorithmic progress. 

There are several proposals for slowing down that don’t significantly delay algorithmic progress: 

• Limits on benchmarks or benchmark aggregates (e.g. ECI or ECI / year)  
• Limits on pretraining validation loss 
• Pacing due to “safety cases”: if an AI company judges that a training run is unsafe, they will pause/delay the training run until they believe they can make it safe.^[2]

These all operate primarily via limiting the capability of a final training run. So let’s assume that AI companies slow capabilities via any of the above methods. Then what happens? 

One possibility is that this slowdown has very little effect on the long-term trajectory of capabilities progress. The company will continue to accumulate algorithmic progress via R&D. Then, once they’ve decided they can proceed safely, they do a big training run, getting them back on the trend that they would have otherwise been on. ^[3]

In this graph, the algorithmic overhang is the difference between the red line and the black line. The red line represents the capability of the most capable model you can train given all of your algorithmic progress, assuming you ended a training run at exactly that point in time. The black line represents the capability of the most capable model that you’ve actually finished training. In the status quo, with no pacing, you bank a small amount of algorithmic progress during each training run, and then use it all on the next training run. 

If we slow down via anything but limiting algorithmic progress, this creates an algorithmic overhang, which, if you go back to training, results in a rapid return back to the original trend. This algorithmic overhang could, in principle, become very large: algorithmic progress is fast and is likely to accelerate in the future: estimates of algorithmic progress today vary between 3x-30x/year; and it seems plausible that algorithmic progress recently has been even faster. Because algorithmic progress is more rapid than compute progress (which is ~5x/yr), an algorithmic overhang could be much worse than a compute overhang. 

Claim 3. Algorithmic overhangs are a major downside, sufficient to make me uncertain about whether slowdown proposals that incur algorithmic overhangs are net-positive. 

The algorithmic overhang creates two major problems: 

1. Discontinuous rapid progress once the limitation is removed. If the pacing is voluntary, AI companies at any point could just decide to stop doing it and throw all of the accumulated algorithmic progress into a training run all at once. I think the “no pacing” line on the graph above seems weakly better to me than the “pacing that breaks down” line in most situations, because the intermediate speedups allow for more alignment uplift. 
2. Poor AI company security means that the unused algorithms still accelerate trailing actors. AI companies are not robust to algorithmic leakage. It is likely that any algorithmic progress made, even if it isn't used in the final training run at the compliant AI company, could get used by other AI projects.

One counterargument to this basic case is that if AI companies are limited from making a more capable final training run, it's possible that they would not make any more algorithmic progress beyond what they're allowed to use. Unfortunately, I don't really buy this: 

• Not all algorithmic progress is created equal. Whatever proxy is used for measuring the capabilities of the final model will be possible to game. For example, with ECI, AI companies will be incentivized to do more algorithmic progress than they would need to hit this cap, so that they can pick and choose the particular mix of algorithms that has the most real-world usefulness per ECI. Similar for safety cases, pretraining loss, or other methods. 
• I expect substantial inertia. Current capabilities researchers will just want to continue their capabilities research, and unless we explicitly shift resources away from it, it's very likely that at least in the immediate term, many capability projects will continue.

The next question is whether these downsides are sufficient in order to render non-algorithmic progress pacing net negative, as opposed to just less good than you might have thought. I think this depends on:

• The chance of the pacing regime collapsing. If it’s very durable, you don’t need to worry about algorithmic overhangs. However, I think confidence in the durability of a regime like this is very difficult, especially if algorithmic progress is leaking. (And algorithmic progress seems very hard to secure). 
• What the returns are to marginal slowdown at this point in time, i.e., how much alignment progress you're making. All else equal it’s more valuable to slow later rather than earlier (provided you haven’t hit the point of no return). 
• How much more risky the takeoff period becomes due to the rate of capabilities progress increasing discontinuously. If ongoing development is, say >50% to lead to AI takeover after the next training run then it might be worth it to directly limit final training run capabilities despite incurring the overhang: we just need to gamble that we can continue pacing for long enough. 
• The extent to which capabilities limits end up acting like a deployment lag, which becomes a more and more important effect later on in the intelligence explosion. After full automation of AI R&D, limiting final training run performance might in practice be a reasonable lever for limiting algorithmic progress because so much algorithmic progress would be unlocked by marginal capabilities progress. However, in this case the algorithmic overhang incurred would also likely be gigantic. 
• The political reaction to very fast, discontinuous capability progress. It’s possible that after we’ve incurred a huge algorithmic overhang, there will be enough political consensus that it would be a terrible idea to throw them all into a giant training run, thus enabling a more durable pause. (A counterconsideration is that maybe the political consensus will be weaker in the more discontinuous regime because there are weaker AIs for longer; so there’s less time for diffusion). 

I’d like to do some quantitative analysis of these considerations. 

Claim 4. The good versions of safety cases or limitations on model capabilities do this via limiting algorithmic progress. 

I’m not in principle against safety cases; in the long term I think we really want a functional safety case based regime. The right way to implement safety cases involves gating algorithmic progress when in a position where you can’t scale capabilities. This is the only way to avoid an algorithmic overhang. This gets you the best of both worlds, but is more costly to the AI company (in part because it durably affects the capability trajectory). In practice, getting these details right seems extremely important and not on track. 

1. ^{^}

   For example by doing research to discover a more efficient architecture but not using it in the final training run.

2. ^{^}

   I’m also worried that we don’t know how to write viable safety cases right now, and so in practice any safety case regime will just lead to motivated reasoning from AI companies to declare their systems are safe when they aren’t. But I’m going to assume this away for the purposes of this analysis. 

3. ^{^}

    Of course in reality, the effect of this policy will be a slight slowdown in algorithmic progress due to the marginal decrease in AI R&D uplift. The point is that the effect that matters is via limiting algorithmic progress. If the goal is to prevent algorithmic progress, limiting the capabilities of a final training run to prevent future automation is probably worse by my lights than preventing this more directly (e.g. via an AI R&D deployment lag). Deeper into the intelligence explosion, when AI R&D uplift is more important than it is today, this effect will be bigger and naive capabilities cap become a more attractive policy.