A competing-risks model for doom and alignment as a function of compute allocation
Surely this analysis has been done before in the bowels of LessWrong. I gave up searching because it was too irritating trying to disambiguate the terms survival and hazard in the technical mathematical sense that I needed, against the more colloquial sense that they are used in AI safety discourse. Feel free to point me to prior work in the comments.
AI usage: much! There might be some residual AI text chunks in this prose but it should be mostly mine.
I recently had a discussion where the contested point was whether getting more compute was, by default, “good” or “bad” for AI safety research. What follows is my attempt to formalize goodness and badness in terms of hazard models, which seems to me to be a nice way of making our assumptions and disagreements explicit; in particular, they are scenario-independent. Slaughterbots? Matrix-style VR capture? Engineered plagues? Meatballs falling from the sky? Doesn’t matter. If we disagree about what ASI disasters or utopias look like, but agree that compute is a forcing function, then we can still talk about things like pauses, slowdowns, and accelerations without sweating the details.
The model setup is as follows. We assume two things can happen — an AI catastrophe (“doom”), or a breakthrough that guarantees alignment (“deliverance”). Both become more likely as more compute accumulates in the world, but they draw on different, potentially overlapping, pools of compute. The question becomes: given a trajectory of total compute and a policy for splitting it between capability and safety, what is the probability that doom arrives before deliverance?
The machinery I use is survival analysis of competing risks — the branch of statistics built for this kind of “which stochastic event comes first?” problem. The motivation for expressing everything in terms of compute comes from the economics of cognition: compute is the fundamental currency of intelligence, so we should be able to express both the problem and the solution in terms of how compute is allocated.
A hazard rate is the instantaneous probability per unit time that an event fires right now, given that it hasn't fired yet.
A survival function is the probability that nothing has happened yet by time . It starts at 1 and decays toward 0 as hazard accumulates.
Competing risks means two or more events are racing to fire first. We observe whichever wins; the loser is never realized.
If you are coming from LessWrong, some translations: the safety fraction is a compute-denominated cousin of the “alignment tax”, which usually means the capability cost of aligning one system rather than a fraction of the world’s compute; “P (doom)” is here derived from a competing-risks integral; and the response-function shapes below map onto familiar worldviews (“fast takeoff, hard alignment” convex / concave ; “slow takeoff, tractable alignment” the reverse).
Compute trajectories
I think we should distinguish between the rate at which compute is performed and the cumulative “stock” of computation that has been done.
Let denote the compute rate — the FLOP/s of AI-related computation happening in the world at time . This is capacity: hardware, data centres, and investment, exogenous in the base model. We treat it as non-negative and right-continuous. We make no distinction between training and inference — is unstructured compute, all of it. The more of it that is running, the more things are happening; we don’t model the internal structure of what kinds of workloads are running for this level of analysis.
The cumulative compute — total FLOPs performed by time — is
We assume there exists a division of the compute stream such that a fraction :
Capability rate: compute aimed at expanding what AI systems can do.
Safety rate: compute focused on ensuring AI systems behave safely.
The cumulative stocks in each pool are then
with at all times. If we want to calibrate this to the actually-existing world where some compute build-out has already occurred, we need to start the clock mid-history, with stocks and flows calibrated accordingly, and , and the integrals above are added to those. For simplicity I do not do that yet, but I return to it later.
The stock/rate distinction seems important because we, or at least I, imagine that capabilities persist. We can’t necessarily “un-train” a frontier model by switching off the data centre; the weights, the papers, the algorithmic insights are already in the world, at least while DeepSeek et al are out there reverse-engineering the western labs and adding their own tricks. Safety progress persists too, ideally — proved theorems, verified architectures, and alignment techniques don’t evaporate (unless these also languish in Anthropic’s file drawers). So the timing of doom should depend on the cumulative stock of capability compute , not the instantaneous rate . The rate determines how fast the stocks grow; the stocks determine the hazard.
Empirically we know (Sweeney and Sterman 2000) that people have really bad intuitions about such systems.
The function is the allocation lever, which a civilization sets by doing whatever governance is necessary (tax incentives, setting research agendas, etc.). The trajectory is the pacing lever: exogenous in the base model, but it is what compute governance acts on — licensing or reporting thresholds for large training runs, chip export controls, caps on data-centre build-out, nuking compute-heavy facilities — and a compute pause, read here as a stop to buildout, holds at its current value. So there are two levers, allocation and pacing, and the pause is a setting of the second rather than a third. There is a lot of texture in those choices, and some deep governance problems, which I will skate over here while hoping that my colleagues in the Credible Theory of Change department have this one in the bag.
Two competing events
I model doom and deliverance as the first arrivals of inhomogeneous point processes — random events whose chance of firing at any moment depends on how much compute we have accumulated so far:
Doom () An X-risk catastrophe happens. This is an irreversible absorbing state, because everyone is dead and rendered into computronium or paperclips or whatever.
Deliverance () We achieve guaranteed alignment — a state after which X-risk from AI is effectively zero, and some kind of end-game high score is unlocked.
Each event has a latent arrival time — and are the times at which doom and deliverance would fire if nothing else intervened. But the race ends at : Doom either happens first or not at all; deliverance either happens first or not at all; or neither fires within any relevant timeframe. When I write “” below, I mean the outcome “doom fires first” — i.e. — not the marginal probability that the doom process would eventually fire in isolation. These are the same in the case that you think that no deliverance is possible.
Each event has a hazard rate (instantaneous arrival intensity given that neither event has yet occurred — i.e. while we are still in limbo) that depends on the cumulative stock of compute in its respective pool:
where are monotonically non-decreasing functions, with and . Monotonicity captures our assumption that the more capability compute we’ve done overall, the higher the catastrophe hazard per unit time; likewise, the more safety compute we’ve done overall, the higher the alignment-breakthrough hazard per unit time.
Note that these hazard rates depend on time via the cumulative compute stocks. The compute rate does not appear directly; it influences the growth rate of and instead. Doubling doesn’t double the hazard at time , but it does make reach any given threshold sooner.
The assumption that doom risk is monotone in and safety progress is monotone in is agnostic about the shape of the response — linear, concave, convex, sigmoidal. That choice, the shape of the response functions and , is where I think non-trivial disagreements in AI safety discourse live.
Figure 2: Response functions g (doom hazard, red) and h (safety hazard, blue) as functions of cumulative compute under four shape assumptions. Use the buttons to switch scenarios. Go to the version on my website for the interactive plot.
Three outcomes of the race
The probability that the race remains in limbo at time — because neither event has occurred — is what survival analysis calls the joint survival function:
In words: the probability of still being in limbo decays exponentially as the hazard accumulates. Here and are cumulative hazard functions. Note the nested structure: each cumulative hazard is an integral over time of a function that itself contains an integral over time.
The race has three mutually exclusive outcomes:
where:
— the doom event fires first,
— the deliverance event fires first,
— neither event ever fires.
In our model the total stock grows without bound, and and are monotone and not both identically zero, so the total hazard is eventually bounded below, , and : the race always resolves eventually. But “eventually” can be a very long time. Over any finite horizon, we have residual probability mass on “neither yet” — the green curve in the plots below — and this residual is a practically relevant quantity. A world where stays large at human-relevant timescales is one where we just muddle through indefinitely, which is arguably closer to the standard baseline expectation than either doom or deliverance.
At each instant , while we’re still in limbo, the probability that some event fires within is , and the conditional probability that the firing event is doom rather than deliverance is
So the probability of doom can also be written
which decomposes into, “probability we’re still waiting at time ” × “probability something happens right now” × “probability that the something is doom rather than deliverance.” Keep your eye on that person. The middle two factors, , are the probability density of the resolution time , so the integral is an expectation:
Doom probability is the conditional ratio averaged over the moment the race resolves. For constant , both hazards are functions of the total stock alone, so we can equally write , an average over the stock at which the race resolves. Everything below is a corollary of this identity. The split sets the shape of as a function of stock. The growth rate of , the overall scale of the hazards, and any pause set the distribution of the resolution stock . The two only meet in the average.
Figure 3: Resolution of the race with superlinear doom risk (g(𝒞) = 0.005𝒞²) and linear safety progress (h(𝒞) = 0.015𝒞). Compute rate c(t) = e^{0.1t}, α = 0.5. The green curve (right axis) is S(t), the probability of limbo — neither doom nor deliverance has yet occurred. The shaded areas (left axis) show doom density λ_d(t)S(t) (red) and deliverance density λ_s(t)S(t) (blue). Their integrals give P(doom) and P(deliverance); the residual 1 − P(doom) − P(deliverance) is P(limbo) over this time window.
The constant-hazard-ratio case
Consider an ultra-simple sanity check. If the ratio remains constant over time, then holds regardless of the compute trajectory — this is a standard competing risks identity. stays constant when and are linear and is constant. If and , then and .
and the cancels. The probability of a doom outcome depends only on and the ratio , not on the compute rate or how fast it grows. This is the regime in which the speed of progress doesn’t matter — only the split between capability and safety compute determines our fate. If you’ve ever heard someone say “it doesn’t matter how fast AI progresses, only whether we invest enough in safety” — this is the (very specific) model in which that’s true.
It’s also not very plausible, in my opinion. Let’s get more realistically complicated.
Interesting response curves
With non-linear response functions, this invariance breaks down. depends on the full trajectory , because the time spent at each cumulative compute level determines how much hazard accumulates at that level. In the plot below, we compute by integration over a trajectory with exponential compute growth , for two different growth rates.
Figure 4: Probability of doom as a function of safety fraction α, computed by integration over an exponential compute trajectory. Solid lines: fast growth (r = 0.3); dashed: slow growth (r = 0.05); dash-dot: no growth (r = 0), a stop to buildout. For the linear case (grey), the curves coincide — growth rate doesn’t matter. For non-linear responses, faster growth shifts the doom probability because it changes how much time the system spends in different hazard regimes.
Notes on the scale: The race ends at , at total computation stock . Clearly the shapes of and matter only over the stocks below that, the functions never being evaluated outside that (random) range. Multiplying and by a common constant leaves their shapes the same but sets how much compute accumulates before the race resolves: larger hazards end it in expectation sooner, at a smaller stock. The convex and concave responses used here, and , cross at , so calling the doom hazard “convex” says something about the race only if with high probability. I chose the constants so that the median resolution stock lands at around {python} f"{C_med:.0f}" units, past the crossover. That is monopoly money; an empirically calibrated model would use in GW-years or FLOPs and scale every unit accordingly.
The same constant also describes the muddling through, in which neither doom nor deliverance is close: hazards are small at human timescales and stays near 1 for as long as anyone cares to plan. Shrink the coefficients by a hundred and the convex-doom race takes about {python} f"{t_med_slow / t_med:.0f}" times as long, but resolves at a stock about {python} f"{C_med_slow / C_med:.0f}" times larger, {python} f"{C_med_slow:.0f}" units rather than {python} f"{C_med:.0f}". Where the race resolves decides which part of the and curve the outcome is determined by. Near the origin the convex and concave curves are hard to tell apart; far out, the asymptotic shape dominates, and sits near 1 in the convex-doom world. While muddling through, our fate is one decided by the tails of the response functions, which are the part we can observe least. Relatedly, note that every trajectory looks like muddling from inside at first.
Some scenarios
Each choice of and is a world, and in each world the sweep above tells us which of the two levers, allocation and pacing , does anything. The four shape assumptions in the plot are toy versions of familiar worldviews.
The linear scenario. Both and are linear, the constant-hazard-ratio case. is the same at every stock, so the average over is trivial: speed is irrelevant and which resolution we arrive at depend only on . This is the world in which “compute is neutral, what matters is the safety budget” is exactly correct. This is the implicit model whenever we reason about a safety budget as a fraction without asking where on the curve we are.
Convex-doom. Convex , concave , “fast takeoff, hard alignment.” Every unit of capability compute adds more doom hazard than the last, and every unit of safety compute adds less deliverance hazard than the last. Early on, while both stocks are small, the safety curve can be the higher of the two. But the doom curve keeps steepening, so past some stock it overtakes the safety curve and pulls away, and from there , the chance that the next thing to happen is doom rather than deliverance, climbs towards 1. Run the race far enough and doom wins no matter how we split the compute. In this scenario the game is getting to deliverance early at all costs, so that pullaway never gets to matter. Faster growth means the world piles up more stock before the race is done, so the race is more likely to end past the overtaking point, in the region where doom dominates. That is why the fast curve sits above the slow one in the sweep. How much above depends on where the overtaking point sits relative to the stocks the race passes through: far beyond them, the convex shape is irrelevant; well inside them, it decides everything. So “fast takeoff, hard alignment” is a claim about where that point is, not about the shape of the curves out at infinity, and shapes at infinity are cheap to assert.
Concave-doom. Concave , convex , “slow takeoff, tractable alignment”, the mirror image. The doom hazard saturates and the deliverance hazard is superlinear once we invest enough, so falls with stock and faster growth helps. This generates “we just need to invest enough in alignment” scenarios where the alignment tax is finite and worth paying.
Compute overhang. Sigmoidal and with different inflection points, “the capability overhang.” Each hazard here more like a switch rather than a slope: near zero until its stock crosses a threshold, then on. Doom’s switch is at a certain capability stock and safety’s at a certain safety stock, and the two are not in the same place. Before either flips, not much happens. After the first flips, one hazard is running and the other is still off, so is about 1 if doom’s flipped first and about 0 if safety’s did, and most of the race resolves in that stretch, since it is the first time either hazard has run at strength. If doom’s flips first, the stretch is the dangerous window: capabilities on the steep part of , safety still on the flat part of . The split decides which flips first. Capability stock is and safety stock is , and in the plots doom’s threshold sits at {python} f"{X0_DOOM:.0f}" units of capability stock and safety’s at {python} f"{X0_SAFE:.0f}" units of safety stock, so safety’s switch flips first only when exceeds {python} f"{ov_alpha_star:.1f}". The growth rate decides how long the stretch lasts. Under slow growth it is long, the first hazard has the race almost to itself, and is a cliff in rather than a slope: {python} f"{ov_slow_half:.2f}" at , then down to {python} f"{ov_slow_09:.2f}" at . Either we fund safety past its threshold or we do not, and there is not much middle. Under fast growth the second switch flips before the race has resolved, both hazards run together, and the cliff softens into a slope; the fast and slow curves cross at an of about {python} f"{ov_cross:.2f}", near the cliff. Speed therefore cuts both ways. At low doom flips first, so faster growth shortens the dangerous window and helps; at high safety flips first, so faster growth shortens safety’s head start and hurts. The low- case is this model’s version of the compute-overhang argument against pausing: slowing down while doom’s switch is on and safety’s is off only prolongs the time spent at . That argument is right in this world and lethal in the convex-doom world, where no safety switch is coming and faster growth hurts at every . And from inside the dangerous window the two worlds look the same, doom hazard climbing and safety hazard flat; we find out which one we are in when safety’s switch flips, or doesn’t.
The pause. A “compute pause” here means a stop to buildout: from the compute rate holds at its current value instead of growing. That is pacing at , the dash-dot curves in the sweep. Under a buildout stop the stock of hazard keep growing, linearly rather than exponentially, so the hazards keep growing too and the race still resolves eventually, possibly at a smaller stock than it counterfactually would. As such, pause helps where rises with stock and hurts where it falls, so the implicit assumption behind calls for a compute pause is the convex-doom while the overhang argument against pausing is a low- overhang case. In the leaky scenario a pause is the only lever that affects the doom hazard at all, since cannot, though whether that moves is again a question of shape. A pause does more than this only if we spend it changing , or itself: spending our time catching up on the safety.
What if safety compute is also capabilities compute?
In the base model the doom hazard depends only on capability compute and the deliverance hazard only on safety compute . That quarantine is itself optimistic. Suppose safety research requires running large models, probing their behaviour, red-teaming, and training oversight systems, all of which also advance capabilities as a side effect; at the limit, all compute advances capabilities regardless of intent, because “capabilities are fungible”. The leaky model makes the doom hazard a function of total stock,
so that cannot lower the doom hazard at all, only raise the competing one. In the linear case, the ratio is , against the separable model’s . With , the separable model reaches even odds at and the leaky one only at ; even there, every step toward deliverance still drags doom closer.
Figure 5: P(doom) vs safety fraction α, comparing the separable model (dashed) with the leaky model (solid) where doom risk depends on total compute. The gap between them is the cost of the capability externality. Exponential compute rate c(t) = e^{0.1t}.
The gap between the dashed and solid curves is the cost of the leak, and it grows with . At the curves nearly coincide; at they differ by {python} f"{leak_gap['Linear']:.2f}" in the linear world and {python} f"{leak_gap['Convex-doom']:.2f}" in the convex-doom world, where the superlinear doom hazard is driven by total compute that cannot touch. In the concave-doom world the leak hardly matters, because deliverance wins the race regardless. Which is to say, the leak matters most to precisely the people who want to spend the most on safety, and the case that “safety research needs frontier models” is also the case that safety research feeds .
Starting in the middle
The sweeps above all start from zero compute, which is pedagogically convenient but not realistic because there is clearly a bunch of compute out there already (I say, asking Claude to deploy 8 H100s to check my spelling). We need to account for the legacy compute stock that brought us here, which, realistically, included a negligible safety fraction . That is to say, legacy stock is nearly all capability stock, and its hazard has been accumulating for a while already. It is reasonable to apply the split only to new compute; it can dilute the legacy stock but not remove it.
So we need to introduce two new parameters in a forward-looking, calibratable model to include the low-safety-compute past as well as a policy change going forward.
In the linear world the conditional doom probability becomes
which starts at the legacy ratio and drifts toward the constant as new compute swamps the old. How fast it drifts depends on , so the growth rate now matters even in the linear world. At a fixed split, faster growth of total compute lowers . The hazard ratio starts at the legacy ratio (i.e., it is nearly all doom), and only new compute, arriving under the current , drags it towards the new split. More safety compute sooner means we spend less time at the legacy ratio and accumulate less doom hazard while there.
Figure 6: P(doom) vs safety fraction α in the linear world, starting from a legacy capability stock 𝒞_{c,0} of 0, 5 or 20 units and no legacy safety stock. Solid: fast growth (r = 0.3); dashed: slow growth (r = 0.05). With no legacy stock the two growth rates coincide; with one, faster growth dilutes it sooner and the curves separate.
I chose the values 5 and 20 relative to the model’s scale: starting from zero, the linear race resolves at a stock of roughly {python} f"{C_med_lin:.0f}" units, so the red curves describe a world that already holds more capability compute than the whole race would otherwise take to resolve. At that ratio of old to new, the whole sweep of under slow growth moves only from {python} f"{legacy_hi:.2f}" to {python} f"{legacy_lo:.2f}", from near-certain to slightly less certain. Whether the split matters is a question about the ratio of legacy stock to , and I don’t know that ratio for the world, only that it is not zero. Legacy stock also pulls the speed verdict the opposite way from convexity: legacy stock makes faster growth good, because it is diluted sooner, and a convex makes it bad, because the race resolves further out along the curve. Which wins is a question about the same two numbers, the legacy ratio and the overtaking point.
What did we just learn?
When asking whether more compute is good or bad for safety, I argue that question is not well posed, but there are some interesting ones we can access via a more explicit model for how doom and deliverance would actually happen. Some well-posed adjacent questions are:
What are the local shapes of and over the stocks the race passes through before it resolves?
How much compute leaks between the two stocks, in each direction?
How large are the legacy stocks relative to ?
This is, I think, the actual content of the disagreement I started with. The people who say more compute is good believe something like “the race will be over before doom’s curve overtakes safety’s”, or that “the safety leak is small”; the people who say it is bad believe the reverse.
I care about that, because each of those quantities is at least in principle estimable, and they give us a way to quantify a “P(doom)” over some interventions. Evals that track how capability scales with training compute are estimates of the local shape of . Whether safety techniques transfer across model scales is an estimate of the shape of . Whether interpretability results get folded into the next capabilities run is an estimate of the leak. None of these are easy to measure, mind you, but they are the kind of thing we could be wrong about in a checkable way—cruxes—and it is not obvious to me that the discourse has disclosed as many such cruxes as it should.
The other thing the model gives us is a sharper vocabulary for the policy levers. “Pause” and “invest in safety” are not rival strategies; they act on different inputs to the hazard functions, and we can more easily understand how they might complement or substitute for one another in the production function of continued existence by spelling them out. Anyone who advocates one lever in all worlds is implicitly claiming to know which world we are in, and I think as such I would like to see that claim made explicit.
What to model next
This framework is minimal and stylized. Some important aspects it ignores:
Multiple actors. There is no game theory in this model — just a single planner choosing . In practice, the allocation is the outcome of many actors with misaligned incentives.
More complicated hazards than pure time integration. Above we assume
which doesn’t seem crazy to me, but in fact we could imagine that the doom process involves both the compute that went before and the compute currently being done, e.g.
Here, and are parameters that determine the relative importance of past versus current compute in driving the doom hazard. I would chase this version to make the model properly robust.
The possibility of partial doom and incremental deliverance. Both events are modelled as discrete point arrivals — single moments at which the state transitions irreversibly. This is reasonable for doom (a single catastrophe), but strange for deliverance. Real alignment progress is incremental: better interpretability, verified properties, scalable oversight, each partially reducing risk. A more plausible model would replace the deliverance point process with one whose arrivals down-modulate the doom hazard rate — each safety milestone reduces rather than ending the race outright. We could also model incremental doom — each catastrophe raises the baseline risk, although that feels less natural. If we’re worried about bad-but-not-doom events, we’d probably move to some continuous badness index, like “dollar value of harm” or “number of lives lost”, rather than a binary doom/deliverance outcome.
Optimal control. We haven’t solved for the optimal . This is a dynamic optimal control problem, because affects cumulative stocks and at all future times . That sounds fun, but probably not worth investigating because even if the model were true we wouldn’t know the response functions well enough to solve it, and even if we knew how to solve it, I cannot imagine us coordinating to implement that solution.
Granular allocation of compute to many different teams or ideas with different safety/capability profiles, rather than a single aggregate . People have made the case to me that this matters. I think we might be able to produce a more granular model by allocating compute to buckets via some kind of stick-breaking process, then taking the max hazard? Definitely out of scope for this post, but maybe worth exploring in the future.
Granular allocation of compute to “training” or “inference”. Deliberately not explored here because I personally don’t think that distinction will survive for long in even the status quo world, let alone the world where continual inference-time learning is incentivised by training bans. But it would not be such a weird distinction to make in a more detailed model if that kind of thing floats your boat.
A competing-risks model for doom and alignment as a function of compute allocation
The model setup is as follows. We assume two things can happen — an AI catastrophe (“doom”), or a breakthrough that guarantees alignment (“deliverance”). Both become more likely as more compute accumulates in the world, but they draw on different, potentially overlapping, pools of compute. The question becomes: given a trajectory of total compute and a policy for splitting it between capability and safety, what is the probability that doom arrives before deliverance?
The machinery I use is survival analysis of competing risks — the branch of statistics built for this kind of “which stochastic event comes first?” problem. The motivation for expressing everything in terms of compute comes from the economics of cognition: compute is the fundamental currency of intelligence, so we should be able to express both the problem and the solution in terms of how compute is allocated.
Compute trajectories
I think we should distinguish between the rate at which compute is performed and the cumulative “stock” of computation that has been done.
Let
The cumulative compute — total FLOPs performed by time
We assume there exists a division of the compute stream such that a fraction
The cumulative stocks in each pool are then
with
The function
Two competing events
I model doom and deliverance as the first arrivals of inhomogeneous point processes — random events whose chance of firing at any moment depends on how much compute we have accumulated so far:
Doom (
An X-risk catastrophe happens. This is an irreversible absorbing state, because everyone is dead and rendered into computronium or paperclips or whatever.
Deliverance (
We achieve guaranteed alignment — a state after which X-risk from AI is effectively zero, and some kind of end-game high score is unlocked.
Each event has a latent arrival time —
Each event has a hazard rate (instantaneous arrival intensity given that neither event has yet occurred — i.e. while we are still in limbo) that depends on the cumulative stock of compute in its respective pool:
where
Note that these hazard rates depend on time via the cumulative compute stocks. The compute rate
Use the buttons to switch scenarios.Go to the version on my website for the interactive plot.Three outcomes of the race
The probability that the race remains in limbo at time
In words: the probability of still being in limbo decays exponentially as the hazard accumulates. Here
The race has three mutually exclusive outcomes:
where:
In our model the total stock
At each instant
So the probability of doom can also be written
which decomposes into, “probability we’re still waiting at time
Doom probability is the conditional ratio
The constant-hazard-ratio case
Consider an ultra-simple sanity check. If the ratio
and the
It’s also not very plausible, in my opinion. Let’s get more realistically complicated.
Interesting response curves
With non-linear response functions, this invariance breaks down.
Notes on the scale: The race ends at
{python} f"{C_med:.0f}"units, past the crossover. That is monopoly money; an empirically calibrated model would useThe same constant also describes the muddling through, in which neither doom nor deliverance is close: hazards are small at human timescales and
{python} f"{t_med_slow / t_med:.0f}"times as long, but resolves at a stock about{python} f"{C_med_slow / C_med:.0f}"times larger,{python} f"{C_med_slow:.0f}"units rather than{python} f"{C_med:.0f}". Where the race resolves decides which part of theSome scenarios
Each choice of
The linear scenario. Both
Convex-doom. Convex
Concave-doom. Concave
Compute overhang. Sigmoidal
{python} f"{X0_DOOM:.0f}"units of capability stock and safety’s at{python} f"{X0_SAFE:.0f}"units of safety stock, so safety’s switch flips first only when{python} f"{ov_alpha_star:.1f}". The growth rate decides how long the stretch lasts. Under slow growth it is long, the first hazard has the race almost to itself, and{python} f"{ov_slow_half:.2f}"at{python} f"{ov_slow_09:.2f}"at{python} f"{ov_cross:.2f}", near the cliff. Speed therefore cuts both ways. At lowThe pause. A “compute pause” here means a stop to buildout: from
What if safety compute is also capabilities compute?
In the base model the doom hazard depends only on capability compute
so that
The gap between the dashed and solid curves is the cost of the leak, and it grows with
{python} f"{leak_gap['Linear']:.2f}"in the linear world and{python} f"{leak_gap['Convex-doom']:.2f}"in the convex-doom world, where the superlinear doom hazard is driven by total compute thatStarting in the middle
The sweeps above all start from zero compute, which is pedagogically convenient but not realistic because there is clearly a bunch of compute out there already (I say, asking Claude to deploy 8 H100s to check my spelling). We need to account for the legacy compute stock that brought us here, which, realistically, included a negligible safety fraction
So we need to introduce two new parameters in a forward-looking, calibratable model to include the low-safety-compute past as well as a policy change going forward.
In the linear world the conditional doom probability becomes
which starts at the legacy ratio and drifts toward the constant
I chose the values 5 and 20 relative to the model’s scale: starting from zero, the linear race resolves at a stock of roughly
{python} f"{C_med_lin:.0f}"units, so the red curves describe a world that already holds more capability compute than the whole race would otherwise take to resolve. At that ratio of old to new, the whole sweep of{python} f"{legacy_hi:.2f}"to{python} f"{legacy_lo:.2f}", from near-certain to slightly less certain. Whether the split matters is a question about the ratio of legacy stock toWhat did we just learn?
When asking whether more compute is good or bad for safety, I argue that question is not well posed, but there are some interesting ones we can access via a more explicit model for how doom and deliverance would actually happen. Some well-posed adjacent questions are:
This is, I think, the actual content of the disagreement I started with. The people who say more compute is good believe something like “the race will be over before doom’s curve overtakes safety’s”, or that “the safety leak is small”; the people who say it is bad believe the reverse.
I care about that, because each of those quantities is at least in principle estimable, and they give us a way to quantify a “P(doom)” over some interventions. Evals that track how capability scales with training compute are estimates of the local shape of
The other thing the model gives us is a sharper vocabulary for the policy levers. “Pause” and “invest in safety” are not rival strategies; they act on different inputs to the hazard functions, and we can more easily understand how they might complement or substitute for one another in the production function of continued existence by spelling them out. Anyone who advocates one lever in all worlds is implicitly claiming to know which world we are in, and I think as such I would like to see that claim made explicit.
What to model next
This framework is minimal and stylized. Some important aspects it ignores:
Further reading
References
Hooker. 2020. “The Hardware Lottery.” arXiv:2009.06489 [Cs].
Sweeney, and Sterman. 2000. “Bathtub Dynamics: Initial Results of a Systems Thinking Inventory.” System Dynamics Review.
Trammell, and Aschenbrenner. 2024. “Existential Risk and Growth.”