1. The Case

You've probably heard something like this before:

1. If we survive this century, the expected value of the future is massive.
2. If we don't survive, the expected value is near zero.
3. Therefore, the value of an intervention is approximately proportional to how much it increases the chance of survival.

You won't go badly wrong following the conclusion, but (3) doesn't actually follow from (1) and (2). That's because interventions might vary in how they affect the expected value of the future conditional on survival.^[1]

Will MacAskill makes roughly this argument in Better Futures (August 2025). See the diagram below: survival-focused interventions target the red rectangle, flourishing-focused interventions target the blue. But the blue rectangle might be much larger than the red rectangle -- if x-risk is 20% then even the best survival intervention can increase EV by at most 1.2x, whereas a flourishing intervention could increase EV by 5x or 5000x.

But is x-risk only 20%? MacAskill thinks so,^[2] but his argument applies even if extinction is very likely — say 99% — so long as there are interventions that increase flourishing by +100x. That's the scenario I want to discuss here in this post.

My conclusion is:

1. When survival is unlikely, focusing on flourishing becomes qualitatively different in ways that cut against focusing on flourishing.
2. There are various strategies for focusing on flourishing when survival is unlikely, but none are attractive. 
3. Overall, when survival is unlikely, we shouldn't focus on flourishing. Even if you buy that the best possible futures are hundreds of times better than the middling futures.

2. Challenges

Recall that flourishing is the expected value conditional on survival. That is, flourishing-focused interventions target the survival posterior, consisting of the green and blue rectangles. Consequentially, if survival is likely then the survival posterior consists of ordinary futures, but if survival is unlikely then the survival posterior consists of weird futures, worlds very different from what we'd expect.^[3]

What kind of weird worlds?

• An AI-enabled coup has occurred
• Terrorist attacks on datacentres have halted development
• Civilisational collapse destroyed most human knowledge
• AGI turned out to be theoretically impossible
• Aliens/simulators sabotaged our AI development
• US and China start a nuclear conflict
• Brain uploads arrived far sooner than expected^[4]
• Extremely surprising civilisational sanity
• Various cases survival without dignity

For more possible futures, see Bart Bussmann's 60+ Possible Futures; which weird survival worlds are most likely will depend on your cause for pessimism.

This poses three problems:

Problem 1: Survival world is harder to reason about. 

If survival is likely, then the survival posterior consists of ordinary worlds, which you can reason about using existing assumptions/models/trends. However, if survival is unlikely, then the survival posterior consists of weird worlds where our assumptions break down. This makes it much harder to estimate the impact of our interventions, because the world is unprecedented. For example, imagine if brain uploads arrive by 2030 -- this should make us more sceptical of extrapolating various economic trends that were observed before uploads.

And when you condition on survival, you update not just your empirical beliefs but your moral beliefs too. Suppose you're uncertain about (A) whether animals have as much moral weight as humans, and (B) whether we can build machines as smart as humans, and that (A) and (B) are correlated, both downstream of a latent variable like "humans are special." Survival is much more likely if ¬B, so conditioning on survival upweights ¬B, which upweights H, which downweights A. Using the numbers below, your credence in animal moral weight drops from 59% to 34% — nearly halved — just by conditioning on survival. Gnarly!

Problem 2: Surviving worlds are more diverse.

When survival is unlikely, the survival worlds are more different from each other; this is because all ordinary worlds are alike but each weird world is weird in its own way. And because the survival worlds vary so much, it's harder to find interventions which are robustly beneficial -- an intervention that looks good in one weird survival world is likely to look poor in another. 

For example, suppose you think that if world governments proceed with the expected level of sanity, then ASI will cause extinction. But we might survive because governments showed unexpectedly low sanity (e.g. initiating nuclear conflict over some mundane issue) or unexpectedly high sanity (e.g. updating on early warning shots). Now consider an intervention which shifts power toward existing world governments at the expense of frontier AI labs: this might decrease flourishing if we survived via low governmental sanity and increase flourishing if we survived via high governmental sanity. The intervention's value flips sign depending on which weird world we end up in.

It was a bit tricky to try to justify this intuition, but here's a toy model: imagine worlds as points in R², with a Gaussian prior centered at the origin. Scatter some "attractors" randomly — some red (extinction), some green (survival). Each point in world-space inherits the fate of its nearest attractor. When most of the Gaussian mass falls in red regions, survival requires landing in one of the scattered green islands. These islands might be far apart in world-space. The survival posterior becomes multimodal, spread across disconnected regions. The diagram below illustrates: when P(survival) is low, Var(world | survival) tends to be high.

This diversity creates practical problems:

• Computational: There are more worlds to reason about when choosing interventions.
• Cluelessness: There are more worlds, so it's more likely that you are clueless about the sign of the intervention in some of the worlds.
• Social: There are more worlds, so it's more likely that there is disagreement about the relative likelihood of the worlds, in ways that cause disagreement about the overall sign of the intervention.

In the next post, I will discuss potential strategies for focusing on flourishing when survival is unlikely. These strategies will aim to overcome some or all of the problems above.

1. ^{^}

   In maths:

   Assuming that E(value|not survival) ≈ 0, we can decompose E(value|intervention) into the product of E(value|survival, intervention) and P(survival | intervention).

   This suggests that P(survival | intervention) is a good proxy for E(value|intervention), but this is only true if E(value|survival, intervention) doesn't vary much across interventions.

   However, E(value|survival, intervention) might vary significantly.

2. ^{^}

   To illustrate, suppose you think that our chances of survival this century are reasonably high (greater than 80%) but that, if we survive, we should expect a future that falls far short of how good it could be (less than 10% as good as the best feasible futures). These are close to my views; the view about Surviving seems widely-held, and Fin Moorhouse and I will argue in essays 2 and 3 for something like that view on Flourishing.

   Introduction Better Futures (MacAskill, 3rd Aug 2025)

3. ^{^}

   Caveat: In principle, survival could be unlikely yet conditioning on it might not make worlds weird. To illustrate: suppose you're certain that humanity's survival depends entirely on the random weight initialisation of a particular pretraining run — 1% chance of good, 99% bad. Conditioning on survival, most survival worlds are ordinary in every respect except for the lucky weight initialisation. The weirdness of the weight initialisation is highly localised, so it doesn't raise the three problems above.

   That said, I don't think such worldviews are plausible, because they require a very high prior on ordinary worlds. I think that plausible worldviews should place +1% probability on "worlds which are globally weird and we survive". And so these worlds will dominate the survival posterior, even if there are also some "worlds which are locally weird and we survive".

4. ^{^}

   See State of Brain Emulation Report 2025 (Zanichelli et al., 17th Oct 2025)