If survival is likely, then the survival posterior consists of ordinary worlds, which you can reason about using existing assumptions/models/trends. However, if survival is unlikely, then the survival posterior consists of weird worlds where our assumptions break down. This makes it much harder to estimate the impact of our interventions, because the world is unprecedented. For example, imagine if brain uploads arrive by 2030 -- this should make us more sceptical of extrapolating various economic trends that were observed before uploads.

And when you condition on survival, you update not just your empirical beliefs but your moral beliefs too. Suppose you're uncertain about (A) whether animals have as much moral weight as humans, and (B) whether we can build machines as smart as humans, and that (A) and (B) are correlated, both downstream of a latent variable like (H) "humans are special." Survival is much more likely if ¬B, so conditioning on survival upweights ¬B, which upweights H, which downweights A. Using the numbers below, your credence in animal moral weight drops from 59% to 34% — nearly halved — just by conditioning on survival. Gnarly!

In principle, survival could be unlikely yet conditioning on it might not make worlds weird. To illustrate: suppose you're certain that humanity's survival depends entirely on the random weight initialisation of a particular pretraining run — 1% chance of good, 99% bad. Conditioning on survival, most survival worlds are ordinary in every respect except for the lucky weight initialisation. The weirdness of the weight initialisation is highly "localised", as opposed the "global weirdness" if (e.g.) aliens have sabotaged our deployment. That said, even if there are some "locally weird" survival worlds in the posterior (e.g. lucky weight initialisation), I expect that on most worldviews, the "globally" weird worlds will dominate.

When survival is unlikely, the survival worlds are more different from each other; this is because all ordinary worlds are alike but each weird world is weird in its own way. And because the survival worlds vary so much, it's harder to find interventions which are robustly beneficial -- an intervention that looks good in one weird survival world is likely to look poor in another. 

For example, suppose you think that if world governments proceed with the expected level of sanity, then ASI will cause extinction. But we might survive because governments showed unexpectedly low sanity (e.g. initiating nuclear conflict over some mundane issue) or unexpectedly high sanity (e.g. updating on early warning shots). Now consider an intervention which shifts power toward existing world governments at the expense of frontier AI labs: this might decrease flourishing if we survived via low governmental sanity and increase flourishing if we survived via high governmental sanity. The intervention's value flips sign depending on which weird world we end up in.

It was a bit tricky to try to justify this intuition, but here's a toy model: imagine worlds as points in R², with a Gaussian prior centered at the origin. Scatter some "attractors" randomly — some red (extinction), some green (survival). Each point in world-space inherits the fate of its nearest attractor. When most of the Gaussian mass falls in red regions, survival requires landing in one of the scattered green islands. These islands might be far apart in world-space. The survival posterior becomes multimodal, spread across disconnected regions. The diagram below illustrates: when P(survival) is low, Var(world | survival) tends to be high.

This diversity creates practical problems:

• Computational: There are more worlds to reason about when choosing interventions.
• Cluelessness: There are more worlds, so it's more likely that you are clueless about the sign of the intervention in some of the worlds.
• Social: There are more worlds, so it's more likely that there is disagreement about the relative likelihood of the worlds, in ways that cause disagreement about the overall sign of the intervention.

Ordinary worlds have more continuity between the present and the future, whereas weird worlds often involve some transitional event that explains why we survived, and these transitional events might 'wash out' your intervention.

For example, suppose you think the current policy landscape is likely to lead to extinction. Then we should be pessimistic about flourishing-focused policy interventions because, conditional on survival, there was probably some large-scale disruption of the policy landscape.