There's another issue where "P(doom)" can be read either as the probability that a bad outcome will happen, or the probability that a bad outcome is inevitable. I think the former is usually what's meant, but if "P(doom)" means "the probability that we're doomed", then that suggests the latter as a distracting alternative interpretation.
I think it's Bostrom's notion of existential catastrophe that introduced the conflation of extinction with losing control over most of the cosmic endowment. The "curtail long-term potential" term is ambiguous between things like making current technological level unachievable (with something like a civilization-decimating pandemic that somehow can be survived but never truly recovered from), and making hypothetical CEV-guided capture of future lightcome unachievable, while still granting something like CEV-guided capture of merely our galaxy.
This was cemented in the common use by arguments about AI risk that boiled down to lack of the full CEV-guided future lightcone almost certainly implying extinction, so that there is little use for the category of outcomes somewhere in the middle. But recently there are possibilities like even significantly-alien-on-reflection LLMs capturing the cosmic endowment for themselves without exterminating humanity and possibly even gifting some post-singularity boons like uploading. This makes the non-extinction existential catastrophe a plausible outcome, one much preferable to an extinction existential catastrophe (which still seems likely with even more alien AGI designs that are not far behind, or with LLMs themselves failing the AI risk resistance check by building unaligned AGIs).
Just for the record, I think there are two important and distinguishable P(doom)s, but not the same two as NathanBarnard:
P(Doom1): Literally everyone dies. We are replaced by either by dumb machines with no moral value (paperclip maximisers) or by nothing.
P(Doom2): Literally everyone dies. We are replaced by machines with moral value (conscious machines?), who go on to expand a rich culture into the universe.
Doom1 is cosmic tragedy - all known intelligence and consciousness are snuffed out. There may not be any other elsewhere, so potentially forever.
Doom2 is maybe not so bad. We all die, but we were all going to die anyway, eventually, and lots of us die without descendants to carry our genes, and we don't think that outcome is so tragic. Consciousness and intelligence spreads thru the universe. It's a lot like what happened to our primate ancestors, before Homo sapiens. In some sense the machines are our descendants (if only intellectual) and carry on the enlightening of the universe.
Oh yes, this is also a very important distinction (although I would only value outcome 2 if the machines were conscious and living good lives.)
Some related work:
I agree. I think we should break "doom" into at least these four outcomes {human extinction, humans remain on Earth} x {lots of utility achieved, little to no utility} ( )
Upvoted for a fresh, non-forced by an ultra-utilitaristic POV. With this approach, p(Giga-doom) is also much lower, I guess.
I think it's actively unhelpful to talk about P(doom) because it fails to distinguish between literally everyone dying and humanity failing to capture 99.99% of the value of the future under a total utilitarian view but in practice, everyone who's born lives a very good life. These are very different outcomes and it's unhelpful not to distinguish between them and everything else in that spectrum.
This is especially the case since astronomical waste arguments really only bite for total utilitarian views. Under moral views where potential people not coming into existence is more similar to preventing someone from living an extraordinarily happy life rather than a merely happy life, as opposed to preventing someone coming into existence being similar to murder, it's quite reasonable to prioritise other goals well above preventing astronomical waste. Under these non-totalalist views preventing totalitarian lock-in or S-risks might look much more important than ensuring we don't create 10^(very large number) of happy lives.
I think this also matters on a practical level when talking about threat models of AI risks. Two people could have the same p(doom) but one is talking about humans being stripped for their atoms and the other is talking about slow disempowerment in which no one actually dies and everyone, in fact, could be living very good lives but humanity isn't able to capture almost all of the value of future from a total utilitarian perspective. These plausibly require different interventions to stop them from happening.
It also seems like one's prior on humanity going extinct as a result of AI should be quite different from disempowerment, but people often talk about what their prior on P(doom) should be as a univariate probability distribution.