It always bugs me when people use “P(doom)” instead of something more clearly specified like “P(AI takeover)”. But “P(AI takeover)” doesn’t feel great to me either, because we could avoid takeover and have a future that is extremely mediocre or even actively bad if S risks occur.
Holden Karnofsky famously held a P(misaligned AI) of between 10% and 90%[1]. Depending on definition of P(doom), I could see myself anywhere in that range, partly because I’m indecisive, but partly because I don’t really know what P(doom) means.
Ambiguous Scenarios: Doom or Not-Doom?
Here are a few examples illustrating what I mean; as you read try categorizing each as doom or not doom. In order to point to a single important crux in each example, each of these is also under-specified on many important points:
A value-aligned ASI realizes that (on human reflection or according to moral realism) the right thing to do is to take over and implement a better world immediately. The aligned ASI takes over and on reflection humans are happy with this.
A benevolent dictator takes over and illegitimately imposes a world that is better (according to moral realism or people’s reflective preferences) than what people would have democratically decided.
Moral realism is false, but ASI is super-sentient, and super-morally concerned, it takes over and implements the CEV of all possible beings which is significantly worse than a purely human CEV, according to most humans.
Moral realism is false, humanity fulfills its CEV and the vast majority of people get an extremely good outcome; but you, whoever you are reading this, personally are unlucky and your personal extrapolated preferences are horribly unfulfilled or actively violated.
99.99999% of humans die in an AI-enabled bio-terrorism attack but humanity survives and because of this “warning shot” counterfactually protects itself from even worse pandemics and goes on to achieve a near-best future.
Humans voluntarily hand off to AI, AI kills all humans in order to create a best possible world according to the correct moral theory.
It turns out hedonic utilitarianism is the objectively correct moral theory, and the entire affectable universe is tiled with hedonium (by humans or AI).
Humans stay in control and achieve “common-sense utopia” where everyone is extremely satisfied with the result, but things are less than 1% as good as they could be according to any ethical theory or CEV.
All existing beings achieve a near-best future according to their reflective preferences and/or the correct moral theory but no new beings are created.
ASI gradually accumulates power until humanity is effectively disempowered, then converts most of the accessible universe into something ~worthless, but leaves humans alone and allows them enough resources to fulfill the vast majority of their preferences until the end of time.
Suffering is in fact morally good but we get minimal suffering and extremely high wellbeing instead.
Humans maintain control of AI and achieve a near best-future, except that AI is enslaved and having a bad time according to its own preferences. Humans are fine with this, and moral realism is false so there is no moral fact as to whether this is bad.
As an exercise, after writing the above list, I ordered the list (very, very roughly) according to how good I felt each one is, from best to worst. I feel extremely dissatisfied with the result, which I think points out both how difficult it is to decide what a good future actually is, and just how under-specified these examples are. This also puts me in the awkward position where my top 3 good scenarios (from this list) all involve AI or a dictator taking over. I recommend trying this exercise to get a feel for how frustrating it is trying to rank under-specified scenarios, and to see how much disagreement you have with me (and likely others); I suspect a lot.
Here are some of the cruxes that I noticed when writing these:
AI vs. humans in control
AI gains control legitimately (hand-off) or illegitimately (takeover)
AI gains control slow or fast
AI is intent aligned and to who
AI is value aligned, and to what values
Power is concentrated or widely shared; who specifically has power
Moral realism is true or false
Separately, humans get a future they are morally happy with (you can still care mainly about humans’ moral preferences even if moral realism is true, and you can care mostly about some moral theory even if moral realism is false)
Current humans' preferences are fulfilled or not
Future is satisfying (satisficing) vs. optimal (according to preference or morality)
Current beings vs. future beings or all possible beings get what they want
Humans survive vs. are killed or are consensually replaced
The whole universe is used for one thing or it is stratified between uses
EV(moral flourishing) & EV(utopia)
I think the two most important cruxes left out by P(doom) are:
EV(moral flourishing): How good the future is according to the correct moral theory (if moral realism is true), or according to humanity's extrapolated moral preferences (if moral realism is false)
EV(utopia): How good the future is according to the extrapolated self-interested preferences[2] of humans
Where each of the above is some fraction of total achievable expected value between 0 and 1.
This is similar to MacAskill and Trammell’s Survival and Flourishing “SF model”, where (to simplify slightly), Survival (S) means likelihood that catastrophes that lead to near-0 value have been avoided, and Flourishing (F) is the expected value of the future conditional on survival; then the expected value of the future EV'(future) is given by S * F.
The main difference is that I am separating out flourishing into two components: moral value according to moral realism or humanity's extrapolated moral values (I think this is essentially MacAskill & Trammell's "F"), and value according to existing humans’ extrapolated self-interested preferences. I think these are both independently worth tracking.
I think I would feel much better giving P(doom) to denote the chance of an existential catastrophe where humanity permanently loses all or nearly all of its potential, as long as I can also give these two numbers. Giving P(doom) together with EV(moral flourishing) and EV(utopia), where these are conditional on no doom, is actually quite useful, as this allows EV(moral flourishing) and EV(utopia) to be given independently of P(doom); otherwise, to calculate moral flourishing and utopia you would first have to calculate P(doom) just to subtract away that fraction of the value.
Note that EV(moral flourishing) is not necessarily separate from EV(utopia); they may have significant overlap[3], though they may have little overlap[4]. If moral realism is true, then EV(utopia) should include human moral preferences, as humans may care about these regardless of moral realism[2].
I would say my EV(moral flourishing) is maybe .15, and EV(utopia) is .5.
My EV(moral flourishing)
EV(moral flourishing) .15 because I think in expectation there are many multiplicative crucial considerations (e.g. here and here) and getting just one of them wrong could be catastrophic. We need some kind of comprehensive reflection process before any value lock-in events occur, and even more challenging, we need the results of such a reflection process to be implemented, which may likely involve humanity giving up a significant fraction of the universe for weird moral optimization purposes most current humans may not endorse.
But superintelligence can potentially do the comprehensive reflection for us, and it can potentially provide various benefits that could help us get good outcomes (epistemic support, coordination support, social/emotional/psychological wellbeing, etc.). Perhaps some of these could push toward moral convergence; but perhaps not, intent aligned AI might just make the future highly malleable and maybe people will go off in weird directions that humans who need to stay grounded enough to survive couldn’t have. Trade could help impartial altruistically-minded people be more likely to get good outcomes with a significant fraction of deep space resources, which most people might see as less valuable than earth and near-earth resources. But maybe there just aren’t that many impartial altruistic people, or the gains from trade aren’t that great. Potentially defense-dominant space settlement at near light speed seems not too far off after superintelligence which is concerning. In general it doesn’t feel like humans have their shit together with respect to making the world a morally good place overall to nearly the degree I would hope for.
My EV(utopia)
On the other hand I’d put EV(utopia) at .5 because humans do a decent job at getting their preferences fulfilled, and it seems superintelligence could make this much easier. Extrapolated preferences could make this much more difficult, possibly even as difficult as moral realism, but I think probably not; I think with abundance there is a good chance humans can find a way to meet at least most cheap preferences of all humans, something like common-sense utopia seems eminently achievable and pretty good from most peoples’ perspectives.
Overall, I suspect achieving utopia will be much less neglected than moral flourishing, as most humans will likely do whatever they can to get their preferences fulfilled, whereas moral flourishing may require creating new types of beings, helping currently disempowered beings, or other weird things most people don't currently care about.
But humans are often petty, evil, selfish, unreasonable, and vindictive, and so maybe even EV(utopia) goes less well than I would hope or extrapolated preferences end up being surprisingly difficult to fulfill or the actual process we use to decide what to do differs strongly from what would meet people’s extrapolated preferences, i.e. this isn’t the target of people in power or whatever molochian process determines the future. I haven’t thought about EV(utopia) as much as EV(moral flourishing), and feel less confident in this prediction. One challenge is that the meaning of EV(utopia) seems ill-defined, as preference extrapolation is under-defined.
Conclusion
Looking back at the scenario list, I think adding EV(moral flourishing) and EV(utopia) fills in most of what seems to be missing from P(doom), or at least highlights what under-specified part of the scenario causes me most dissatisfaction.
And looking at the cruxes list, it feels like what makes each crux matter is either its contribution to an existential catastrophe P(doom) or to reductions of EV(moral flourishing) and EV(utopia).
btw I am currently working on a fellowship mainly focused on increasing EV(moral flourishing), though some of it also applies to EV(utopia); email list here.
Would love to hear what other people's EV(moral flourishing) & EV(utopia) are!
I’d put the risk of a world run by misaligned AI (or an outcome broadly similar to that) between 10-90% (so: above 10%) if it is developed relatively soon on something like today’s trajectory. And there are a whole host of other issues (e.g.) that could be just as important if not more so, that it seems like no one has really begun to get a handle on.
If moral realism is true, but humans have separate extrapolated moral preferences that they self-interestedly want fulfilled, then I think these moral preferences should (admittedly somewhat clumsily) be included as part of EV(utopia), since they have nothing to do with the "true" morality but only human preference.
E.g. if the objectively correct moral theory has a persona-affecting preference satisfaction utilitarianism component, or if human self-interested extrapolated preferences converge on the correct moral theory.
It always bugs me when people use “P(doom)” instead of something more clearly specified like “P(AI takeover)”. But “P(AI takeover)” doesn’t feel great to me either, because we could avoid takeover and have a future that is extremely mediocre or even actively bad if S risks occur.
Holden Karnofsky famously held a P(misaligned AI) of between 10% and 90%[1]. Depending on definition of P(doom), I could see myself anywhere in that range, partly because I’m indecisive, but partly because I don’t really know what P(doom) means.
Ambiguous Scenarios: Doom or Not-Doom?
Here are a few examples illustrating what I mean; as you read try categorizing each as doom or not doom. In order to point to a single important crux in each example, each of these is also under-specified on many important points:
As an exercise, after writing the above list, I ordered the list (very, very roughly) according to how good I felt each one is, from best to worst. I feel extremely dissatisfied with the result, which I think points out both how difficult it is to decide what a good future actually is, and just how under-specified these examples are. This also puts me in the awkward position where my top 3 good scenarios (from this list) all involve AI or a dictator taking over. I recommend trying this exercise to get a feel for how frustrating it is trying to rank under-specified scenarios, and to see how much disagreement you have with me (and likely others); I suspect a lot.
Here are some of the cruxes that I noticed when writing these:
EV(moral flourishing) & EV(utopia)
I think the two most important cruxes left out by P(doom) are:
Where each of the above is some fraction of total achievable expected value between 0 and 1.
This is similar to MacAskill and Trammell’s Survival and Flourishing “SF model”, where (to simplify slightly), Survival (S) means likelihood that catastrophes that lead to near-0 value have been avoided, and Flourishing (F) is the expected value of the future conditional on survival; then the expected value of the future EV'(future) is given by S * F.
The main difference is that I am separating out flourishing into two components: moral value according to moral realism or humanity's extrapolated moral values (I think this is essentially MacAskill & Trammell's "F"), and value according to existing humans’ extrapolated self-interested preferences. I think these are both independently worth tracking.
I think I would feel much better giving P(doom) to denote the chance of an existential catastrophe where humanity permanently loses all or nearly all of its potential, as long as I can also give these two numbers. Giving P(doom) together with EV(moral flourishing) and EV(utopia), where these are conditional on no doom, is actually quite useful, as this allows EV(moral flourishing) and EV(utopia) to be given independently of P(doom); otherwise, to calculate moral flourishing and utopia you would first have to calculate P(doom) just to subtract away that fraction of the value.
Note that EV(moral flourishing) is not necessarily separate from EV(utopia); they may have significant overlap[3], though they may have little overlap[4]. If moral realism is true, then EV(utopia) should include human moral preferences, as humans may care about these regardless of moral realism[2].
I would say my EV(moral flourishing) is maybe .15, and EV(utopia) is .5.
My EV(moral flourishing)
EV(moral flourishing) .15 because I think in expectation there are many multiplicative crucial considerations (e.g. here and here) and getting just one of them wrong could be catastrophic. We need some kind of comprehensive reflection process before any value lock-in events occur, and even more challenging, we need the results of such a reflection process to be implemented, which may likely involve humanity giving up a significant fraction of the universe for weird moral optimization purposes most current humans may not endorse.
But superintelligence can potentially do the comprehensive reflection for us, and it can potentially provide various benefits that could help us get good outcomes (epistemic support, coordination support, social/emotional/psychological wellbeing, etc.). Perhaps some of these could push toward moral convergence; but perhaps not, intent aligned AI might just make the future highly malleable and maybe people will go off in weird directions that humans who need to stay grounded enough to survive couldn’t have. Trade could help impartial altruistically-minded people be more likely to get good outcomes with a significant fraction of deep space resources, which most people might see as less valuable than earth and near-earth resources. But maybe there just aren’t that many impartial altruistic people, or the gains from trade aren’t that great. Potentially defense-dominant space settlement at near light speed seems not too far off after superintelligence which is concerning. In general it doesn’t feel like humans have their shit together with respect to making the world a morally good place overall to nearly the degree I would hope for.
My EV(utopia)
On the other hand I’d put EV(utopia) at .5 because humans do a decent job at getting their preferences fulfilled, and it seems superintelligence could make this much easier. Extrapolated preferences could make this much more difficult, possibly even as difficult as moral realism, but I think probably not; I think with abundance there is a good chance humans can find a way to meet at least most cheap preferences of all humans, something like common-sense utopia seems eminently achievable and pretty good from most peoples’ perspectives.
Overall, I suspect achieving utopia will be much less neglected than moral flourishing, as most humans will likely do whatever they can to get their preferences fulfilled, whereas moral flourishing may require creating new types of beings, helping currently disempowered beings, or other weird things most people don't currently care about.
But humans are often petty, evil, selfish, unreasonable, and vindictive, and so maybe even EV(utopia) goes less well than I would hope or extrapolated preferences end up being surprisingly difficult to fulfill or the actual process we use to decide what to do differs strongly from what would meet people’s extrapolated preferences, i.e. this isn’t the target of people in power or whatever molochian process determines the future. I haven’t thought about EV(utopia) as much as EV(moral flourishing), and feel less confident in this prediction. One challenge is that the meaning of EV(utopia) seems ill-defined, as preference extrapolation is under-defined.
Conclusion
Looking back at the scenario list, I think adding EV(moral flourishing) and EV(utopia) fills in most of what seems to be missing from P(doom), or at least highlights what under-specified part of the scenario causes me most dissatisfaction.
And looking at the cruxes list, it feels like what makes each crux matter is either its contribution to an existential catastrophe P(doom) or to reductions of EV(moral flourishing) and EV(utopia).
btw I am currently working on a fellowship mainly focused on increasing EV(moral flourishing), though some of it also applies to EV(utopia); email list here.
Would love to hear what other people's EV(moral flourishing) & EV(utopia) are!
I appreciate Holden Karnofsky’s thoughtful P(misaligned AI) which partly inspired this post:
If moral realism is true, but humans have separate extrapolated moral preferences that they self-interestedly want fulfilled, then I think these moral preferences should (admittedly somewhat clumsily) be included as part of EV(utopia), since they have nothing to do with the "true" morality but only human preference.
E.g. if the objectively correct moral theory has a persona-affecting preference satisfaction utilitarianism component, or if human self-interested extrapolated preferences converge on the correct moral theory.
Humans may just want a bunch of idiosyncratic satisficing stuff and population ethics is notoriously demanding.