In my experience the "extinction is special" argument is commonly made and needs to be answered. There's a whole fusillade of different arguments made by different people in AI risk, and in the feedback on my essays, I often find people saying "you didn't engage enough with the real reason for X" meaning the reason that person in particular feels is most relevant. I try to touch on a range of arguments, but I agree they're not all equally compelling or popular.
Since we're here, let me sketch my response to the "what if AI extinction is just really likely" argument. In a nutshell, I think it's unhelpful regardless of whether it's true. What matters for planning our future is whether a risk is actionable, and what the side effects of different risk-mitigating actions would be. The leading plan for reducing AI risk, Plan A from AI 2040, would probably do very little to reduce AI risk while causing massive negative side effects, as I explained here.
I think a "normal" approach to AI risk will work well enough, and better than extreme measures in any case.
2This seems like a crux.
For sure it is. But note that I did link an entire article vigorously criticizing Plan A, so I don't see this as a "agree to disagree" matter.
But note that whether it's true is unrelated to whether it's unhelpful. So claiming it's unhelpful is not clearly any sort of relevant response to a "what if AI extinction is just really likely" argument.
The quality of being helpful or unhelpful is intrinsically from the perspective of the person describing it as such. For those who place high intrinsic value on estimating p(doom), regardless of the actionability of such estimates, sound arguments for a high value of p(doom) are helpful. I'm not one of those people.
The only reason I'm here talking to LessWrongers at all is because I care about what we do about AI risk, and this is a forum that influences what humanity will do.
I place no intrinsic value on accurate estimates of p(doom). To me they are only instrumental. What I care about intrinsically is the prospective impact of our potential actions to address AI doom, and their impact on humanity's future overall.
1OK, very interesting! I was not expecting that response. Your view is kind of hard for me to fathom.
I definitely care at a gut level about there being happy more-or-less human people 100,000 years from now, and it seems hard for me to separate this feeling (in my own psyche, not conceptually[1]) from "I care about my own longevity and happiness, and that of people I know." Maybe one way to try to communicate this feeling is "I expect that 'I' will still still be effectively 10-30% 'alive' as long as any humans exist, for the meanings of "I" and "alive" that seem important to me." But I don't think that that is the very core of it for me--it's not really about me.
(The other thing to say is that one of the things I don't like about accelerating AI development is its likely effect on my and my loved ones' longevity. But that is a separate question.)
I also felt confident that a study would find most people being closer to my view. I searched "moral circle future extinction empirical" in Google Scholar and the fourth result is almost the extinction-is-special question as I put it.
Unfortunately, they ask (in the Control condition) whether the difference between 0% and 80% of people dying is more or less important than the difference between 80% and 100%, instead of using equal intervals. Obviously having the non-extinction one be 0%-vs-80% adds complications compared to 60%-vs-80%: not only four times the number of deaths, but also the difference that with 0% there is no world-disrupting catastrophe at all, which might seem like a "special" consideration in the same way as extinction. (A more minor possible problem is that the wording of the questions is that a catastrophe "immediately" kills X%--it could be that some people are thinking that an immediate 80% catastrophe would be followed relatively soon by extinction due to mass starvation.)
They try to correct for these problems but by adding two conditions where there aren't human deaths (called "Animal", with zebras in place of humans, and "Sterilization", with sterilization rather than death). They also have a condition ("Salience") where they remind participants to consider long-term effects and that a recovery from a large population decline is possible, and another condition ("Utopia") where they specify that in the 0% and 80% cases humanity will (not just possibly could) eventually achieve a utopian society (after recovering to its initial numbers, in the latter case). Salience and Utopia use the Control scenario where the catastrophe involves human deaths.
Here are the results, for four different populations of respondents. "Found extinction uniquely bad" y axis means just that they considered the difference between 80 and 100 worse than that between 0 and 80:
My takeaways are:
- Come on guys, why not use 60/80/100? I have to make a bunch of guesses about how much people care about zebras, sterilization, deaths of existing people versus non-births of future people, and world-disrupting-but-not-existential catastrophes to draw conclusions here.
- The Control results are below what I would have expected, even accounting for the weirdness of the 0/80/100 setup. I no longer think that my opinion about the 98/99/100 version is obviously very widespread.
- The framing of the question matters a lot. The Salience condition doesn't add any new information (except to clarify that the 80% catastrophe can be recovered from, and doesn't just spiral into extinction soon after), but it roughly doubles "extinction uniquely bad" for both the UK and US groups. The results from Salience seem more likely to reflect peoples' actual values on reflection than the results from Control.
- The Salience results are still way below what I would have expected on that condition (I'd guess that the unequal intervals thing still probably skews the results somewhat, but still, a large chunk of people seem to have a view I can't fathom). I think they are also higher than you would have expected?
- Sterilization seems like the scenario that most clearly isolates "extinction" from "catastrophe." (Although the mass-sterilization is still stated to have been inflicted by "a catastrophe.") I'm a little surprised at how much it varies between the US and UK--I wonder if this has something to do with different associations with the idea of sterilization. It would be nice to see Sterilization + Salience as a condition.
- The belief that the default non-extinction future is utopian, which many/most longtermists seem to hold, might be the thing making them seem unusually fixated on x-risk, rather than (or above and beyond) a view that extinction is special per se. If you ask people to adopt that belief within the imagined scenario, they suddenly prioritize x-risk much more highly, even when they didn't seem to think extinction was special under the Salience condition.
- ^
Actually, I do think that the common-sense conception of the self that includes every person-moment of the person with your name and chain of memories and personality quirks (presumably including person-moments where those memories and quirks are inactive or very much in the background), and also is expansive enough that wishing your life and psychology were different in significant ways isn't the same as wishing you didn't exist, but not so expansive that it includes person-moments from other very similar people in the future (or the past, or the present), is kind of hard to make conceptually coherent in a naturalistic worldview. But the contradictions could be resolved in a few different ways, some of which might leave you pretty close to the common-sense idea. I haven't fully worked out my thoughts about this.
There is a meaningful risk of AI doom. It is possible that humanity could go extinct at the hands of a future superintelligent machine. Forecasts of AI doom are informative. They’re epistemically helpful.
But epistemically helpful is not the same as practically helpful. An informative forecast is not by itself a guide to action. Knowing a company is destined for bankruptcy does not mean you will profit by shorting its stock; if you get the timing wrong, the market will crush you first. Similarly, knowing one destination we should avoid doesn’t give us a roadmap for the 21st century. There is a large gap between a partial map and a helpful guide.
The century we are now navigating is riddled with potentially catastrophic dangers: climate change tipping points, demographic transition-induced socioeconomic fracture, biological threats, geopolitical chaos. AI could mitigate all of these risks by generating technological and material abundance, decreasing net human extinction risk rather than increasing it. Indefinitely pausing AI development could prevent us from acquiring the very tools we need to survive the other threats of the 21st century.
A recent and increasingly influential theory of ethics, called longtermism, asserts that none of the competing risks to AI doom matter because they are unlikely to destroy the human race entirely, whereas a dangerous superintelligence would probably wipe us out. Longtermism says that extinction is infinitely worse than civilizational collapse because it prevents untold numbers of future humans from living, and our greatest moral duty is to those future humans, who plausibly outnumber us by many orders of magnitude.
But longtermism is not a binding moral framework. It is barely known to humanity, let alone widely accepted. That doesn’t make it wrong — many moral ideas take a while to be accepted — but it does mean that longtermists are less than fully qualified to make unilateral assertions about what humans should value. There is no moral authority — no divine stone tablet, bulletproof logical argument, or even democratically-passed legislation — carrying the commandment thou shalt consider extinction risk infinitely worse than civilizational collapse. Once this is acknowledged, no rational argument remains that we must scrupulously avoid certain kinds of risks with our species.
The broad sweep of history shows that humans have a tendency to seek liberation along dangerous paths. Whether this instinct should be followed in the case of AI depends on the certainty of AI doom, and this should be admitted as at best highly speculative by anyone who comes to the question with integrity.
I think of humanity like Odysseus facing Scylla and Charybdis on his way home. Odysseus was not morally obligated to sit and rot on Circe’s island to minimize the risk of his early demise. He was within his rights to brave the danger. And he could not afford to think only of one monster. He had to watch both as he passed through.
Adapted for LessWrong from my Substack.