"Everyone dying" does not emotionally feel that bad to me. I know that it is quite a bad thing intellectually, but I have a missing mood. I'm guessing I'm not alone here. When people say things like "If anyone builds it, everyone dies", there is an implicit assumption that everyone is on the same page that "everyone dies" is a really bad, perhaps even the worst, outcome. But not everyone feels this way, and among those, not many have shut up and multiplied to correct for their brain's biases. I thought it would be useful to explore possible reasons why extinction may not feel as bad as it should.
Extinction lacks some negative features of lesser catastrophes
To be clear, I am not a death fan or extinction enjoyer, I want us to all live forever. The lack of these negative features in extinction clearly does not outweigh the negative fact of the deaths themselves, but are something that some parts of your brain may unduly weigh.
No one is left to suffer
One reason death is bad is that the people who remain must now suffer the loss of the deceased. This is the thing that causes the most visceral bad feeling when I think of death. If 10% of people die, then probably you or some people you know will be among the 10%. That's a lot of people mourning loved ones and friends, coworkers and acquaintances.
But if we all go together, this very emotionally distraught and negative valence experience of grieving does not happen at all (or if there is some lag in the disassembling of our atoms, the grief will be brief).
It also feels really bad to imagine the world just moving on after someone passes away. This also does not happen at all under extinction: the main "world" we care about is the world of people. Imagining that machines are doing their thing, with no people around, does not evoke much feelings within me.
Everyone is in it together
One of the emotional reactions to premature death is that it is unfair that the deceased didn't get to live as long as everyone else. It's like when you get in trouble in school by yourself versus if you and all your friends get in trouble together. The latter is worse than the former, but it feels less bad for you. Even though it is strictly a worse situation than "only some people die from AI", everyone being in it together in extinction means that no one is singled out, which feels comforting in a perverse way.
I remember in the book Einstein's Dreams, there was a vignette where, upon learning that the world will end at a fixed date in the near future, people stop being mean to each other and stop worrying about trivial things in life. They all hold hands and celebrate together in the end. Although one should not generalize from fictional evidence, you can take from this that it is "narratively natural" for some people, such as this book's author, to see positives in extinction.
Compare with simulation shutdown
Suppose we were in a simulation. What if one day the simulation were paused and never turned back on again? This is basically everyone dying. This also doesn't feel that bad though. It feels like turning off a computer. I think it is because nothing bad happens "in universe" to anyone. There is no graphic violence or gradual decay via aging. No one experiences suffering from the pause, and no one can feel any sense of suffering or loss after the pause.
The situation where AI suddenly causes extinction looks pretty much the same from our perspective as simulation shut down.
Copes for natural death work on extinction too
One prominent rationalist stance is "Death is bad". As discussed by nostalgebraist here, this is not a universal stance. A lot of people accept death (at least, death from aging) as a necessary fact of life.
To a rationalist/EA/longtermist anti-deathist, "everyone dies" means that basically the worst thing that can happen to a person happens to everyone, whereas if things went better, we could eventually have conquered death and lived very very long happy lives. It also means that all the future people who could've existed now very much do not.
But, if you assume the frame of the non-anti-deathist, "everyone dies" means that everyone dies earlier than they would have. Non-anti-deathists have a mental posture they take towards natural death, which makes natural death seem "ok". The mechanics of this posture might be learned helplessness, nihilism, "it is what it is", or a deep okayness. The same mental move can be applied to extinction, such that extinction is "ok".
It is gauche to reveal your religiosity
A lot of people are religious, and this means they might have very different ideas about the nature and badness of death and extinction, since so much of religious machinery goes into making death seem ok. You may not think extinction is super bad if you think people will continue on in a decent afterlife. But these things feel socially inappropriate to bring up in scientific materialist spaces, such as conversations about AI risk. It is hard to have productive conversations when people feel they must hide their cruxes.
Extinction is emotionally and practically hard to visualize
People visualize “a single exhausted bird, its feathers soaked in black oil, unable to escape.” This image, or prototype, calls forth some level of emotional arousal that is primarily responsible for willingness-to-pay [to save the birds]—and the image is the same in all cases [of 2000, 20000, and 200,000 birds].
What is the image you come up with when you think of extinction? The image that comes to my mind is an empty Earth. But my mind's eye has limited resolution. The picture looks like a picture of Earth from space. You can't really see people even if they were there. So the prototypical image I see is basically just a picture of Earth, and some mental label overlaid emphasizing "no one is here", which isn't that scary looking at all.
There are worse images I could try but I'd have to do it deliberately. Since no one is left to grieve, the archetypal image of a funeral for the deceased doesn't really make sense. Maybe people dying from AI created viruses? That is pretty uncomfortable to visualize so I'm not gonna do that. I think some people, perhaps myself included, do not really want to think about things like death and extinction in vivid detail, so their reactions might be to shoo the concept away, or to interact with the words with mental gloves, not really reflecting on what they mean.
In addition, the badness of how the moments leading to extinction look like is not where most of the full badness of extinction lies. The major badness is the loss of all the potential and experiences and joy and flourishing that we could have had. You could instead visualize some really good lives, and imagine these being blocked off? Taken away? But it is pretty hard to capture this kind of loss/absence into a mental representation that resonates the right way. And even then if you have such a picture, it is unlikely to capture the real, full version of extinction due to scope insensitivity.
Extinction is hard to believe
I've gone over some reasons why extinction may not evoke strong emotional reactions, but probably many people do not even consider it as a possibility. It sounds like fiction. People want to hear a plausible sequence of steps for how AI can cause extinction, but for any possible such sequence a human may say, there is some place you can say, "But why can't we just X to stop that?" or "But AI could never do that!" Unhelpfully, history has no prior examples of human extinction. (Well, there were various hominid species, but isn't that different? They were not as intelligent as us!) Even if someone believes it is plausible, lots of people, including AI researchers and lab CEOs, say there is a very small chance of it happening. If probabilities are not your native language, it is easy to round a few percentage points down to zero.
Conclusion
Extinction is invoked by AI safety folks as the "infinitely bad outcome", but, even though most people may say they agree to not look like a mundicider, I'm unsure most people feel this way in a way that changes their behavior. This can lead to a lot of talking past each other.
I'm not sure if it's a good or bad thing to not feel the badness of extinction "at a gut level". I hear about people who care about x-risk getting really really stressed out and mentally ill. It seems like it might be good to not be super stressed out and afraid? Keep the grim-o-meter firmly detached?
Curious to hear what others think and how they've related to the idea of extinction emotionally, and also how you've seen other people react to it.
"Everyone dying" does not emotionally feel that bad to me. I know that it is quite a bad thing intellectually, but I have a missing mood. I'm guessing I'm not alone here. When people say things like "If anyone builds it, everyone dies", there is an implicit assumption that everyone is on the same page that "everyone dies" is a really bad, perhaps even the worst, outcome. But not everyone feels this way, and among those, not many have shut up and multiplied to correct for their brain's biases. I thought it would be useful to explore possible reasons why extinction may not feel as bad as it should.
Extinction lacks some negative features of lesser catastrophes
To be clear, I am not a death fan or extinction enjoyer, I want us to all live forever. The lack of these negative features in extinction clearly does not outweigh the negative fact of the deaths themselves, but are something that some parts of your brain may unduly weigh.
No one is left to suffer
One reason death is bad is that the people who remain must now suffer the loss of the deceased. This is the thing that causes the most visceral bad feeling when I think of death. If 10% of people die, then probably you or some people you know will be among the 10%. That's a lot of people mourning loved ones and friends, coworkers and acquaintances.
But if we all go together, this very emotionally distraught and negative valence experience of grieving does not happen at all (or if there is some lag in the disassembling of our atoms, the grief will be brief).
It also feels really bad to imagine the world just moving on after someone passes away. This also does not happen at all under extinction: the main "world" we care about is the world of people. Imagining that machines are doing their thing, with no people around, does not evoke much feelings within me.
Everyone is in it together
One of the emotional reactions to premature death is that it is unfair that the deceased didn't get to live as long as everyone else. It's like when you get in trouble in school by yourself versus if you and all your friends get in trouble together. The latter is worse than the former, but it feels less bad for you. Even though it is strictly a worse situation than "only some people die from AI", everyone being in it together in extinction means that no one is singled out, which feels comforting in a perverse way.
I remember in the book Einstein's Dreams, there was a vignette where, upon learning that the world will end at a fixed date in the near future, people stop being mean to each other and stop worrying about trivial things in life. They all hold hands and celebrate together in the end. Although one should not generalize from fictional evidence, you can take from this that it is "narratively natural" for some people, such as this book's author, to see positives in extinction.
Compare with simulation shutdown
Suppose we were in a simulation. What if one day the simulation were paused and never turned back on again? This is basically everyone dying. This also doesn't feel that bad though. It feels like turning off a computer. I think it is because nothing bad happens "in universe" to anyone. There is no graphic violence or gradual decay via aging. No one experiences suffering from the pause, and no one can feel any sense of suffering or loss after the pause.
The situation where AI suddenly causes extinction looks pretty much the same from our perspective as simulation shut down.
Copes for natural death work on extinction too
One prominent rationalist stance is "Death is bad". As discussed by nostalgebraist here, this is not a universal stance. A lot of people accept death (at least, death from aging) as a necessary fact of life.
To a rationalist/EA/longtermist anti-deathist, "everyone dies" means that basically the worst thing that can happen to a person happens to everyone, whereas if things went better, we could eventually have conquered death and lived very very long happy lives. It also means that all the future people who could've existed now very much do not.
But, if you assume the frame of the non-anti-deathist, "everyone dies" means that everyone dies earlier than they would have. Non-anti-deathists have a mental posture they take towards natural death, which makes natural death seem "ok". The mechanics of this posture might be learned helplessness, nihilism, "it is what it is", or a deep okayness. The same mental move can be applied to extinction, such that extinction is "ok".
It is gauche to reveal your religiosity
A lot of people are religious, and this means they might have very different ideas about the nature and badness of death and extinction, since so much of religious machinery goes into making death seem ok. You may not think extinction is super bad if you think people will continue on in a decent afterlife. But these things feel socially inappropriate to bring up in scientific materialist spaces, such as conversations about AI risk. It is hard to have productive conversations when people feel they must hide their cruxes.
Extinction is emotionally and practically hard to visualize
What is the image you come up with when you think of extinction? The image that comes to my mind is an empty Earth. But my mind's eye has limited resolution. The picture looks like a picture of Earth from space. You can't really see people even if they were there. So the prototypical image I see is basically just a picture of Earth, and some mental label overlaid emphasizing "no one is here", which isn't that scary looking at all.
There are worse images I could try but I'd have to do it deliberately. Since no one is left to grieve, the archetypal image of a funeral for the deceased doesn't really make sense. Maybe people dying from AI created viruses? That is pretty uncomfortable to visualize so I'm not gonna do that. I think some people, perhaps myself included, do not really want to think about things like death and extinction in vivid detail, so their reactions might be to shoo the concept away, or to interact with the words with mental gloves, not really reflecting on what they mean.
In addition, the badness of how the moments leading to extinction look like is not where most of the full badness of extinction lies. The major badness is the loss of all the potential and experiences and joy and flourishing that we could have had. You could instead visualize some really good lives, and imagine these being blocked off? Taken away? But it is pretty hard to capture this kind of loss/absence into a mental representation that resonates the right way. And even then if you have such a picture, it is unlikely to capture the real, full version of extinction due to scope insensitivity.
Extinction is hard to believe
I've gone over some reasons why extinction may not evoke strong emotional reactions, but probably many people do not even consider it as a possibility. It sounds like fiction. People want to hear a plausible sequence of steps for how AI can cause extinction, but for any possible such sequence a human may say, there is some place you can say, "But why can't we just X to stop that?" or "But AI could never do that!" Unhelpfully, history has no prior examples of human extinction. (Well, there were various hominid species, but isn't that different? They were not as intelligent as us!) Even if someone believes it is plausible, lots of people, including AI researchers and lab CEOs, say there is a very small chance of it happening. If probabilities are not your native language, it is easy to round a few percentage points down to zero.
Conclusion
Extinction is invoked by AI safety folks as the "infinitely bad outcome", but, even though most people may say they agree to not look like a mundicider, I'm unsure most people feel this way in a way that changes their behavior. This can lead to a lot of talking past each other.
I'm not sure if it's a good or bad thing to not feel the badness of extinction "at a gut level". I hear about people who care about x-risk getting really really stressed out and mentally ill. It seems like it might be good to not be super stressed out and afraid? Keep the grim-o-meter firmly detached?
Curious to hear what others think and how they've related to the idea of extinction emotionally, and also how you've seen other people react to it.
And Happy Petrov Day. May death be destroyed.