There is a huge psychological difference between "discussing far-off things in the abstract" and "oh shit we're actually doing this now."
I, too, once thought that the Singularity would be sort of fun. You know, back when I was reading Nanosystems (1992) and "Lobsters" (2001) and admiring Manfred Macx. Even then, I knew that the Singularity wasn't actually likely to be a good thing. But a few years layer I figured there was no plausible way it would happen sooner than, oh, 2025–2045. [1] And it was still pretty abstract and science fictional.
Well, 2025 rolled around, and by that time, I had firmly passed from "far off future" to "this is thing that might actually happen soon, to me and my family, on the same kind of time horizons that I plan for my kids' college or my own retirement."
And then I start to draw the nasty conclusions:
- We still have no idea how to align something smarter than us.
- If we could align a superintelligence, then some specific human beings would control that superintelligence. Probably either politicians or billionaire CEOs, as the fight over Fable suggests. Whether a potential SkyNet obeys the Epstein class or merely its own impulses is not that comforting a question.
- Claude Fable 5 already has more raw intelligence than most human beings. It's just poor at many practical applications of intelligence that don't fit in an hour or two. Once that's fixed, we're going straight to ASI, with barely enough time at "AGI" to blink, let alone hold a public discussion. [2]
And so I've viscerally felt that shift from "far off future" to "oh shit this is happening." Those people you hear saying things like "Maybe human extinction wouldn't be that bad?" When they realize that their actual kids might die in the near future, you'll hear an entirely different story from most of them.
That was roughly when I expected 10^15 to 10^18 FLOPS, the lowest possible bounds on the algorithms equivalent to human cognition, to become pretty widely available to organizations. An RTX Pro 6000 Blackwell is about 10^15 marketing FLOPS at
fp8fp4. So not bad, as guesstimates ~20 years in advance go. ↩︎EDIT: I argue for these points in more detail elsewhere (see my profile). Or see If Anyone Builds It, Everyone Dies, which makes a slightly different argument along similar lines. My goal here is to convey the psychological state that hit me when I thought "oh shit this is real", not to specifically argue for these points. ↩︎
1Many of them don't have kids. But yeah, otherwise I agree.
A counterpoint though is that many people just won't allow themselves to make that realization until it's too late.
Playing devil's advocate, I think there's a very substantial portion of the public that genuinely is sufficiently unhappy with the state of the world that dying in general and the end of the world specifically don't feel like that big of a deal to them. The set of people who see their future as relatively happy is increasingly slim.
As anecdotal support, my friends have noted, even in areas where the demographics and economy are unchanged, a lot riskier driving and other behavior from people on the road, spanning pedestrians, cyclists, motorcyclists, and drivers, and jokes about suicide among the youth seem way higher than they've historically been, although the latter have been tamped down somewhat by platforms aggressively trying to curb them.
More likely I think, AI catastrophe looks like being with your family and friends as usual, and over an extended period of time watching one another succumb to things like starvation due to having no money
I think most takeover / existential risk advocates say explicitly that joblessness isn't the main concern.
If you're talking about non-takeover scenarios, I'm skeptical of this. There are plenty of people in the U.S. who have no real economic contribution even now, and they certainly aren't at risk of starving.
The set of people who see their future as relatively happy is increasingly slim
I'd like to test your intuition. Can you write down your predicted answers to these questions, then check the answers and let me know how you did?
- From the years 2001-2025, Gallup polled Americans on the question "In general, are you satisfied or dissatisfied with the way things are going in your personal life at this time?" In 2025, what percentage responded "satisfied"?
- Gallup also asked, "Are you very [satisfied/dissatisfied], or just somewhat [satisfied/dissatisfied]?" In 2025, what percent of all respondents were "very satisfied", and what percent "very dissatisfied"?
- On average (mean) across the 23 polls conducted from 2001-2025, what percent responded with each option?
- In 2025, Pew conducted a poll primarily about American's attitudes towards their country. When asked "When you think about the future, do you feel hopeful?" on this poll, what percent said "yes"? When asked "When [...] feel happy?", what percent said "yes"?
- In the 2024 National Survey on Drug Use and Health conducted by the US government, what percent of adults reported serious thoughts of suicide in the past 12 months? What percent overall attempted suicide?
- What percent had thoughts of suicide or attempted suicide in 2015?
Answers and Commentary
Answers:
1. 81% satisfied, 17% dissatisfied. 2. 44% very satisfied, 8% very dissatisfied. 3. 83% satisfied (54/30% very/somewhat), 15% dissatisfied (6/9% very/somewhat). 4. 68% hopeful, 54% happy. 5. 5.5% ideation, 0.8% attempt. 6. 4% ideation, 0.6% attempt.
Commentary:
I predict that you will respond ≤70% satisfied to Q1, ≥15% very dissatisfied to Q2, a "satisfied" mean ≥5% greater than your Q1 response to Q3, hopeful ≤60% to Q4, and ≥8% ideation or ≥1% attempt to question 5. No strong opinions on the others (I believe that question 4 has unusually low numbers due to priming respondents to think about politics). While "very substantial" and "increasingly slim" are relative statements, I believe your phrasing reflects an overestimation of the rate of severe unhappiness in the US, so I predict your intuition would be off.
Sources: Gallup, Pew, NSDUH 2024 and 2015.
- From the years 2001-2025, Gallup polled Americans on the question "In general, are you satisfied or dissatisfied with the way things are going in your personal life at this time?" In 2025, what percentage responded "satisfied"?
My guess would be 60-80 percent, leaving 20-40 percent (openly) in various states of despair. I'd give a similar answer to (2), with maybe 30 saying very satisfied and ~10 saying very unsatisfied. Important to note that my position is not that a majority of people would be fine seeing the world end, only that a substantial portion of people would be.
I would qualify this with an assertion that the positive results likely skew older, and the negative results skew younger. This is relevant in this context because the open-to-the-idea-of-existential-risk-from-AI crowd skews young. I'd also add that I'm predicting responses, and that the true values likely skew a bit more negative due to respondent bias - people don't like admitting they're sad.
Answers: 1. 81% satisfied, 17% dissatisfied. 2. 44% very satisfied, 8% very dissatisfied.
(1 Falls towards the better end of my predicted range. Qualifications above still important, though. 2 is pretty close, 30->44 and 10->8.)
- In 2025, Pew conducted a poll primarily about American's attitudes towards their country. When asked "When you think about the future, do you feel hopeful?" on this poll, what percent said "yes"? When asked "When [...] feel happy?", what percent said "yes"?
On 'hopeful', I'd expect that the right direction/wrong track polls, which generally track with party affiliation and then move downwards towards the middle of a presidential term, unless I interpret the question differently than the respondents did.
Answer: 4. 68% hopeful
(Not much to say here. I think they read it as "any hope at all", rather than "more optimistic than pessimistic".)
5. In the 2024 National Survey on Drug Use and Health conducted by the US government, what percent of adults reported serious thoughts of suicide in the past 12 months? What percent overall attempted suicide?
6. What percent had thoughts of suicide or attempted suicide in 2015?
On ideation, I'd say somewhere in the middle of the single digit range, but reporting bias confounds this.
On attempt, I'd argue that reporting bias is severe enough that I can't make a good prediction. Most suicidal people aren't going to tell someone and get shuffled off to a psych ward, and it's been established in the public eye that that's what will happen. The 'false positives' (people joking about it nonchalantly) have been cut down by this, but we're also seeing way more false negatives.
I'll note here that these are likely floors rather than ceilings on people who'd be fine with the world ending. A suicide leaves behind responsibilities to the world, whereas 'all going together when we go' doesn't. I've seen this mindset quite a bit, more often nowadays.
Answers: 5. 5.5% ideation, 0.8% attempt. 6. 4% ideation, 0.6% attempt.
(5.5/4 is essentially the dead center of the single digit range. I stand by making no prediction on attempt rates, we don't have a remotely reliable source of data there.)
Commentary:
I predict that you will respond ≤70% satisfied to Q1, (Middle of my range, +1) ≥15% very dissatisfied to Q2 (That's a miss.), a "satisfied" mean ≥5% greater than your Q1 response to Q3 (Rolled that into the first two because I expected them to be the same thing; I think you miss on that one.), hopeful ≤60% to Q4 (Not much to say on this; I guessed at the question's semantics.), and ≥8% ideation (I'd say semi-miss; 8% is near the middle of the single digit range, but the true value is closer to the middle.) or ≥1% attempt (No prediction made, but after seeing the recorded result, I would say that the true value probably is above 1 percent, assuming that half of attempts are unreported, and you count things like a guy driving to a quiet place with a gun and then not going through with it as an attempt. Firearm suicides are all or nothing, and consist of around half of total suicides.) to question 5. No strong opinions on the others (I believe that question 4 has unusually low numbers due to priming respondents to think about politics). While "very substantial" and "increasingly slim" are relative statements, I believe your phrasing reflects an overestimation of the rate of severe unhappiness in the US (I think this is where you were off. I'm not saying most people would accept the end of the world, but I think the numbers provided present a strong case that this demographic exists, and it's almost definitely at its largest among the young, relatively alienated, and relatively anxious tech community), so I predict your intuition would be off.
I am not sure about other people.
My formative idea of AI takeover came from Biting the Sun by Tanith Lee, when it was still sci-fi stuff. I see it as everyone becoming drug addicted hedonists and losing autonomy and losing everything it means to be human, with only a few people caring and being motivated to do more. Then I got my current job and I see it happening. I work in disability services and see some people want others to do everything for them, these people lose the ability to do things themselves. Others want to learn independence, but only a few really keep it up over time. Most drift more and more into the first camp as it is available.
My pdoom is very high as a result.
1A lot of people take solace in the "everyone dies" narrative. It's painfully clear that democratic control of AI is just not going to happen. Implicitly, that leaves the choice between humanity continuing to exist as somewhere between a few million people to one singular person and humanity not continuing to exist at all. I don't think it's all that unreasonable to want 100% of people rather than 99.99...9% to die when the 0.000...1% is responsible for the rest of the death.
In fact, this may(at least on the subconscious level) be a kind of deterrence - a "you're going down with me" arrangement. I do assign substantial probability to things actually playing out this way - an actor(maybe a country, maybe a single person) faced with certain death chooses to destroy the(now-barely-populated) world rather than let their killers go free.
Thanks for writing this. I think this is a very important dimension.
I sometimes hear smart, young, philosophically-inclined people saying "maybe AIs just should replace us, because [some reason like they-are-better/natural-order/this-is-how-the-world-is/whatever-else]". And I think it's fine to entertain these thoughts, i.e. I like this not being a taboo. But the good answer is - "Are you happy with seeing the people you love slowly and painfully die, and die yourself too, because [the same reason]?".
I’m personally skeptical that this is really the main thing going on, since it seems unlikely that many people are really mostly concerned for their own non-death out of selfless regard for the feelings of others. I’m also skeptical that this would be their view on a bunch more consideration.
Since I was a child I've always been convinced that life is more negatives than positives. What has always stopped me from offing myself was my unwillingness to hurt my parents and the people I love. I was afraid of death and of committing the act too of course. I'd say that at least a person out of ten that I talked to about this stuff in real life feels similarly.
An instantaneous extinction as described, for instance, in AI 2027 would be a perfect solution to my fears. I suspect that most of those who "don't really care if everyone dies" do indeed believe that.
I still see life as a net negative, but now I wish to live until we develop AGI/ASI and I hope that happens soon. That's likely to drastically change the way I experience life. I'd prefer more utopical outcomes of course, but an instantaneous extinction is definitely better than dying in several decades after a long kind of life like the one I've experienced so far. A quick-ish but not immediate extinction would be preferable too, as long as it doesn't come with too much extra sufferance.
More likely I think, AI catastrophe looks like being with your family and friends as usual, and over an extended period of time watching one another succumb to things like starvation due to having no money, illness due to having no money, or literally overheating or being crushed in the midst of a breathtakingly fast technological buildout that doesn’t need any of the specific conditions we depend on to live, doesn’t care about us, and does need to do vasts amount of computation quickly. Meanwhile, there won’t be a moment when you know this has happened—there will be increasing confusion and misinformation and decreasing resources put into making it legible to you. It will probably look more like the atrocities we are used to than the perfect, clean instantaneous end to the human story.
Why do you think this scenario is more likely?
A slow extinction like you describe it is likely to lead to a lot of unnecessary destruction/entropy: wars, riots, people bombing data centers and burning natural resources.
I don't have the presumption to understand what an hypothetical ASI might decide: your scenario is of course possible. But I suspect that for an ASI with technology indistinguishable from magic it'd be cheaper (in terms of resources and entropy) to kill everyone quickly before we have time to react, rather than letting us go wild for months or years.
A notable fraction of people respond to hearing about existential risk from AI by saying they don’t really care if everyone dies. I think the idea is often along the lines of ‘well if we are all dead, then there’s nobody to be unhappy about it’.
I’m personally skeptical that this is really the main thing going on, since it seems unlikely that many people are really mostly concerned for their own non-death out of selfless regard for the feelings of others. I’m also skeptical that this would be their view on a bunch more consideration.
So to help with the consideration—
My guess is that an important thing going on here is that the ‘everyone dying at once’ image seems kind of like a thought experiment—abstract, hypothetical, neat, not very sinister. Also, you literally can never see it, so it feels pretty surreal.
But it is interesting that we even have this assumption that everyone will die together.
It’s true that in some prominent AI catastrophe stories, a single AI system suddenly emerges fantastically more powerful than anyone else and builds technology to quickly kill everyone, perhaps before they notice.
But this doesn’t seem like the bulk of probability.
More likely I think, AI catastrophe looks like being with your family and friends as usual, and over an extended period of time watching one another succumb to things like starvation due to having no money, illness due to having no money, or literally overheating or being crushed in the midst of a breathtakingly fast technological buildout that doesn’t need any of the specific conditions we depend on to live, doesn’t care about us, and does need to do vasts amount of computation quickly. Meanwhile, there won’t be a moment when you know this has happened—there will be increasing confusion and misinformation and decreasing resources put into making it legible to you. It will probably look more like the atrocities we are used to than the perfect, clean instantaneous end to the human story.