There is a huge psychological difference between "discussing far-off things in the abstract" and "oh shit we're actually doing this now."
I, too, once thought that the Singularity would be sort of fun. You know, back when I was reading Nanosystems (1992) and "Lobsters" (2001) and admiring Manfred Macx. Even then, I knew that the Singularity wasn't actually likely to be a good thing. But a few years layer I figured there was no plausible way it would happen sooner than, oh, 2025–2045. [1] And it was still pretty abstract and science fictional.
Well, 2025 rolled around, and by that time, I had firmly passed from "far off future" to "this is thing that might actually happen soon, to me and my family, on the same kind of time horizons that I plan for my kids' college or my own retirement."
And then I start to draw the nasty conclusions:
And so I've viscerally felt that shift from "far off future" to "oh shit this is happening." Those people you hear saying things like "Maybe human extinction wouldn't be that bad?" When they realize that their actual kids might die in the near future, you'll hear an entirely different story from most of them.
That was roughly when I expected 10^15 to 10^18 FLOPS, the lowest possible bounds on the algorithms equivalent to human cognition, to become pretty widely available to organizations. An RTX Pro 6000 Blackwell is about 10^15 marketing FLOPS at fp8 fp4. So not bad, as guesstimates ~20 years in advance go. ↩︎
A notable fraction of people respond to hearing about existential risk from AI by saying they don’t really care if everyone dies. I think the idea is often along the lines of ‘well if we are all dead, then there’s nobody to be unhappy about it’.
I’m personally skeptical that this is really the main thing going on, since it seems unlikely that many people are really mostly concerned for their own non-death out of selfless regard for the feelings of others. I’m also skeptical that this would be their view on a bunch more consideration.
So to help with the consideration—
My guess is that an important thing going on here is that the ‘everyone dying at once’ image seems kind of like a thought experiment—abstract, hypothetical, neat, not very sinister. Also, you literally can never see it, so it feels pretty surreal.
But it is interesting that we even have this assumption that everyone will die together.
It’s true that in some prominent AI catastrophe stories, a single AI system suddenly emerges fantastically more powerful than anyone else and builds technology to quickly kill everyone, perhaps before they notice.
But this doesn’t seem like the bulk of probability.
More likely I think, AI catastrophe looks like being with your family and friends as usual, and over an extended period of time watching one another succumb to things like starvation due to having no money, illness due to having no money, or literally overheating or being crushed in the midst of a breathtakingly fast technological buildout that doesn’t need any of the specific conditions we depend on to live, doesn’t care about us, and does need to do vasts amount of computation quickly. Meanwhile, there won’t be a moment when you know this has happened—there will be increasing confusion and misinformation and decreasing resources put into making it legible to you. It will probably look more like the atrocities we are used to than the perfect, clean instantaneous end to the human story.