This is a special post for quick takes by Hestia. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I understand why people talk about "human extinction" from AI, because it's a lot more attention-grabbing, but it's much easier for me to see a way in which AI kills a billion people or 5 billion people than every single last person down to the uncontacted tribes.
The crux is taking the idea of superintelligence seriously and looking at the implications. A lot of "techno-optimism" for example is about not taking this seriously, and thus not having a real problem to worry about. So literal extinction being attention-grabbing is not the reason to talk about it, same as it's not about marketing for AI companies. It's the model of what superintelligence actually is that is the reason.
In the long term (once there is actually a superintelligence, rather than the current harmless precursors to it), the scope of impact of AI is not bounded, what could affect a billion people soon affects the Earth as a whole (uncontacted tribes and all), and then the Sun, and the reachable universe.
That's the main issue with addressing AGI risk as an engineering problem: a catastrophic failure has the kind of blast radius that doesn't allow future iterative improvement. With any other technology, there will be opportunity to learn from mistakes and do better, again and again, until you succeed. But not here.
For what it's worth, I think most people saying human extinction really do anticipate exactly that outcome, and aren't just saying it for rhetorical purposes.
As an intuition pump, human created climate change has driven some species to extinction. An AI that doesn't care about preserving humanity might kill the final holdouts in analogous fashion.
It's very difficult for me to come up with believable scenarios which include disempowerment and death of a majority of humans, but don't eventually kill everyone. The uncontacted tribes (from what I understand, not truly unknown, just prefer minimal contact) may last a few hundred years in these scenarios, but biological civilization doesn't come back.
What about an AI which cares a tiny bit about keeping humans alive, but still kills lots of people in the process of taking over? It could care about this terminally or could be paid off via acausal trade.
What about an AI which cares a tiny bit about keeping humans alive
This is quite believable in the short (a few generations of humans) term. It's much less so in the long term. I model "caring about a lot of things" as an equilibrium case of competing reasons for action, and almost nothing that is "care a little bit" is going to be stable.
I suppose "care a lot about maintaining a small quantity of biologicals" is possible as well for inscrutable AI reasons, but that doesn't bring any civilization back.