This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I think there might be a bias on here to assume that if humanity goes extinct as the result of something an AI does, it will specifically be something the AI did on purpose, with the intent of causing humanity's extinction or the unrecoverable collapse of industrial civilization, or at least with the knowledge that that was a likely outcome. But it seems to me that one obvious way we "accidentally" end up in such a situation is
AI systems become capable of training successor systems which are more capable of doing particular dangerous actions than they are, and better at training later successors than they are
The successors are not enough better at risk assessment than their ancestors to outweigh the increase in ability to accidentally cause harm
Iterate
I notice that, if you look at the history of near misses with civilization-ending events, they were not ones where someone tried and failed to end civilization. People have tried and failed, but they're not thr ones that got close.
Anyway, on that note, happy day-of-Solstice everyone :)
I agree. For what it's worth, in most of these cases, it seems to me that for catastrophe to occur, some AI eventually has to take over on purpose. But I agree that a lot of the action (and many of the points for intervention) might have occurred earlier with AIs that didn't intentionally take over and instead were not as helpful as they could be, either because of misalignment or other problems.
Humans might go extinct very soon after AI because AI accelerates technological progress and there was an extinction technology ahead of us. That is, we would’ve gone extinct from the same technology without the AI in 2500, but 2500 is “reached” a decade after AI, so it looks like the AI was the cause.
(This point is from Paul Christiano iirc.)
Note that AI needn’t take over (voluntarily or forcefully) to accelerate technological progress like this.
That's "taking an action with the knowledge that human extinction is a likely outcome". Unless you're counting multi-agent competitive scenarios where there was no single action that an AI took where the AI considered that that particular specific action could plausibly to lead to extinction or unrecoverable collapse, and decided to do the action anyway.
If an AI considered that their actions could with montrivial probability cause human extinction and decided to proceed with those actions anyway, and humans go extinct due to the actions of AI, I would consider that to be the AIs "intentionally" causing human extinction.
"AIs collectively buy all the land and humans have no food" is probably not a case of any particular AI intentionally causing human extinction. And as such preventing scheming won't save us from that threat model.