This is a special post for quick takes by Algon. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In an Open Letter to Scott Alexander, Stephen Pinker claims RAND [1] found there is "no describable scenario in which an AI could conclusively pose an extinction threat to humanity."
This is at best sloppy research, and at worst actively misleading, as Cody Fenwick points out that Rand's conclusions significantly differed from what Pinker claimed.
RAND actually said AI plausibly could wipe out humanity in 2 of 3 scenarios they looked at, though they couldn't definitively assert that any of the three scenarios is likely or unlikely.
I can see where Pinker got his claim--Rand did not show there is obviously a describable scenario in which an AI wipes out humanity. But:
1) Rand didn't search very hard
2) RAND certainly doesn't find it implausible that AI wipes out humanity, which is what Pinker is trying to argue against.
Overall, I continue to be disappointed by Pinker's arguments.
It has the smell of methodological rigour at the top - but at crucial points of concentration for the argument they make weird choices.
Consider bio risk - which is the one they couldn’t rule out. The reason they didn’t rule it in is primarily because they couldn’t find a plausible AI-initiated deployment mechanism with the current level of technology, yet they put this caveat in the deployment section:
“Without the means to interact in the physical world in a general way (e.g., using robotics or manipulating expert humans to perform certain tasks), it is ... (read more)
I think it's weasly to characterise it that way, as the RAND report also goes out of its way to say that they don't conclude extinction risk is unlikely. Plus, as you say, it's not a good paper.
2Algon
AFAICT, they don't even talk about plain old robots killing everyone. That's a pretty obvious way to exterminate an enemy, even if it does not look respectable.
For another, they seem to limit themselves to scenarios that only use existing tech in existing ways. Like, why assume the AI can't ramp up nuclear stockpiles massively? Or, more pertinently, dismiss nanotech, not on the grounds of it taking too long to create, but technical feasibility? AFAICT drexlerian nanotech is pretty feasible, let alone programmable biology.