I highly suspect that the lack of global response to AI safety concerns isn't because there's no codified mathematically valid processes giving us probabilities of extinction (roughly ~0% of people understand statistics anyway); but because of a lack of salience.
One big place you can see a similar issue occuring is Nuclear War risk; where despite the risk of extinction-inducing nuclear war having only increased (the number of nuclear states has increased dramatically, now including even tiny unimportant countries like Israel and North korea; while at the same time MIRVs have gotten very advanced), people no longer view it as the threat that they used to. Although with nukes this is probably in part because we haven't died yet (unfortunately, like the general public, exponential distributions are memoryless, so that doesn't help much), I think it's mostly just because they're not as much of a hot topic anymore.
Most AI researchers put at least 5% on advanced AI causing human extinction or extreme disempowerment. The most cited scientists of all time are Yoshua Bengio and Geoffrey Hinton and they both put extinction at 10% or more. There is a distinct lack of global response to this kind of message from relevant scientists.
One reason people purport to not be impressed by these chances of extinction is that they are subjective probabilities, not verified frequencies or arising from a more formal process.
My natural response to this concern is to argue about this, since I think it is very wrong and misguided.
But a different direction would be to just try to get admissible numbers on these things that some of our more normal processes of responding to risk can respond to.
It’s going to be hard to get straight up frequentist numbers due to extinction being not very repeatable.
But in other areas of life, we have systems that must deal with subjective probabilities and non-repeating circumstances, and take them seriously and quantitatively. The fact that a specific company can’t go bankcrupt a large enough number of times to get a good outside view on it happening in a given month doesn’t mean investors just write off that possibility as speculation. It gets a quantitative part in the price of that company. And the chance of it happening isn’t an output of a formal method—it’s a bunch of people’s guesses, from looking at a detailed situation. Somehow the subjective guesses about this and a multitude of other sources of future value and disvalue get processed into an extremely specific formal price number that official processes of the world take seriously. If the price of a company goes down because people think it might go bankrupt, a bunch of people will make different decisions as a result.
And in the end, very few uncertainties in the world can be turned into clear frequencies—even if you have a disease that a million other people get every year, you aren’t the perfectly average person with the perfectly average body and life and your prognosis and what you should do can’t be pulled out of a formal process. And this is a relatively good case for using pure frequentism.
So how else does human society turn subjective judgments into admissible forms of uncertainty?
~
Inspired by a talk by Paul Christiano.