Sometimes the following interaction plays out in public forums:
Lab Employee: I think there is a 20% chance of AI killing us all.
Cope Merchant (Quote Tweet): AI safety people are fearmongering, by pretending they have a hard scientific model, as part of their plan to [unknown]
Rationalist (Quote Tweet): But you see, a probability is just a report of their own epistemic state...
I sympathize with (some of) the words that the cope merchant is saying here. While in our subculture (and possibly Silicon Valley) the report "I think there is a 20% chance that AI kills everyone" means something cousined to "it seems unlikely but far from impossible", that is not all it's conveying to a regular person. To most normies, an ML researcher at a lab sounds like some sort of expert. And an "expert" that states a probability about something within their domain, is not just communicating an opinion on an event's likelihood. There's an implication that the expert arrived at it by some well-understood procedure, because most of the time the probabilities a normie reads are derived in this way (i.e. in rain forecasts, surgery odds, polls.)
This implication affects how most listeners update on the result, and probably does for you in regular life too. For example, if your financial advisor told you that your portfolio had a 0.05% chance of going bust, and then later on claimed that they had just "guessed" and ran no actual calculations, you would probably feel rather angry. Even if they explained that it was truly their subjective probability at the time, the misleading context might have made your subjective probabilities a lot different.
I don't know any great ideas for solving this particular public communications problem in the short term. It's hard to convey the alarming thing that these researchers want to convey, at the precision they want to convey it. Some things you could do if you were inclined:
Clarifythe whole situation. "There are no good models for this, and researchers disagree on background assumptions. But if I were forced to give a number on the probability that AI kills everyone, it would be 20%." This works but does not usually survive being quoted out of context.
Use weasel words in place of numbers. "There is a small but serious chance that AI kills everyone." This works if your need for precision is low, but you lose the actual estimate.
Quote survey statistics instead. “The median ML researcher in the last big survey said X%, with error bars of Y.” That's not your personal answer, but it is factually correct.
Basically refuse to answer. "I don't like giving hard probability estimates because it doesn't accurately convey how much knightian uncertainty there is about the problem. But it's definitely way, way too high, enough to be our most important policy priority." This works, but you hardly conveyed anything at all.
In the long term we are all dead, so I guess these are better than nothing.
Sometimes the following interaction plays out in public forums:
I sympathize with (some of) the words that the cope merchant is saying here. While in our subculture (and possibly Silicon Valley) the report "I think there is a 20% chance that AI kills everyone" means something cousined to "it seems unlikely but far from impossible", that is not all it's conveying to a regular person. To most normies, an ML researcher at a lab sounds like some sort of expert. And an "expert" that states a probability about something within their domain, is not just communicating an opinion on an event's likelihood. There's an implication that the expert arrived at it by some well-understood procedure, because most of the time the probabilities a normie reads are derived in this way (i.e. in rain forecasts, surgery odds, polls.)
This implication affects how most listeners update on the result, and probably does for you in regular life too. For example, if your financial advisor told you that your portfolio had a 0.05% chance of going bust, and then later on claimed that they had just "guessed" and ran no actual calculations, you would probably feel rather angry. Even if they explained that it was truly their subjective probability at the time, the misleading context might have made your subjective probabilities a lot different.
I don't know any great ideas for solving this particular public communications problem in the short term. It's hard to convey the alarming thing that these researchers want to convey, at the precision they want to convey it. Some things you could do if you were inclined:
In the long term we are all dead, so I guess these are better than nothing.