During the Manhattan Project, Arthur Compton reportedly decided the bomb should only go ahead if the chance of igniting the atmosphere was under about three in a million. Hans Bethe later disputed that there was ever such a threshold: ignition was “not a matter of probabilities; it is simply impossible.” So the two positions on record are “a few in a million” and “zero.”
When physicists worried that the RHIC collider might produce something that would set off a chain reaction and “eat” the Earth, the reassuring bound was a catastrophe probability below 2×10⁻⁸. Adrian Kent pointed out that this is not actually small: multiplied by the world’s population it is still an expected 120 deaths, and holding it to the same standard as ordinary radiation policy would require something like 10⁻¹⁵ per year.
In Iain Banks’ fictional Culture universe, the superintelligent Minds evaluate risk at similar scales, and if/when a risk is taken at even these odds it’s considered insane:
[…] Displacing was inherently and unfinessably dangerous; the risk of something going horribly, terminally wrong was only about one in eighty million for any single Displacement event, but that was still enough to put the average, fussily perfectionist ship Mind off using the process for anything alive except in the direst of emergencies, and [it...] must have carried out thirty-thousand-plus Displacements in a minute or less, nudging the odds up well into the sort of likelihood-of-fuckup range any sane Mind would normally recoil from in utter horror. [1]
I think when we look back, we will look back at “1-99%” or “10-25%” chance of extinction from superintelligence in abject horror.
Banks, Iain M. Excession. Bantam Books, 1998. p. 268 ↩︎
Sorry I'm not sure I understand what you mean. What I'm interpreting this as is that also having some kind of potentially traumatic experience with mortality, such as losing a family member to a disease, may also lead to similar downplaying of the costs. Am I understanding correctly?
If so I definitely agree and can see an argument as to why you'd think it's worth it to potentially save more people if you develop the technology. I think there's many possible responses to death and in an ideal world it would be a choice for everyone rather than something for... (read more)