During the Manhattan Project, Arthur Compton reportedly decided the bomb should only go ahead if the chance of igniting the atmosphere was under about three in a million. Hans Bethe later disputed that there was ever such a threshold: ignition was “not a matter of probabilities; it is simply impossible.” So the two positions on record are “a few in a million” and “zero.”
When physicists worried that the RHIC collider might produce something that would set off a chain reaction and “eat” the Earth, the reassuring bound was a catastrophe probability below 2×10⁻⁸. Adrian Kent pointed out that this is not actually small: multiplied by the world’s population it is still an expected 120 deaths, and holding it to the same standard as ordinary radiation policy would require something like 10⁻¹⁵ per year.
In Iain Banks’ fictional Culture universe, the superintelligent Minds evaluate risk at similar scales, and if/when a risk is taken at even these odds it’s considered insane:
[…] Displacing was inherently and unfinessably dangerous; the risk of something going horribly, terminally wrong was only about one in eighty million for any single Displacement event, but that was still enough to put the average, fussily perfectionist ship Mind off using the process for anything alive except in the direst of emergencies, and [it...] must have carried out thirty-thousand-plus Displacements in a minute or less, nudging the odds up well into the sort of likelihood-of-fuckup range any sane Mind would normally recoil from in utter horror. [1]
I think when we look back, we will look back at “1-99%” or “10-25%” chance of extinction from superintelligence in abject horror.
Banks, Iain M. Excession. Bantam Books, 1998. p. 268 ↩︎
You don't need the benefit of hindsight to look at “1-99%” or “10-25%” chance of extinction from superintelligence in abject horror.
If we look back.
I often think that, if you believe x-risk is non-negligible, in order to continue doing capabilities work you have to have so thoroughly abstracted away the concept of death/suffering that it’s functionally meaningless to you.
My outlook on life was drastically altered by suddenly losing a few friends in my 20’s. Prior to that death was an abstract concept which I largely read about or had seen coming in the case of elderly people. After having seen surprise catastrophes take place it is now something quite different to me and I only view this field in horror most of the time with the thought of how callously and flippantly I see people discuss the possibility of everyone dying or the hope that only a few million people die or some other framing.
I can’t necessarily blame them since — without happenstance — I’d likely be talking in the same way and having similar ideas but I think something has gone very wrong when you have a group building such a dangerous technology for whom catastrophe is an abstract concept as to be discussed casually at a party before they go back to doing the same thing the next day. Often my initial vibe check on whether or not I can even have a meaningful conversation with someone about these ideas is if they believe all manners of death are equivalent in the harm they cause (I know that is contrary to the beliefs of many here) but it is typically a sign to me of whether someone has had to actually grapple with mortality beyond an abstraction.
In my mind much of the rationalization of the risks here is causing people to clearly underweight the downside of this endeavor.
Sorry I'm not sure I understand what you mean. What I'm interpreting this as is that also having some kind of potentially traumatic experience with mortality, such as losing a family member to a disease, may also lead to similar downplaying of the costs. Am I understanding correctly?
If so I definitely agree and can see an argument as to why you'd think it's worth it to potentially save more people if you develop the technology. I think there's many possible responses to death and in an ideal world it would be a choice for everyone rather than something forced on you. My initial comment is likely too overgeneralizing based on the typical conversation I end up having with people about this, and the response that generally bothers me the most are those which over abstract the costs of catastrophe. I think we are in general substantially worse at actually internalizing these risks into our daily decisions than we think we are and the rationalist practice is likely good on net at improving that although it seems like it can be a mixed bag for different individuals on where it takes them. In general a lack of intention bothers me much more than consciously coming to a different decision.
Do you have a good estimate of what is and what will be possible with massively scaled up inference-time compute over the next 3 months? 6 months? Are you thinking about how this will effect others' priorities? Resource allocation? Governance and policy?
IMO having good answers to these questions feels super important for prioritizing where you spend your time.