Existential risk

An existential risk (or x-risk) is a risk posing permanent large negative consequences to humanity which can never be undone. In Nick Bostrom's seminal paper on the subject ¹, he defined an existential risk as:

One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.

The total negative impact of an existential risk is one of the greatest negative impact known. Such event could not only annihilate life as we value it from earth, but would also severely damage all Earth-originating intelligent life potential.

Bostrom ² proposes a series of classifications for existential risks:

• Bangs - Earthly intelligent life is extinguished relatively suddenly by any cause; the prototypical end of humanity. Examples of bangs include deliberate or accidental misuse of nanotechnology, nuclear holocaust, the end of our simulation, or an unfriendly AI.
• Crunches - The potential humanity had to enhance itself indefinitely is forever eliminated, although humanity continues. Possible crunches include an exhaustion of resources, social or governmental pressure ending technological development, and even future technological development proving an unsurpassable challenge before the creation of a superintelligence.
• Shrieks - Humanity enhances itself, but explores only a narrow portion of its desirable possibilities. As the criteria for desirability haven't been defined yet, this category is mainly undefined. However, a flawed friendly AI incorrectly interpreting our values, a superhuman upload deciding its own values and imposing them on the rest of humanity, and an intolerant government outlawing social progress would certainly qualify.
• Whimpers - Though humanity is enduring, only a fraction of our potential is ever achieved. Spread across the galaxy and expanding at near light-speed, we might find ourselves doomed by ours or another being's catastrophic physics experimentation, destroying reality at light-speed. A prolonged galactic war leading to our extinction or severe limitation would also be a whimper. More darkly, humanity might develop until its values were disjoint with ours today, making their civilization worthless by present values.

The total negative results of a existential risk could amount to the total of potential future lives not being realized. A rough and conservative calculation³ gives us a total of 10^54 potential future humans lives – smarter, happier and kinder then we are. Hence, almost no other task would amount to so much positive impact than existential risk reduction.

Existential risks also present an unique challenge because of their irreversible nature. We will never, by definition, experience and survive an extinction risk⁴ and so cannot learn from our mistakes. They are subject to strong observational selection effects ⁵. One cannot estimate their future probability based on the past, because bayesianly speaking, the conditional probability of a past existential catastrophe given our present existence is always 0, no matter how high the probability of an existential risk really is. Instead, indirect estimates have to be used, such as possible existential catastrophes happening elsewhere. A high extinction risk probability could be functioning as a Great Filter and explain why there is no evidence of spacial colonization.

Another related idea is that of a suffering risk (or s-risk), which can be considered a form of "shriek" as outlined above.

Blog posts

• Intelligence enhancement as existential risk mitigation by Roko
• Our society lacks good self-preservation mechanisms by Roko
• Disambiguating doom by steven0461
• Existential Risk by lukeprog

Organizations

• Machine Intelligence Research Institute
• The Future of Humanity Institute
• The Oxford Martin Programme on the Impacts of Future Technology
• Global Catastrophic Risk Institute
• Saving Humanity from Homo Sapiens
• Skoll Global Threats Fund (To Safeguard Humanity from Global Threats)
• Foresight Institute
• Defusing the Nuclear Threat
• Leverage Research
• The Lifeboat Foundation

Notes