[ Question ]

Forecasting Thread: Existential Risk

by Amandango2 min read22nd Sep 202040 comments


Forecasts (Specific Predictions)Existential RiskWorld Optimization

This is a thread for displaying your probabilities of an existential catastrophe that causes extinction or the destruction of humanity’s long-term potential.

Every answer to this post should be a forecast showing your probability of an existential catastrophe happening at any given time.

For example, here is Michael Aird’s timeline:

The goal of this thread is to create a set of comparable, standardized x-risk predictions, and to facilitate discussion on the reasoning and assumptions behind those predictions. The thread isn’t about setting predictions in stone – you can come back and update at any point!


How to participate

  1. Go to this page
  2. Create your distribution
    • Specify an interval using the Min and Max bin, and put the probability you assign to that interval in the probability bin.
    • You can specify a cumulative probability by leaving the Min box blank and entering the cumulative value in the Max box.
    • To put probability on never, assign probability above January 1, 2120 using the edit button to the right of the graph. Specify your probability for never in the notes, to distinguish this from putting probability on existential catastrophe occurring after 2120.
  3. Click 'Save snapshot' to save your distribution to a static URL
    • A timestamp will appear below the 'Save snapshot' button. This links to the URL of your snapshot.
    • Make sure to copy it before refreshing the page, otherwise it will disappear.
  4. Click ‘Log in’ to automatically show your snapshot on the Elicit question page
    • You don’t have to log in, but if you do, Elicit will:
      • Store your snapshot in your account history so you can easily access it.
      • Automatically add your most recent snapshot to the x-risk question page under ‘Show more’. Other users will be able to import your most recent snapshot from the dropdown, shown below.
    • We’ll set a default name that your snapshot will be shown under – if you want to change it, you can do so on your profile page.
    • If you’re logged in, your snapshots for this question will be publicly viewable.
  5. Copy the snapshot timestamp link and paste it into your LessWrong comment
    • You can also add a screenshot of your distribution in your comment using the instructions below.

Here's an example of how to make your distribution:


How to add an image to your comment

  1. Take a screenshot of your distribution
  2. Then do one of two things:
    1. If you have beta-features turned on in your account settings, drag-and-drop the image into your comment
    2. If not, upload it to an image hosting service like imgur.com, then write the following markdown syntax for the image to appear, with the url appearing where it says ‘link’: ![](link)
  3. If it worked, you will see the image in the comment before hitting submit.


If you have any bugs or technical issues, reply to Ben from the LW team or Amanda (me) from the Ought team in the comment section, or email me at amanda@ought.org.


Questions to consider as you're making your prediction

  • What definitions are you using? It’s helpful to specify them.
  • What evidence is driving your prediction?
  • What are the main assumptions that other people might disagree with?
  • What evidence would cause you to update?
  • How is the probability mass allocated amongst x-risk scenarios?
  • Would you bet on these probabilities?


Comparisons and aggregations

Here's a comparison of the 8 predictions made so far (last updated 9/26/20).


Here's a distribution averaging all the predictions (last updated 9/26/20). The averaged distribution puts 19.3% probability before 2120 and 80.7% after 2120. The year within 2021-2120 with the greatest risk is 2040.

Here's a CDF of the averaged distribution:


New Answer
Ask Related Question
New Comment

6 Answers

I've made a distribution based on the Metaculus community distributions:

(I used this Colab notebook for generating the plots from Elicit distributions over specific risks. My Elicit snapshot is here). 

In 2019, Metaculus posted the results of a forecasting series on catastrophic risk (>95% of humans die) by 2100. The overall risk was 9.2% for the community forecast (with 7.3% for AI risk). To convert this to a forecast for existential risk (100% dead), I assumed 6% risk from AI, 1% from nuclear war, and 0.4% from biological risk. To get timelines, I used Metaculus forecasts for when the AI catastrophe occurs and for when great power war happens (as a rough proxy for nuclear war). I put my own uninformative distribution on biological risk.

This shouldn't be taken as the "Metaculus" forecast, as I've made various extrapolations. Moreover, Metaculus has a separate question about x-risk, where the current forecast is 2% by 2100. This seems to me hard to reconcile with the 7% chance of AI killing >95% of people by 2100, and so I've used the latter as my source. 

Technical note: I normalized the timeline pdfs based on the Metaculus binary probabilities in this table, and then treated them as independent sources of x-risk using the Colab. This inflates the overall x-risk slightly. However, this could be fixed by re-scaling the cdfs.

Big thanks to Amanda, Owain, and others at Ought for their work on this!

My overall forecast is pretty low confidence — particularly with respect to the time parameter.

Snapshot is here: https://elicit.ought.org/builder/uIF9O5fIp (Please ignore any other snapshots in my name, which were submitted in error)

My calculations are in this spreadsheet

For my prediction (which I forgot to save as a linkable snapshot before refreshing, oops) roughly what I did was take my distribution for AGI timing (which ended up quite close to the thread average), add an uncertain but probably short delay for a major x-risk factor (probably superintelligence) to appear as a result, weight it by the probability that it turns out badly instead of well (averaging to about 50% because of what seems like a wide range of opinions among reasonable well-informed people, but decreasing over time to represent an increasing chance that we'll know what we're doing), and assume that non-AI risks are pretty unlikely to be existential and don't affect the final picture very much. To an extent, AGI can stand in for highly advanced technology in general.

If I start with a prior where the 2030s and the 2090s are equally likely, it feels kind of wrong to say I have the 7-to-1 evidence for the former that I'd need for this distribution. On the other hand, if I made the same argument for the 2190s and the 2290s, I'd quickly end up with an unreasonable distribution. So I don't know.

Epistemic status: extremely uncertain

I created my Elicit forecast by:

  • Slightly adjusting down the 1/6 estimate of existential risk during the next century made in The Precipice
  • Making the shape of the distribution roughly give a little more weight to time periods when AGI is currently forecasted to be more likely to come

[I work for Ought.]

For my prediction, like those of others, I basically just went with my AGI timeline multiplied by 50% (representing my uncertainty about how dangerous AGI is; I feel like if I thought a lot more about it the number could go up to 90% or down to 10%) and then added a small background risk rate from everything else combined (nuclear war, bio stuff, etc.)

I didn't spend long on this so my distribution probably isn't exactly reflective of my views, but it's mostly right.

Note that I'm using a definition of existential catastrophe where the date it happens is the date it becomes too late to stop it happening, not the date when the last human dies.

For some reason I can't drag-and-drop images into here; when I do it just opens up a new window.

Elicit prediction: https://elicit.ought.org/builder/0n64Yv2BE


Epistemic Status: High degree of uncertainty, thanks to AI timeline prediction and unknowns such as unforeseen technologies and power of highly developed AI.

My Existential Risk (ER) probability mass is almost entirely formed from the risk of unfriendly Artificial Super Intelligence (ASI) and so is heavily influenced my predicted AI timelines. (I think AGI is most likely to occur around 2030 +-5 years, and will be followed within 0-4 years of ASI, with a singularity soon after that, see my AI timelines post: https://www.lesswrong.com/posts/hQysqfSEzciRazx8k/forecasting-thread-ai-timelines?commentId=zaWhEdteBG63nkQ3Z  ). 

I do not think any conventional threat such as nuclear war, super pandemic or climate change is likely to be an ER, and super volcanoes or asteroid impacts are very unlikely. I think this century is unique and will constitute 99% of the bulk of ER, with the last <1% being from more unusual threats such as simulation being turned off, false vacuum collapse, or hostile alien ASI. But also, for unforeseen or unimagined threats.

I think the most likely decade for the creation of ASI will be the 30’s, with an 8% ER chance (From not being able to solve the control problem or coordinate to implement it even if solved).

Considering AI timeline uncertainty as well as how long an ASI takes to acquire techniques or technologies necessary to wipe out or lock out humanity I think an 11% ER chance for the 40’s. Part of the reason this is higher than the 30’s ER estimate is to accommodate the possibility of a delayed treacherous turn. 

Once past the 50’s I think we will be out of most of the danger (only 6% for the rest of the century), and potential remaining ER’s such as runaway nanotech or biotech will not be a very large risk as ASI would be in firm control of civilisation by then.  Even then though some danger remains for the rest of the century from unforeseen black ball technologies, however interstellar civilisational spread (ASI high percent of speed of light probes) by early next century should have reduced nearly all threats to less than ERs. 

So overall I think the 21st Century will pose a 25.6% chance of ER. See the Elicit post for the individual decade breakdowns.

Note: I made this prediction before looking at the Effective Altruism Database of Existential Risk Estimates.