Finally, I’ve updated some based on my experience with Samotsvety forecasters when discussing AI risk… When we discussed the report on power-seeking AI, I expected tons of skepticism but in fact almost all forecasters seemed to give >=5% to disempowerment by power-seeking AI by 2070, with many giving >=10%.
It looks like Samotsvety also forecasted AI timelines and AI takeover risk - are you willing and able to provide those numbers as well?
We separately received a request from the FTX Foundation to forecast on 3 questions about AGI timelines and risk.
I sent out surveys to get Samotsvety’s up-to-date views on all 5 of these questions, and thought it would be valuable to share the forecasts publicly.
A few of the headline aggregate forecasts are:
25% chance of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe
81% chance of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe
32% chance of AGI being developed in the next 20 years
Forecasts
In each case I aggregated forecasts by removing the single most extreme forecast on each end, then taking the geometric mean of odds.
To reduce concerns of in-group bias to some extent, I calculated a separate aggregate for those who weren’t highly-engaged EAs (HEAs) before joining Samotsvety. In most cases, these forecasters hadn’t engaged with EA much at all; in one case the forecaster was aligned but not involved with the community. Several have gotten more involved with EA since joining Samotsvety.
Unfortunately I’m unable to provide forecast rationales in this post due to forecaster time constraints, though I might in a future post. I provided my personal reasoning for relatively similar forecasts (35% AI takeover by 2100, 80% TAI by 2100) in my WWOTF review.
WWOTF questions
Aggregate (n=11)
Aggregate, non-pre-Samotsvety-HEAs (n=5)
Range
What's your probability of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe?
25%
14%
3-91.5%
What's your probability of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe?
81%
86%
45-99.5%
FTX Foundation questions
For the purposes of these questions, FTX Foundation defined AGI as roughly “AI systems that power a comparably profound transformation (in economic terms or otherwise) as would be achieved in [a world where cheap AI systems are fully substitutable for human labor]”. See here for the full definition used.
Unlike the above questions, these are not conditioning on no pre-AGI/TAI catastrophe.
Aggregate (n=11)
Aggregate, non-pre-Samotsvety-HEAs (n=5)
Range
What's the probability of existential catastrophe from AI, conditional on AGI being developed by 2070?[1]
38%
23%
4-98%
What's the probability of AGI being developed in the next 20 years?
32%
26%
10-70%
What's the probability of AGI being developed by 2100?
73%
77%
45-80%
Who is Samotsvety Forecasting?
Edited to add: Our track record is now online here.
Samotsvety Forecasting is a forecasting group that was started primarily by Misha Yagudin, Nuño Sempere, and myself predicting as a team on INFER (then Foretell). Over time, we invited more forecasters who had very strong track records of accuracy and sensible comments, mostly on Good Judgment Open but also a few from INFER and Metaculus. Some strong forecasters were added through social connections, which means the group is a bit more EA-skewed than it would be without these additions. A few Samotsvety forecasters are also superforecasters.
How much do these forecasters know about AI?
Most forecasters have at least read Joe Carlsmith’s report on AI x-risk, Is Power-Seeking AI an Existential Risk?. Those who are short on time may have just skimmed the report and/or watched the presentation. We discussed the report section by section over the course of a few weekly meetings.
~5 forecasters also have some level of AI expertise, e.g. I did some adversarial robustness research during my last year of undergrad then worked at Ought applying AI to improve open-ended reasoning.
How much weight should we give to these aggregates?
My personal tier list for how much weight I give to AI x-risk forecasts to the extent I defer:
Individual forecasts from people who seem to generally have great judgment, and have spent a ton of time thinking about AI x-risk forecasting e.g. Cotra, Carlsmith
Samotsvety aggregates presented here
A superforecaster aggregate (I’m biased re: quality of Samotsvety vs. superforecasters, but I’m pretty confident based on personal experience)
Individual forecasts from AI domain experts who seem to generally have great judgment, but haven’t spent a ton of time thinking about AI x-risk forecasting (this is the one I’m most uncertain about, could see anywhere from 2-4)
Everything else I can think of I would give little weight to.[2][3]
Acknowledgments
Thanks to Tolga Bilge, Juan Cambeiro, Molly Hickman, Greg Justice, Jared Leibowich, Alex Lyzhov, Jonathan Mann, Nuño Sempere, Pablo Stafforini, and Misha Yagudin for making forecasts.
Unlike the WWOTF question, this includes any existential catastrophe caused by AI and not just misaligned takeovers (this is a non-negligible consideration for me personally and I’m guessing several other forecasters, though I do give most weight to misaligned takeovers).
Why do I give little weight to Metaculus’s views on AI? Primarily because of the incentives to make very shallow forecasts on a ton of questions (e.g. probably <20% of Metaculus AI forecasters have done the equivalent work of reading the Carlsmith report), and secondarily that forecasts aren’t aggregated from a select group of high performers but instead from anyone who wants to make an account and predict on that question.
Why do I give little weight to AI expert surveys such as When Will AI Exceed Human Performance? Evidence from AI Experts? I think most AI experts have incoherent and poor views on this because they don’t think of it as their job to spend time thinking and forecasting about what will happen with very powerful AI, and many don’t have great judgment.
Crossposted to EA Forum and Foxy Scout
Introduction
In my review of What We Owe The Future (WWOTF), I wrote:
In the comments, Peter Wildeford asked:
We separately received a request from the FTX Foundation to forecast on 3 questions about AGI timelines and risk.
I sent out surveys to get Samotsvety’s up-to-date views on all 5 of these questions, and thought it would be valuable to share the forecasts publicly.
A few of the headline aggregate forecasts are:
Forecasts
In each case I aggregated forecasts by removing the single most extreme forecast on each end, then taking the geometric mean of odds.
To reduce concerns of in-group bias to some extent, I calculated a separate aggregate for those who weren’t highly-engaged EAs (HEAs) before joining Samotsvety. In most cases, these forecasters hadn’t engaged with EA much at all; in one case the forecaster was aligned but not involved with the community. Several have gotten more involved with EA since joining Samotsvety.
Unfortunately I’m unable to provide forecast rationales in this post due to forecaster time constraints, though I might in a future post. I provided my personal reasoning for relatively similar forecasts (35% AI takeover by 2100, 80% TAI by 2100) in my WWOTF review.
WWOTF questions
Aggregate (n=11)
Aggregate, non-pre-Samotsvety-HEAs (n=5)
Range
What's your probability of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe?
25%
14%
3-91.5%
What's your probability of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe?
81%
86%
45-99.5%
FTX Foundation questions
For the purposes of these questions, FTX Foundation defined AGI as roughly “AI systems that power a comparably profound transformation (in economic terms or otherwise) as would be achieved in [a world where cheap AI systems are fully substitutable for human labor]”. See here for the full definition used.
Unlike the above questions, these are not conditioning on no pre-AGI/TAI catastrophe.
Aggregate (n=11)
Aggregate, non-pre-Samotsvety-HEAs (n=5)
Range
What's the probability of existential catastrophe from AI, conditional on AGI being developed by 2070?[1]
38%
23%
4-98%
What's the probability of AGI being developed in the next 20 years?
32%
26%
10-70%
What's the probability of AGI being developed by 2100?
73%
77%
45-80%
Who is Samotsvety Forecasting?
Edited to add: Our track record is now online here.
Samotsvety Forecasting is a forecasting group that was started primarily by Misha Yagudin, Nuño Sempere, and myself predicting as a team on INFER (then Foretell). Over time, we invited more forecasters who had very strong track records of accuracy and sensible comments, mostly on Good Judgment Open but also a few from INFER and Metaculus. Some strong forecasters were added through social connections, which means the group is a bit more EA-skewed than it would be without these additions. A few Samotsvety forecasters are also superforecasters.
How much do these forecasters know about AI?
Most forecasters have at least read Joe Carlsmith’s report on AI x-risk, Is Power-Seeking AI an Existential Risk?. Those who are short on time may have just skimmed the report and/or watched the presentation. We discussed the report section by section over the course of a few weekly meetings.
~5 forecasters also have some level of AI expertise, e.g. I did some adversarial robustness research during my last year of undergrad then worked at Ought applying AI to improve open-ended reasoning.
How much weight should we give to these aggregates?
My personal tier list for how much weight I give to AI x-risk forecasts to the extent I defer:
Acknowledgments
Thanks to Tolga Bilge, Juan Cambeiro, Molly Hickman, Greg Justice, Jared Leibowich, Alex Lyzhov, Jonathan Mann, Nuño Sempere, Pablo Stafforini, and Misha Yagudin for making forecasts.
Unlike the WWOTF question, this includes any existential catastrophe caused by AI and not just misaligned takeovers (this is a non-negligible consideration for me personally and I’m guessing several other forecasters, though I do give most weight to misaligned takeovers).
Why do I give little weight to Metaculus’s views on AI? Primarily because of the incentives to make very shallow forecasts on a ton of questions (e.g. probably <20% of Metaculus AI forecasters have done the equivalent work of reading the Carlsmith report), and secondarily that forecasts aren’t aggregated from a select group of high performers but instead from anyone who wants to make an account and predict on that question.
Why do I give little weight to AI expert surveys such as When Will AI Exceed Human Performance? Evidence from AI Experts? I think most AI experts have incoherent and poor views on this because they don’t think of it as their job to spend time thinking and forecasting about what will happen with very powerful AI, and many don’t have great judgment.