This is an 8-page comprehensive summary of the results from Threshold 2030: a recent expert conference on economic impacts hosted by Convergence Analysis, Metaculus, and the Future of Life Institute. Please see the linkpost for the full end-to-end report, which is 80 pages of analysis and 100+ pages of raw...
Full version on arXiv | X Executive summary AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI...
Deger Turan, Tantum Collins, davidad and Charbel-Raphael Segerie contributed to the writing of this post. Andrew Critch, Ben Goldhaber, Ozzie Gooen, and Evan Miyazono reviewed this post. tl;dr: Open Agency Architecture (OAA) is a bold proposal for AI alignment. It requires a wide ranging formal-modeling enterprise that integrates into a...
This is an 8-page comprehensive summary of the results from Threshold 2030: a recent expert conference on economic impacts hosted by Convergence Analysis, Metaculus, and the Future of Life Institute. Please see the linkpost for the full end-to-end report, which is 80 pages of analysis and 100+ pages of raw writing and results from our attendees during the 2-day conference.
The Threshold 2030 conference brought together 30 leading economists, AI policy experts, and professional forecasters to evaluate the potential economic impacts of artificial intelligence by the year 2030. Held on October 30-31st, 2024 in Boston, Massachusetts, it spanned two full days and was hosted by Convergence Analysis and Metaculus, with financial support... (read 2852 more words →)
Full version on arXiv | X
Executive summary
AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment. This loss of human influence will be centrally driven by having more competitive machine alternatives to humans in almost all societal functions, such as economic labor, decision making, artistic creation, and even companionship.
A gradual loss of control of our own civilization might sound implausible. Hasn't technological disruption usually improved aggregate human... (read 322 more words →)
On behalf of the Metaculus team here: If @Dan H (or anyone) wants to actually easily, rigorously demonstrate a single-prompt forecasting bot that can outcompete Metaculus forecasters, I invite them to do so. We’ll even cover your API credits. Just enter your bot in the benchmarking series. We’ve completely solved the data leakage issue—which does appear to be a fatal flaw in Dan’s approach. I myself am skeptical that a bot using a single prompt can really compete given everyone's attempts so far, but I would love to be proven wrong!
Deger Turan, Tantum Collins, davidad and Charbel-Raphael Segerie contributed to the writing of this post.
Andrew Critch, Ben Goldhaber, Ozzie Gooen, and Evan Miyazono reviewed this post.
tl;dr: Open Agency Architecture (OAA) is a bold proposal for AI alignment. It requires a wide ranging formal-modeling enterprise that integrates into a comprehensive world model. We plan to begin developing OAA by iterating on smaller, domain-specific applications that can find near-term use as institutional decision-making tools and provide feedback to continue improving the OAA framework . This document decomposes that goal into sub-agendas designed to engage different technical communities in an international collaboration.
The AI Objectives Institute (AOI) aims to ensure that AI and future economic systems... (read 3471 more words →)