Summary

For my MLSS project, I created this distillation of “Is power-seeking AI an existential risk?” which was originally written in 2021 by Joe Carlsmith, an analyst at Open Philanthropy. After a while, Stephen McAleese contacted me and proposed to re-edit it and upload it as a post. Here is the result.

The report estimates the level of existential risk from AI by breaking the possibility of an existential catastrophe from AI down into several enabling steps, assigning probabilities to each one, and estimating the total probability by multiplying the probabilities. The report concludes that advanced AI is a potential source of existential risk to humanity and has a roughly 5% chance of causing an existential catastrophe by 2070. A year later Carlsmith increased his estimate to >10%.

Introduction

The report is about risks from advanced artificial intelligence by the year 2070. The year was chosen by the author because it’s near enough to be within the lifetimes of people alive today while being far enough to reason about the long-term future. Carlsmith reasons that highly intelligent reasoning and planning agents will have an instrumental incentive to acquire and maintain power by default because power increases the probability of agents achieving their goals for a wide range of goals. Power-seeking in AI systems would make them particularly dangerous. Unlike passive risks such as nuclear waste, since power-seeking AI systems would actively search for ways to acquire power or cause harm, they could be much more dangerous.

Therefore, the existential risk posed by advanced AI systems could be relatively high. Risk can be decomposed such that Risk ≈ (Vulnerability x Exposure x Severity) / Ability to Cope. Advanced AI seems like a risk that would have high severity and exposure and possibly high overall risk. The general background of the report includes two important points: (a) intelligent agency is an extremely powerful force for controlling and transforming the world, and (b) building agents much more intelligent than humans is “playing with fire”. In the following sections, I’ll describe the six events mentioned in the report that would need to happen for AI to cause an existential catastrophe and estimate the probability of each one to calculate an overall estimate of existential risk from AI by the year 2070. Note that whenever I am describing a...

Posts

Wikitag Contributions

Comments