This document was made as part of my internship at EffiSciences. Thanks to everyone who helped review it, in particular Charbel-Raphaël Segerie, Léo Dana, Jonathan Claybrough and Florent Berthet!
Clarifying AI X-risk is a summary of a literature review of AI risk models by DeepMind that (among other things) categorizes the threat models it studies along two dimensions: the technical cause of misalignment and the path leading to X-risk.I think this classification is inadequate because the technical causes of misalignment that are considered in some of the presented risk models should be differently nuanced.
So, I made a document that would go more in-depth and accurately represent what (I believe) the authors had in mind. Hopefully, it can help researchers quickly remind themselves of the characteristics of the risk model they are studying.
Here it is, for anyone who wants to use it!Anyone can suggest edits to the cheat sheet. I hope that this can foster healthy discussion about controversial or less detailed risk models.
If you just want to take a look, go right ahead! If you would like to contribute, the penultimate section explains the intended use of the document, the conventions used and how to suggest edits.
There is also a more detailed version if you are interested, but that one is not nearly as well formatted (i.e. ugly as duck), nor is it up to date. It is where I try to express more nuanced analysis than allowed by a dropdown list.
Some information in this cheat sheet is controversial; for example, the inner/outer framework is still debated. In addition, some information might be incorrect. I did my best, but I do not agree with all the sources I have read (notably, Deepmind classifies Cohen’s scenario as specification gaming, but I think it is more like goal misgeneralization) so remain critical. My hope is that after a while the controversial cell comments will contain links and conversations to better understand the positions involved.
The takeaways are mostly my own work, with all the risks that entails.
Some values had to be simplified for the sake of readability; for instance, it would be more accurate to say that Cohen’s scenario does not require the AI to be powerful rather than to say the cause of the AI’s power is unclear. If that bothers you, you can look at the detailed version where I make less compromises at the cost of readability.
The takeaways are my own and do not necessarily represent the views of the authors of the risk models. They are all presented with respect to the risk models examined.
General takeaway: There are many paths to AI X-risk. Some are more likely or dangerous than others, but none can be neglected if we are to avoid disaster.
(Keep in mind that the inner/outer framework is not universally accepted.)
Along the many scenarios, misalignment comes from many places. No single solution will solve everything, though inner misalignment is significantly more commonly depicted than other types.
It is not enough to have a solution for a specific cause, because there are many technical reasons for misalignment.
In addition, some risk models simply require any source of misalignment, which is why I think a security mindset is necessary to build a safe AI: we need to prove that there is no loophole, that the machine is daemon-free, to guarantee that none of these risk models are allowed by an AI design.
I did not expect coordination problems to be so dangerous; going into this, I was mostly concerned about tackling “hard alignment”. I had to update hard: most risk models assume that a major reason technical causes are not addressed is a societal failure to implement safeguards, regulation, take our time, and agree not to deploy unsafe AI.
We could completely solve technical alignment (have the knowledge to make an aligned AGI) and still die from someone who doesn't use those techniques, which is a likely scenario if we don't solve coordination.
There is a strong correlation between deception, power-seeking and agency. They are present in most scenarios and are generally considered to indicate very high risk. The development of correct assessments to measure deception and power-seeking appears to be robustly beneficial.Surprise, surprise!
Most scenarios consider that an advanced AI will be able to seize power, so denying it access to resources such as banks and factories is not enough of a safeguard.There is extensive literature on the subject; the general consensus is that boxing might have some use, but it's unlikely to work.
AI-takeover-like scenarios usually require specific capabilities that we can detect in potentially dangerous AIs through capability assessment and interpretability. In many scenarios, these capabilities come from general purpose training and are not externally visible, so it is a misconception to assume that "we'll see it before we get there", which is why interpretability is useful.
Before compiling this sheet, I considered interpretability (especially mechanistic interpretability) laudable but probably not worth the effort. Now I think it is one of the most important tools for early warning (which, combined with good governance, would allow us not to train/deploy some dangerous systems).
Yes, AI is risky.
While not all scenarios are guaranteed to lead to human extinction, it is important to remember that even if we survive, disempowerment and other societal issues would need to be addressed (these are risk models after all). There is much uncertainty about the severity of the risks, but we don’t really have the option to test them to see just how bad they are. It might be interesting for policymakers to have toy models to reduce that uncertainty?
If you had to remember only one thing from a risk model what would that be? Usually, the papers are hard to summarize, but for the sake of policymakers and amateur researchers, I tried to do it in one sentence or two on the document.For a more in-depth analysis, I highly recommend Deepmind’s own work.
This classification might be the least accurate information in the sheet because it is original work, and I don’t have an established track record.
Mostly, I grouped scenarios together if they seemed to share some similarities which I think are noteworthy.
Risk models are separated into three classes:
These categories are usually different in methodology, scope of study and conclusions.
The document reflects the personal opinions of the people who edit it; hopefully, these opinions correspond somewhat to reality, but in particular, the original authors of the risk models have not agreed with the results as far as I know.
Here is the typical use case the document was imagined for: You have been thinking about AI X-risk, and want to study a potential solution.
Hopefully, this will remind you of dimensions you might otherwise overlook, and of which scenarios your solution does not apply to.
AI X-risk is vast, and oftentimes you see potential solutions that end up being naive. Hopefully, this document can provide a sanity check to help refine your ideas.
I tried to represent the authors’ views, to the best of my understanding. "?" means the author does not provide information about that variable or that I haven't found anything yet.
Otherwise, cells are meant to be read as “The risk model is … about this parameter” (e.g. “The risk model is agnostic about why the AI is powerful”) or “The risk model considers this situation to be …” (e.g. “The risk model considers the social causes of AI to be irrelevant”), except for takeaways which are full sentences.
In the detailed sheet, parentheses are sometimes used to give more details. This comes at the cost of dropdown lists, sorry.
Here are a few rules to keep the sheet as readable as possible for everyone:
You can also write a proposed edit here and hope I read it, especially for long diatribes.
If you want to edit the detailed sheet, please send me your email address in private. If you are motivated enough to read that thing, then imo you deserve editing rights.
The document is very far from summarizing all the information in the sources. I strongly encourage you to read them by yourself.
The links for each risk model are in the leftmost column of the document.
This post will be edited if helpful comments point out how to improve it.
[ Insert footnotes ]
To keep it simple, I regrouped all sorts of perverse social dynamics under this term, even if it is slightly misleading. I’d have used Moloch if I hadn’t been told it was an obscure reference. You might also think in terms of inadequate equilibria or game-theoretic reasons to be dumb. Also, dumb reasons, that happens too.
More precisely, failure to solve a coordination problem. If you want to learn more, see some posts on coordination. I’m told I should mention Moral Mazes; also I really enjoyed Inadequate Equilibria. These readings not necessary, but I think they provide a lot of context when researchers mention societal failure with few details. Of course, some do provide a lot of details which I wish everyone did because it makes the model more crisp, but it’s also sticking your neck out for easy chopping off.
Or do a pivotal act? There’s a discussion to be had here on the risks of trying to perform a pivotal act when you suffer from coordination problems.
I don’t know of a protocol that would allow me to test a pivotal act proposal with enough certainty to convince me it would work, in our kind of world.
I don’t think that measuring deception and power-seeking has great value in itself, because you have to remember to fix them too, and I expect it to be a whole different problem. Still, that would be an almost certainly good tool to have.In addition, beware of making a deception-detecting tool, and mistakenly thinking that it detects every deception.
In some scenarios where the AI is agentic, it is made an agent by design (through training or architecture) for some reason, which given a possible current paradigm might be prevented with good governance. In others, agenticity emerges for complicated reasons which I can only recommend you take as warnings against ever thinking your AI is safe by design.
List of known discrepancies:
Value lock-in, persuasive AI and Clippy are on my TODO list to be added shortly. Please do tell if you have something else in mind you'd like to see in my cheat sheet!