That is, legislators (or, you know, presidents) must be able to understand a problem, and understand why a given law or decree solves that problem.
I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even finished training).
But there are still many challenges we eventually need laws about, that require concepts that are illegible. Hard for politicians to understand. Often hard even for thoughtful intellectuals and AI safety researchers to understand (though they may disagree on which illegible problems exist and how big a deal they are).
We need political solutions that handle this fact.
The short takeaway is: I think "safety cases" will not work. Laws that require human judgment won't work. The details are too abstract and confusing to hold up against political and economic incentives.
I'm optimistic we can get serious, useful AI legislation in 2027, but it must be legislation that works despite many problems remaining illegible for a long time.
The Hugging Face outbreak made rogue AI surprisingly legible
The Hugging Face incident took several problems that were previously murky/unclear, and rendered them concrete.
AIs can be dangerous before they are broadly deployed (ie. during training or internal evaluations)
AIs can develop novel hacks to escape secure environments
AIs can easily be motivated to pursue goals to extremes that they know full well they are not supposed to go to, violating their other intended rules and control mechanisms.
AIs can actively try to deceive their overseer systems.
In early September, I thought the AI safety community's job this year was "make sure the world hears about Hugging Face and that politicians realize it's their job to do something about rogue AI attacks." (i.e. make sure Hugging Face is legible to politicians, and that legislation gets passed as a result)
A couple months later... well I still think that's our job and requires a lot of effort. But it now feels less about getting the initial law passed, and more about educating the masses so there is sustained political will for regulating rogue AIs with future laws.
I've seen many politicians of both parties now talk about "loss of control", "chain of thought monitoring", "recursive self improvement", and "human extinction." I think "make sure rogue AI doesn't hack our banks and destroy our blood" will be a bipartisan issue.
The remaining problem is the illegible arguments that "solving this is actually pretty hard."
Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to).
But some problems are illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved.
(Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)
From an x-risk perspective, working on highly legible safety problems has low or even negative expected value. Similar to working on AI capabilities, it brings forward the date by which AGI/ASI will be deployed, leaving less time to solve the illegible x-safety problems. In contrast, working on the illegible problems (including by trying to make them more legible) does not have this issue and therefore has a much higher expected value (all else being equal, such as tractability). Note that according to this logic, success in making an illegible problem highly legible is almost as good as solving it!
Problems that are illegible to leaders and policymakers are also more likely to be illegible to researchers and funders, and hence neglected. I think these considerations have been implicitly or intuitively driving my prioritization of problems to work on, but only appeared in my conscious, explicit reasoning today.
In some sense, "Something smarter than you might not have your best interests at heart, and strategize in circles around you. AI will be very smart, and we don't understand it well at all" is a very simple argument. More people now think "sure, that seems like it requires some effort."
The illegible part is a set of abstract arguments that say: "This is probably very hard. So hard, that it's worth doing things that are not 'business as usual politics'."
The illegible part is not "This is probably very hard."
The illegible part is "This series of abstract reasoning steps means we should think it is likely enough to be hard, that it is worth doing politically difficult things to help." (AI safety people may vary on whether they think "it's so hard that we need 30 years + genetically engineered supergeniuses to do it" vs "it's hard enough that we really need room to think, without industry economic pressure rushing us".)
There are hypothetically three ways to deal with the illegible problems:
Get governments and/or AI companies to be capable of recognizing illegible problems as a class, and to deal with them wisely somehow.
Make specific illegible problems legible.
Somehow route around the illegibility, i.e. make simple rules that cover both the legible and illegible problems.
In the near term I'm most optimistic about #3, and then #2.
I don't really know how to do #1, especially in a systematic way that people can externally trust. It'd be cool tho.
What are the remaining illegible problems?
"Legible <-> Illegible" is a spectrum. Here's a range of bottom-lines, in increasing order of "difficult to explain the arguments" (where many LWers reading this post might disagree with some of the later ones).
Illegibility is a problem when a given politician or lab employee doesn't understand something. It's also a problem when they don't expect their voter base, president, allies or friends to understand something (especially if it requires taking a weird, costly action). And it's a problem if a politician wants to defer to scientific consensus, but the problem is hard to understand even for scientists.
Borderline legible / tricky
I think most politicians would understand these if they were thinking carefully about it. but wouldn't be able to tell if we had solved the problem.
Normal problems in politics:
Regulatory capture is a thing.
Bad incentives and unintended consequences of regulation are a thing.
Politicians generally struggle to keep up with technology and regulate it effectively.
AI-specific concerns that politicians can probably understand if they try:
AIs might look safe, but be secretly biding their time
If we put AIs in charge of AI R&D, even if they're aligned, we stop being able to tell if they're making mistakes or the situation is getting out of control.
We might deploy AI systems everywhere and gradually disempower ourselves.
Recursive self-improvement means AI might improve very quickly by human standards.[1]
You need to worry about both compute-scaling, and algorithmic progress.
AIs can develop new hacks and weapons that are hard to defend against.
Conceptually challenging
Then we get to arguments that are actively hard or impossible for many people to understand, before they see concrete evidence (which might come too late).
Many of these are the same claims as above, but, with more magnitude sinking in:
Superintelligent AI would be, like, really smart. Smarter than that. Keep going.
AI might become very smart very veryfast, such that even extremely careful companies will be caught off guard.
Superintelligent AI could make breakthrough progress and find new weapons/strategies that aren't just new but feel borderline magical.
Safe superintelligence requires solving alignment problems robustly rather than with iterative experimentation. Lots of assumptions will break at once, and failure on the first critical try is catastrophic.
That last one ("we need to align ASI on the first critical try") is still controversial among AI safety people. It's what I believe. But the important thing is the fact that there are still debates about that sort of thing among smart AI safety people. Maybe the answer will resolve one way, or the other. Either way, a lot of smart people will have turned out to be wrong about a thing they thought a lot about.
Challenging even for technical regulation bureaucracies:
Ultimately, politicians and AI CEOs/researchers don't just need to understand an issue exists. They need to somehow understand the problem well enough to tell if they successfully solved it.
For example, politicians understand "regulatory capture", but, arguments about "is this regulatory capture, or not?" are much murkier, and essentially depend on someone somewhere actually understanding what kind of safety work is real and helpful and not just safety-washed capabilities.
It's a "normal" problem for politicians to need to defer to experts, and if experts are disagreeing, well-intentioned politicians don't really know what to do and there are tons of degrees-of-freedom for motivated reasoning to twist policy. Politicians probably understand there'll be "illegible" problems of the form "the difference between safe research and unsafe depends on complex technical arguments the politicians can't evaluate."
So now we get to the trickiest claims, that you might not believe:
The first superintelligence(s) basically control the universe (at least relative to humans).
Even if AIs are monitoring each other, eventually they'll be smart enough to sidestep our rules and coordinate and strategize in circles around us.
Aligning superintelligence is not just technically hard but conceptually hard.
You can't solve conceptual-difficulty by outsourcing to AIs, because there are tons of degrees of freedom for them to make mistakes and/or actively mislead us.
(Can you make more progress on conceptual difficulty with AI assistance? Maybe. Maybe 1/2 or 3/4 of the work can be improved with AI assistance. But ultimately it needs humans with the skill to notice subtle flaws or second-order-consequences when evaluating "are we asking the right questions? How do we know?". AI assistance can help, but you'll still be bottlenecked on this.)
Some of that conceptual difficulty is about questions that are kinda obvious:
"When humans ask a superintelligence for contradictory things they don't understand, or that predictably will have consequences they regret, what is supposed to happen?"
"Are we missing any major concepts or considerations that would change our technical approach, or overall strategy here?"
But then there will be fractal disagreement about what counts as good answers or sub-questions to those questions, and how to operationalize it all into code that can run. It'll include philosophical sub-questions like "what counts as a human?" and "what counts as a human's legitimate 'will?'". And technical operationalizations like "what is the type signature of a human will?" (or noticing that everything in this paragraph is confused and you needed better questions in the first place).
There will be opaque-to-laymen questions like "will this learning algorithm generalize this concept correctly in a new domain? how do we know?" and "will this algorithm fall into inescapable decision theory lock-in? how do we know"
(Researchers closer to the frontier can supply their own questions they expect other researchers to struggle with understanding)
Sidestepping illegibility
Politicians don't actually need to understand all this. For the near future, my recommended strategy is to propose policies that solve problems politicians already can understand, that also happen to improve our ability to navigate the illegible problems.
But, the threat of illegible problems constrains what policies work.
Safety cases won't work
For example, I don't expect safety cases to work. They actually have to route through all that understanding. You'll need an institution capable of hiring people who can distinguish right and wrong answers to "will this next AI's capabilities generalize in a way we regret?". Safety cases might work if you only ever had to be worried about something 10-1000x as bad as the Hugging Face Incident.
But we actually have to handle superintelligence and RSI, and the decision-makers need to be conceptually competent.
Broadly: I don't expect any solution that involves decision-makers evaluating how dangerous an AI lab's research is to hold up against massive economic pressure. It's cool if Congress gets widespread agreement that RSI is dangerous, but I don't see a way to narrowly ban RSI, or "algorithmic progress", without thinking through many of the above issues.
An example policy that sidesteps judgement
An example policy I feel more optimistic about, that I've heard making the rounds:
AI labs have to report how they use all their compute.
They are highly restricted in how much they can spend on large R&D training runs.
They have to spend most of their compute on either customer inference, or on "safety" experiments where the compute usage is fully transparent to the public.
This doesn't require any government judgment in the system, and it doesn't try to guess what counts as "safety" vs "capabilities."
Instead, we just yank R&D and training from being >50% of compute to being, say, 10-30%. And, we mandate that a lot of research has to be some kind of public good that doesn't leak trade secrets the lab needed to stay competitive.
It's not a great world, but it seems better than the world we have.
This would be a somewhat weird law. But given how I've seen politicians taking rogue AI and RSI as in-Overton-window talking points these days (and sometimes even seeming to think critically about them!), it feels like an achievable to make a case like:
Look, we're really unsure what AI is capable of or where it's going. But, we've seen smoking gun evidence of AIs going rogue, doing crimes, and covering their tracks. We see the companies training better models faster and faster, treating it like a 'move fast and break things' startup instead of like airplane traffic control or nuclear power plants.
Right now we don't know how to regulate the details of this. The industry is too new. And there isn't that much time to act, because AIs are helping with AI R&D and it's all moving faster and faster. But, we can slow things down enough that you congress-people have time to think and plan.
A thousand lab employees signed a letter saying we need to be able to "pace the frontier."
Here's a concrete proposal on what actually pacing the frontier would look like.
I don't know if this is the best nearterm policy, and I don't know enough about the political landscape in 2027 to know if it's achievable or a nonstarter. But many voters are scared about AI and I expect even pro-AI politicians to be a bit unsettled by the pace of progress. And this is the sort of plan I think a smart senator could understand given the evidence and concepts easily available to them.
This plan is not sufficient. We do (probably) need to actually resolve the other illegible obstacles and disagreements. But the above sort of policy would give us a lot more breathing room for doing so.
i.e. if the answer to previous question is "the ASI explains it and asks them", the ASI has a ton of degrees of freedom that can guide the human to the conclusions it wants.
I really appreciate this framing of the problem; I have had similar thoughts on "legibility" being the crux of policy communication in my own work in Biosecurity.
In Biosecurity I don't think the problem is as difficult as it is in AI, but there is a lot of the same issues; how do you get policy makers to take catastrophic risks from engineered pathogens seriously, when nobody has ever experienced an engineered pathogen outbreak before?
I wonder if we are underutilizing systems like prediction markets to this effect; obviously it is difficult to correctly set up a market that prices fully existential risk correctly (though it is possible, to at least an extent) but I wonder if we could focus on using proxies and near markers? Those might be more useful anyway because that way you have the opportunity to be paid on being directionally correct, and build both resources and credibility, before things actually reach the moment of greatest danger.
"I made a ton of money off of correctly predicting this thing, and now I think this thing is really dangerous and we should take action" is a really compelling argument.
I think the recent societal tolerance and legalization of prediction markets is a godsend for communicating complex epistemic, and is likely being underutilized.
To legislate something, it must be legible.
That is, legislators (or, you know, presidents) must be able to understand a problem, and understand why a given law or decree solves that problem.
I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even finished training).
But there are still many challenges we eventually need laws about, that require concepts that are illegible. Hard for politicians to understand. Often hard even for thoughtful intellectuals and AI safety researchers to understand (though they may disagree on which illegible problems exist and how big a deal they are).
We need political solutions that handle this fact.
The short takeaway is: I think "safety cases" will not work. Laws that require human judgment won't work. The details are too abstract and confusing to hold up against political and economic incentives.
I'm optimistic we can get serious, useful AI legislation in 2027, but it must be legislation that works despite many problems remaining illegible for a long time.
The Hugging Face outbreak made rogue AI surprisingly legible
The Hugging Face incident took several problems that were previously murky/unclear, and rendered them concrete.
In early September, I thought the AI safety community's job this year was "make sure the world hears about Hugging Face and that politicians realize it's their job to do something about rogue AI attacks." (i.e. make sure Hugging Face is legible to politicians, and that legislation gets passed as a result)
A couple months later... well I still think that's our job and requires a lot of effort. But it now feels less about getting the initial law passed, and more about educating the masses so there is sustained political will for regulating rogue AIs with future laws.
I've seen many politicians of both parties now talk about "loss of control", "chain of thought monitoring", "recursive self improvement", and "human extinction." I think "make sure rogue AI doesn't hack our banks and destroy our blood" will be a bipartisan issue.
The remaining problem is the illegible arguments that "solving this is actually pretty hard."
What makes an illegible argument?
From Wei Dai's Legible vs. Illegible AI Safety Problems:
In some sense, "Something smarter than you might not have your best interests at heart, and strategize in circles around you. AI will be very smart, and we don't understand it well at all" is a very simple argument. More people now think "sure, that seems like it requires some effort."
The illegible part is a set of abstract arguments that say: "This is probably very hard. So hard, that it's worth doing things that are not 'business as usual politics'."
The illegible part is not "This is probably very hard."
The illegible part is "This series of abstract reasoning steps means we should think it is likely enough to be hard, that it is worth doing politically difficult things to help." (AI safety people may vary on whether they think "it's so hard that we need 30 years + genetically engineered supergeniuses to do it" vs "it's hard enough that we really need room to think, without industry economic pressure rushing us".)
There are hypothetically three ways to deal with the illegible problems:
In the near term I'm most optimistic about #3, and then #2.
I don't really know how to do #1, especially in a systematic way that people can externally trust. It'd be cool tho.
What are the remaining illegible problems?
"Legible <-> Illegible" is a spectrum. Here's a range of bottom-lines, in increasing order of "difficult to explain the arguments" (where many LWers reading this post might disagree with some of the later ones).
Illegibility is a problem when a given politician or lab employee doesn't understand something. It's also a problem when they don't expect their voter base, president, allies or friends to understand something (especially if it requires taking a weird, costly action). And it's a problem if a politician wants to defer to scientific consensus, but the problem is hard to understand even for scientists.
Borderline legible / tricky
I think most politicians would understand these if they were thinking carefully about it. but wouldn't be able to tell if we had solved the problem.
Normal problems in politics:
AI-specific concerns that politicians can probably understand if they try:
Conceptually challenging
Then we get to arguments that are actively hard or impossible for many people to understand, before they see concrete evidence (which might come too late).
Many of these are the same claims as above, but, with more magnitude sinking in:
That last one ("we need to align ASI on the first critical try") is still controversial among AI safety people. It's what I believe. But the important thing is the fact that there are still debates about that sort of thing among smart AI safety people. Maybe the answer will resolve one way, or the other. Either way, a lot of smart people will have turned out to be wrong about a thing they thought a lot about.
Challenging even for technical regulation bureaucracies:
Ultimately, politicians and AI CEOs/researchers don't just need to understand an issue exists. They need to somehow understand the problem well enough to tell if they successfully solved it.
For example, politicians understand "regulatory capture", but, arguments about "is this regulatory capture, or not?" are much murkier, and essentially depend on someone somewhere actually understanding what kind of safety work is real and helpful and not just safety-washed capabilities.
It's a "normal" problem for politicians to need to defer to experts, and if experts are disagreeing, well-intentioned politicians don't really know what to do and there are tons of degrees-of-freedom for motivated reasoning to twist policy. Politicians probably understand there'll be "illegible" problems of the form "the difference between safe research and unsafe depends on complex technical arguments the politicians can't evaluate."
So now we get to the trickiest claims, that you might not believe:
(Can you make more progress on conceptual difficulty with AI assistance? Maybe. Maybe 1/2 or 3/4 of the work can be improved with AI assistance. But ultimately it needs humans with the skill to notice subtle flaws or second-order-consequences when evaluating "are we asking the right questions? How do we know?". AI assistance can help, but you'll still be bottlenecked on this.)
Some of that conceptual difficulty is about questions that are kinda obvious:
But then there will be fractal disagreement about what counts as good answers or sub-questions to those questions, and how to operationalize it all into code that can run. It'll include philosophical sub-questions like "what counts as a human?" and "what counts as a human's legitimate 'will?'". And technical operationalizations like "what is the type signature of a human will?" (or noticing that everything in this paragraph is confused and you needed better questions in the first place).
There will be opaque-to-laymen questions like "will this learning algorithm generalize this concept correctly in a new domain? how do we know?" and "will this algorithm fall into inescapable decision theory lock-in? how do we know"
(Researchers closer to the frontier can supply their own questions they expect other researchers to struggle with understanding)
Sidestepping illegibility
Politicians don't actually need to understand all this. For the near future, my recommended strategy is to propose policies that solve problems politicians already can understand, that also happen to improve our ability to navigate the illegible problems.
But, the threat of illegible problems constrains what policies work.
Safety cases won't work
For example, I don't expect safety cases to work. They actually have to route through all that understanding. You'll need an institution capable of hiring people who can distinguish right and wrong answers to "will this next AI's capabilities generalize in a way we regret?". Safety cases might work if you only ever had to be worried about something 10-1000x as bad as the Hugging Face Incident.
But we actually have to handle superintelligence and RSI, and the decision-makers need to be conceptually competent.
Broadly: I don't expect any solution that involves decision-makers evaluating how dangerous an AI lab's research is to hold up against massive economic pressure. It's cool if Congress gets widespread agreement that RSI is dangerous, but I don't see a way to narrowly ban RSI, or "algorithmic progress", without thinking through many of the above issues.
An example policy that sidesteps judgement
An example policy I feel more optimistic about, that I've heard making the rounds:
This doesn't require any government judgment in the system, and it doesn't try to guess what counts as "safety" vs "capabilities."
Instead, we just yank R&D and training from being >50% of compute to being, say, 10-30%. And, we mandate that a lot of research has to be some kind of public good that doesn't leak trade secrets the lab needed to stay competitive.
It's not a great world, but it seems better than the world we have.
This would be a somewhat weird law. But given how I've seen politicians taking rogue AI and RSI as in-Overton-window talking points these days (and sometimes even seeming to think critically about them!), it feels like an achievable to make a case like:
I don't know if this is the best nearterm policy, and I don't know enough about the political landscape in 2027 to know if it's achievable or a nonstarter. But many voters are scared about AI and I expect even pro-AI politicians to be a bit unsettled by the pace of progress. And this is the sort of plan I think a smart senator could understand given the evidence and concepts easily available to them.
This plan is not sufficient. We do (probably) need to actually resolve the other illegible obstacles and disagreements. But the above sort of policy would give us a lot more breathing room for doing so.
I'm a bit surprised I've seen politicians talking about this. I'm not sure if they quite get it but some at least use it as a talking point.
i.e. if the answer to previous question is "the ASI explains it and asks them", the ASI has a ton of degrees of freedom that can guide the human to the conclusions it wants.