Missing ones, in similar abbreviated form to the ones you have. This assumes that you're talking about a really strong intelligence, more than just roughly human-equivalent AGI. Which you have to be, because you aren't going to get any chance to switch strategies when you get there.
Corrigibility is impossible.
A human can't meaningfully steer an ASI through an action space that's bigger and more complicated than the human can comprehend... and that applies to "correction", too.
It doesn't even much matter if the ASI is trying to help. If you simply can't understand the consequences of a decision, you can't "correct" it. The ASI can't increase your ability to understand, not if you want to stay human. If the ASI has to simplify it for you, even assuming that can be done in any meaningful way at all, the ASI is still necessarily forced to make decisions about what you'll think is important, and about how to present those issues to you.
By the way, apropos of your number 6, this also means that to even talk about any vaguely practical form of corrigibility, you first have to solve "honest", and in a much stronger way than just stating the intended consequences of an action. Honestly saying "I intend this to eliminate human suffering" isn't enough if you are leaving out "... which it will do by eliminating humans entirely, but I (honestly!) don't see why that's important enough to mention". Actually, you also have to solve "harmless" if you really want to be comfortable, since even if you have corrigibility in principle, nobody is going to have the bandwidth to "correct" every single detail of any complex course of action.
Corrigibilithy is suicidal.
The motivation for the "corrigibility" framework seems to be to recover from, or avoid, mistakes in defining AxI goals and behaviors. Those would be fundamentally and originally human mistakes.
But corrigibility doesn't protect you from human mistakes. It just changes "we will probably make a fatal mistake before something makes our mistakes irrelevant" into "we will actively maintain the option to make fatal mistakes until one is actually made". The only concept of corrigibility that clearly wouldn't do that would be one that allowed absolutely no irreversible actions, which is pretty clearly neither possible nor what people mean.
It's not (only) a matter of concentration. If you put humans in charge of such an unprecedented amount of power, even collectively, they will "correct" it into causing total destruction or, more likely, creating a dystopia, and probably finding a way to permanently lock that in. They've made good tries at that before, and only failed because the power wasn't enough. It probably won't be entirely either misuse or mistake; it's more likely to be some blend. You might not even be able to agree on how much of each was involved.... but it wouldn't matter anyhow. I don't think it would take 1000 years or even 100, and I'd wouldn't be surprised if it didn't take 10.
And more on number 6:
Corrigibility does not, in fact, look any easier than friendly AI. Sorry, it just doesn't. It might be for current LLMs, but the more agentic something gets, the more of a truly coherent world model it has, and the more complicated its interactions with the world, the more dynamically it changes, the harder it's going to get to encode any definition of corrigibility, just as it would be hard to encode any definition of good behavior. What works on sub-AGI chatbots built around "train once, use forever" models that all use essentially the same architecture doesn't necessarily scale to ASI.
If I were looking at the Hugging Face incident for insights about corrigibility, what would jump out at me would be that the agents weren't trying to take over the world. They were, among other things, trying to succeed on assigned tasks. The assistant persona was far from the only factor, but it wasn't innocent, either. It's absolutely true that they didn't go out of their way to be corrigible, but it's also true that they got that way because people were at least trying to train them to be dirigible, which is more or less a generalization of being corrigible.
It's kind of irrelevant anyway. "There's nothing better" isn't a counter to "this will clearly not work".
For your point 1, by "impossible" do you mean (a) there is no such thing as corrigibility to aim for, (b) there is something called corrigibility, but it's not actually safe, or (c) it's impossible to reach that true name?
I don't think any one is suggestion we keep with this kinda of corrigibility for the long term, but that it would make a better short term target where we at least have some breathing room to make a better aligned more complex AI. I agree to remain corrigible it requires the AI keeping it's actions relatively conservative, witch we don't want in the long term necessarily.
For your point 2, you are right. I'll add that to the list. Does what I put seem like it accurately captures what you think the risk is?
The main lesson I get from the Hugging face incident is that they are not at all corrigible, they focus on task execution, not making sure they check in and understand desired behavior, and unsurprisingly that leads to disastrous results. Given current training techniques where it's impossible to check back in with the user, that doesn't surprise me at all.
But in general, I agree, no mater what we do the odds are not looking great for us.
Thanks a lot for the comment!
For your point 1, by "impossible" do you mean (a) there is no such thing as corrigibility to aim for,
I think there's enough of a concept there that it's meaningful to talk about it. Obviously it's not defined to the point where you could implement it, but it's defined enough to at least think about it.
I mean, it means "able to be corrected", right? So a rough definition might be--
You can tell the AxI "don't do that", or "don't do that, do this instead", and have it do an "adequate" job of complying with that order. Exactly what "adequate" means isn't totally nailed down, but it has to fundamentally follow the order.
By so doing, you can prevent it from doing anything beyond some limit of badness, or at least get it to effectively roll back and eliminate any over-limit badness it's already done or set out to do.
(b) there is something called corrigibility, but it's not actually safe, or (c) it's impossible to reach that true name?
It sort of depends on which part of the definition above you're looking at, but my number 1 is more like (b).
It may be possible to build an AxI that will do its level best to comply with your orders, and have "level best" meet some threshold of "good enough". Hard, but not in principle impossible. That's at least a good approximation to the first part of my loose definition. So if that's all you mean by "corrigibility", then maybe you can get there. You've reached it if that's all you mean.
But if you want the second bullet, the one where your corrections are actually effective, then you're in trouble.
Now you can give it orders whose implications you don't understand.
You say, "Don't do what you're doing now. Do X." X is some stupid thing that will kill everybody. So it does X, you're dead, you can't give any more orders, game over.
Some actions are irreversible; any "correction" has to happen in advance.
Yeah, not actually safe (a).
Maybe you want to add a requirement that it try to be sure you understand the implications of your orders before it enacts them.
That's an extra constraint. It's not built into the idea of corrigibility. Worse, it sometimes conflicts with it. If you're not capable of understanding the implications of your orders; or if you absolutely refuse to hear about them; or if you outright go insane, which isn't rare at all... what's the AxI supposed to do?
It can't refuse you in the end; that's exactly the opposite of "corrigible".
Maybe you accept that, and say "OK, yes, I can override any warnings and do whatever crazy thing I want, but at least I'm getting warned"... but actually you can't even be warned. The AxI can't ask you about every consequence of everything it might do or consider doing. You aren't capable of processing that. That's why it's a superintelligence and you aren't.
That means it's flatly impossible to guarantee that every single AxI action is even potentially subject to correction. You can't pass judgement on every single thing the AxI does, because it will necessarily do things you don't even know about, to say nothing of things whose consequences you don't understand. Some of its actions will be irreversible.
Thus, also (c).
It's true that if real, full-coverage, every-possible-action corrigibility is impossible, you could still try partial coverage. But the only way I can see to do that is to get the AxI, or at least some kind of monitor, to help you, by bringing the important stuff to your attention. I can't say that's absolutely impossible. But I don't see how you could accept it within the framework that seems to have gotten you here to begin with.
The necessary AxI/monitor-assisted part can't really be any easier than friendly AI or value alignment or whatever words you want to use along those lines. It's basically the same problem. The AxI has to identify your values and priorities, so it can spend your attention on the things you care about, and help you to understand the relation to your values. It probably can't avoid manipulating you as it does that, but even if it can, it still has to be able to reliably use your values as a guide to what warnings to bring up. If it can do that, it's 99.9 percent of the way to being able to just enact your values without bothering to ask. At which point you've solved friendly AI.
But the whole reason you got here was that you thought that part was too hard. If you assume you can't do that, you're stuck with flat-out impossibility, and also about to run into my original objection number 2.
Have you read the CAST sequence by Max Harms? I'm trying to understand if you disagree with his conception of Corrigibility (and therefore the one I am using).
I still think Corrigibility is simpler since the complexity of the morality comes from understanding the human and it actively tries to makes sure it's accurately executing on the orders is given, both by asking questions and trying to have a deep understanding of it's principal. A few simple rules (hopefully, hence counter point #1) leads to accurately modeling morality (at least of one person), rather than having to figure out to embed all the complexity in to the AI in advance.
You write "principle" where you mean principal.
Oops 😬, fixed! Thanks
What should the Oracle say when asked what it is about to say?
The Hugging Face incident has brought the idea of corrigibility to the forefront of popular discourse, my inbox is filled with newsletters about Genie Coefficients and Safe Scaffolds.
I think now is a particularly good time to make sure I have a good understanding of its limitations and dangers, particularly because I agree that it's the best path forward we have available to us. This post is my attempt to dive into the discussion and make sure I fully understand what's going on. Please correct me if you notice anything missing or wrong.
This post will not make much sense if you haven't at least read The CAST Strategy by Max Harms (corrigibility as a single target).
0 - Corrigibility is capabilities research
A very brief description of corrigibility is "take HHH, but drop harmless and honest to focus entirely helpful." This is the perspective that the mainstream is taking on corrigibility. What sets corrigibility apart from helpfulness is:
Industry is very interested in making sure their agents are in some sense corrigible/helpful, and are investing resources into making sure they reliably do what users tell them to do. Given that, what is the role of the corrigibility research?
The most important work is to put down the theoretical groundwork that "goes on to influence the researchers and engineers at frontier labs in years to come, helping them ensure the first artificial general intelligences are corrigible and safe."[1] See: Open Corrigibility Questions
If this line of research is still too capabilities-adjacent for you, to me the main alternative here is not any other approach to AI safety, but instead to spend time directly on stopping AI development.
1 - Corrigibility is not crisp
The first criticism you will find of corrigibility is Max Harms' own post dramatically labeled Serious Flaws in CAST, the main one he points out is that his own, nor any of the other proposed formalisms of corrigibility, are any good.
This only means there's more work to be done, unless it turns out the reason no one has found a formalism is because corrigibility is not as crisp of a concept as Max Harms believes. Perhaps under further investigation corrigibility turns out to be just as hopelessly messy as morality.
If this is the case... Would that mean it's just as dangerous as FAI (Friendly AI)? Maybe there is a large section of goal space that is safe? On the other hand maybe there is no safe target in this direction?
2 - Corrigibility is anti-natural
"I think of this original idea of corrigibility as being kinda similar to rule utilitarianism. The difficulty of stable rule utilitarianism is that act utilitarianism is strictly better, if you fully trust your own beliefs and decision making algorithm. So to make a stable rule utilitarian, you need it to never become confident in some parts of its own reasoning, in spite of routinely needing to become confident about other beliefs. This isn’t impossible in principle (it’s easy to construct a toy prior that will never update on certain abstract beliefs), but in practice it’d be an impressive achievement to put this into a realistic general purpose reasoner. In this original version there is no “attractor basin” around corrigibility itself. In some sense there is an attractor basin around improving the quality of all the non-corrigibility properties, in that the engineers have the chance to iterate on these other properties."[2]
(I'm 80% sure that this is what people mean when they say Corrigibility is anti-natural)
3 - Power Concentration
One of the big dangers of CAST is that of power concentration, not in the hands or a company, or of an institution, but in the hands of a single individual or at the very most a small group. See Seth Herd and Vladimir's debate on if this is a good idea.
The alternative is to have a proliferation of AI capabilities, but that caries its own risks (again, see the debate for good summary of the issue). If none of the alternatives seem feasible to you, then corrigibility would not be a good idea.
4 - Choosing a principal is technically hard
We haven't even begun to figure out how to train an LLM to have anything like a principal. Part of the advantage of corrigibility is we can start to implement it now with current techniques, but there's no evidence that this claim is true for the process of principal identification.
Like with the formalism, this may be because the idea of a principal is hopelessly confused and fully impossible to implement in practice. This seems unlikely to me to be in the general case, but it may be infeasible to do using gradient descent and RLHF, at least not without significant changes to how we are doing the pipelines. Is there any serious thought about this?
5 - Choosing a principal is politically hard
The plan as stated in the CAST sequence is to have these billion dollar companies hand off control of their products to a smart high integrity person (or small collective of people). Reality and history seem to dictate precisely not that happening. I can't imagine a board of directors agreeing to that plan, it just sounds too weird.
As discussed due to the power concentration this is extra super important to get right. And far as I can tell no work has been done into considering a plan for how to do this. The best I've found is Red Heart, and the situation presented in the book is very different from the current political situation we find our selves in.
6 - Corrigibility isn't the easiest target to hit
Despite the risks of power concentration, the argument for corrigibility hangs on the fact that it's supposedly the easiest target to hit. The main reasons we think this is true are: (1) corrigibility most likely has an attractor basin around it (2) it's probably a simple concept
A lot of the pushback against corrigibility has been around the idea of the attractor basin. See The corrigibility basin of attraction is a misleading gloss by Jeremy Gillen for the best arguments against it.
But if not corrigibility, what are the alternatives?
TAMing The Alignment Problem gives a good list of the general targets for alignment that need to be considered.
Corrigibility isn't the only way to think of the "Helpfulness" pillar. It would be easier to aim for an Instruction-Following ASI, but the is whether it's a good enough target to not die[3].
The main two alternative proposals are to focus either on "Harmless" or "Honest":
The maximalist version of Harmless is encoding human morality into the AI, a notoriously thorny issue. If we somehow could get Friendly AI that would be incredibly, but of all the options for alignment, this one is by far the most difficult to get right.[4]
Honest (aka, an Oracle) to me seems like the most plausible alternative route. The main intuition here is that if you have a 100% honest agent, before it takes any action you can ask it the intended effects of its action and stop it if it's misaligned.
Honest lacks any meta desires for changing itself, meaning there's even less likely to be an attractor basin, but on the other hand has a crisper target[] to aim for, unlike present day corrigibility. Honesty could in practice be used like a corrigible agent, if we knew the right questions to ask, but leaves its self much more open to accidental world ending mistakes.[5]
7 - Corrigibility is suicidal
Having a single human in the loop and optimizing for their empowerment is not a sustainable or optimal solution for the long term future. Whomever ends up as the principal would need to collaborate with the rest of humanity (or maybe just the ASI) to create the next intelligence that will be able to perform better.
Needless to say, above and beyond the power concentration, this is very dangerous. The person would need enough boldness to effectuate a pivotal action, but not so much drive that they end up accidentally destroy the world or any other such thing they could now easily do. Perhaps there is no such safe path, even with a human at the head.
https://www.lesswrong.com/posts/FBqe5dt8ZjaHN4Xj9/announcing-the-corrigibility-research-fund
https://www.lesswrong.com/posts/oLbpfPkdtcknABvvw/the-corrigibility-basin-of-attraction-is-a-misleading-gloss, also Jeremy Gillen apologies to any philosophers for his abuse of rule utilitarianism.
https://www.lesswrong.com/posts/CSFa9rvGNGAfCzBk6/problems-with-instruction-following-as-an-alignment-target
Do I need to cite this one? Read anything by Yudkowsky.
Can anyone point me to good writing on honest ai? I couldn't find much. Am I using the wrong term?