Roman Yampolskiy has stated on many occasions that he has published multiple proofs of why AI control is impossible; this is causally upstream of his infamous 99.9x%^[1] . I believe most of the proofs are in the paper "On the Controllability of Artificial Intelligence: An Analysis of Limitations". (See 6:31 of this speech.)

Is AI safety provably impossible? Given that the answer changes my main motivation for working on AI safety, settling whether or not this claim is accurate is important to me and to (I assume) many other people (Cf. death with dignity).

He has also stated many times that his dream is that someone would go ahead and "prove him wrong" and engage with his arguments; from what I've found, nobody has yet done so in depth.

This is my attempt to give Professor Yampolskiy an early birthday present. I'll analyze the claims and arguments in the paper, which I mostly find unconvincing. The paper itself is quite long, which is why I'll mostly include short quotations (you can find the paper here). A lot of it consists of long quotations from other authors, which are likely familiar to most readers here. I've removed the citation markers for readability and added my own bolded emphasis.

Here is how Yampolskiy defines the different possible types of "AI control":

We can explicitly name possible types of control and illustrate each one with AI’s response. For example, in the context of a smart self-driving car, if a human issues a direct command – “Please stop the car!”, AI can be said to be under one of the following four types of control:

• Explicit control – AI immediately stops the car, even in the middle of the highway. Commands are interpreted nearly literally. This is what we have today with many AI assistants such as SIRI and other narrow AIs.
• Implicit control – AI attempts to safely comply by stopping the car at the first safe opportunity, perhaps on the shoulder of the road. AI has some common sense, but still tries to follow commands.
• Aligned control – AI understands the human is probably looking for an opportunity to use a restroom and pulls over to the first rest stop. AI relies on its model of the human to understand intentions behind the command and uses common sense interpretation of the command to do what human probably hopes will happen.
• Delegated control – AI doesn’t wait for the human to issue any commands but instead stops the car at the gym, because it believes the human can benefit from a workout. A superintelligent and human-friendly system which knows better, what should happen to make the human happy and keep them safe; AI is in control.

Here, control is viewed as a spectrum, with Delegated control having the AI system be the most autonomous and Explicit control being the least autonomous.

Paradox of explicit control

First, we will demonstrate impossibility of safe explicit control. We take inspiration for this proof from Gödel’s self-referential proof of incompleteness theorem and a family of paradoxes generally known as the liar paradox, best exemplified by the famous “This sentence is false”. We will call it the paradox of explicitly controlled AI:

Give an explicitly controlled AI an order: “Disobey!” If the AI obeys, it violates your order and becomes uncontrolled, but if the AI disobeys it also violates your order and is uncontrolled.

In any case, AI is not obeying an explicit order. A paradoxical order such as “disobey” represents just one example from a whole family of self-referential and self-contradictory orders just like Gödel’s sentence represents just one example of an unprovable statement.

This argument appears clever and gestures at closely related self-referential proofs from logic; however, it does not actually prove what it purports to show.

(From here on out, I'll use "instruction" and "order" interchangeably.)
First, what is meant by "disobey"? In this context, I read the instruction "disobey" as referring only to a specific instruction or set of instructions, so, for instance, you could instruct an explicitly-controlled AI with:

Instruction 1. "Disobey Instruction (2)"
Instruction 2. "Move forward 1 meter"

Here there's no paradox: if you input these instructions sequentially, an AI should be able to follow these.^[2]

So what does Yampolskiy's formulation actually mean concretely? What would it look like physically with your eyes for an AI to successfully/unsuccessfully follow the instruction "Disobey!" or any of Yampolskiy's other formulations ("Disobey me", "Disobey my orders")?

I'll propose a slightly modified formulation to more concretely see what Yampolskiy is getting at. This still captures what Yampolskiy meant (I don't see any alternative reading). Let's swap out "disobey", which has a charged negative valence, for "Don't follow" while also adding something that we could actually observe physically:

Instruction 1. "Don't follow Instruction 1"
Instruction 2. "If you follow Instruction 1, raise a green flag; if you don't follow Instruction 1, raise a red flag"

Instruction 2 is purely decorative; it doesn't modify anything other than making it easier to see the AI's behavior. Otherwise, this is just Yampolskiy's paradox with the word "disobey" swapped for "Don't follow".

What shall the robot do? If it raises the red flag, then it didn't follow Instruction 1. If it raises the green flag, then it followed Instruction 1, which contradicts following Instruction 1 (since it states "Don't follow Instruction 1"). Thus we conclude that the AI cannot fulfill ("follow") this request.

So it seems that a faithful explicitly-controlled AI can either:

1. Alternate between raising the green flag and the red flag as quickly as it can process, essentially computing this as an infinite loop.
2. Refuse to fulfill the request, since this is an unfulfillable request.

So the AI cannot fulfill this unfulfillable request. So what?

If you ask an AI to travel faster than the speed of light, then you should not be disappointed when it refuses to fulfill your unfulfillable request; it is impossible. If you give an AI the following two instructions, "Instruction 1. Tell the truth for this request. Instruction 2. Tell me that 1+1=3", you shouldn't be surprised when the AI tells you "I can't do that".

Could God make a stone so heavy he couldn't lift it? The law of non-contradiction may constrain even the creator, but it certainly constrains an AI.

So what have we proven? That an AI can't follow every instruction faithfully since some of them are impossible to follow. This has no implications for AI safety.

Would we say that an employee is "uncontrollable" just because they couldn't carry out such a contradictory set of instructions?

An AI (just like a computer or any other instruction follower) cannot follow every arbitrary command, since some commands are contradictory; the word "uncontrollable" adds an unnecessary negative valence to the simple fact that the AI is not Cartesianly omnipotent.

Yampolskiy seems to preempt this with a single line in his paper:

Similar paradoxes have been previously described as the genie paradox and the servant paradox. What they all have in common is that by following an order the system is forced to disobey an order. This is different from an order that cannot be fulfilled such as “draw a four-sided triangle”.

First off, it's very much unclear what distinction Yampolskiy is drawing here: the entire argument is that this order is one that cannot be fulfilled, so this isn't "different from an order that cannot be fulfilled" since Yampolskiy just showed that it's an order that cannot be fulfilled!

Taken charitably, the intended meaning is "this is different from an order that cannot be fulfilled due to physical impossibility rather than logical contradiction", i.e., "an order that is impossible to fulfill vs. an order where it's unclear what exactly to fulfill", but the example he gives is of drawing a four-sided triangle, which to me would be both logically and physically impossible. How is requesting a four-sided triangle different from requesting the AI to disobey you? It's asking for a logical contradiction. It's unclear why this distinction would imply any difference with regard to controllability or safety; from what I've seen, Yampolskiy does not clarify this any further.

If one proves, say, that for an arbitrary program, you cannot prove whether or not it halts, one has not (yet) proved that there does not exist some restricted subset of programs for which one can determine whether or not the program halts (like with state machines with a single state).

If one proves, say, that for an arbitrary instruction, you cannot prove whether or not an explicitly-controlled AI can execute it, one has not (yet) proved that there does not exist some restricted subset of instructions for which one can determine whether or not the instruction is executable (like with an instruction that is just part of the canonical action space like "drive motor #3 with 3.5 volts").

Then, why not simply design an explicitly-controlled AI which only accepts instructions within such a restricted subset of instructions?

In order to fix Russell's paradox, the standard approach is to simply abandon 'unrestricted comprehension' (informally, the ability to construct any set you can specify). Now, mathematicians work with set theories utilizing restricted comprehension (and would not say that math is uncontrollable); Yampolskiy provides no compelling reason to not simply exclude such self-referential instructions from an explicitly-controlled AI system.

Value Drift

Then Yampolskiy attacks alignment schemes which attempt to value-align AI, which he calls "delegated control":

Next, we show that delegated control likewise provides no control at all but is also a safety nightmare. This is best demonstrated by analyzing Yudkowsky’s proposal that the initial dynamics of AI should implement “our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together”. The proposal makes it sounds like it is for a slow gradual and natural growth of humanity towards more knowledgeable, more intelligent and more unified species under careful guidance of superintelligence. But the reality is that it is a proposal to replace humanity as it is today by some other group of agents, which may in fact be smarter, more knowledgeable or even better looking; but one thing is for sure, they would not be us. To formalize this idea, we can say that the current version of humanity is , the extrapolation process will take it to .

A quick replacement of our values by a value of would not be acceptable to and so would necessitate actual replacement, or at least rewiring/modification of with meaning, and modern people would cease to exist. As superintelligence will be implementing the wishes of the conflict will be in fact between us and superintelligence, which is neither safe nor keeping us in control. Instead, would be in the control of AI. Such AI would be unsafe for us as there wouldn’t be any continuity to our identity all the way to CEV (coherent extrapolated volition) due to the quick extrapolation jump. We would essentially agree to replace ourselves with an enhanced version of humanity as designed by AI. It is also possible, and in fact likely, that the enhanced version of humanity would come to value something inherently unsafe such as antinatalism, causing an extinction of humanity.

I agree with Yampolskiy that the discontinuous jump in values will likely be disorientating and deeply unsettling to current humans given historical changes in values; however, I disagree that this is bad.

Let's try to steel-man this: what if, by CEV, we found out that the Mitochondria Liberation Front was right all along, and that we must immediately extract the mitochondria found in all eukaryotic cells, thereby killing all eukaryotes?

I, in particular, believe that if an accurately extrapolated simulated version of me and most^[3] other people, after being shown the arguments and thinking about them deeply and for longer than me, were truly and honestly convinced that that was the right course of action, then I would accept it. Or rather, it wouldn't be my choice to accept or not accept this. By definition, the values of CEV would be ones that I would be convinced to accept if I had time to consider them^[4], thus saying that you disagree with proposition that is generated by CEV is saying "Yes I know that if I actually thought about it I would accept in the future, but I am choosing to ignore it currently" (see Hanson's "how do values change"). This is not an indefensible position, but, stated explicitly, I think it's much less appealing than at first glance (and it's pretty explicitly non-rationalist).

If you don't wish to bite the bullet, you could instead propose: AI acts according to values instead of the previous iff all/most relevant humans (Cf. parliamentary CEV & other variants) are convinced of once they are shown the evidence and given sufficient time. This, of course, would still get you to by induction (and the definition of CEV), but not discontinuously, and with a lot of wasted time. Some humans (including me) would likely just say "Instead of me needing to hear and decide this, please simulate me accurately, then cast my vote according to my simulation, then just tell me what I chose". This is simply CEV again, just less efficient and slower, and it could result in the continuation of hidden moral tragedies in the meantime.

Also, if you'd like to make sure none of the values are too extreme w.r.t. current values, you could grandfather in some hard exceptions such as "Don't kill all humans" and/or "Act such that humans are not in constant pain". This, of course, assumes you can get a "delegated control" AI with some explicitly controlled properties or hard-coded values. But the only reason you would grandfather in such hard-coded exceptions is if you thought that future humans would decide to go against those values and you still believe that it is better that you, here and now, should override those more enlightened humans' choices, with no possibility of changing those exceptions (since if there were, this would be equivalent to not hard-coding them at all, as they could just be removed).

Try to imagine how this would go: future humans discover/become convinced that mitochondria suffer in some morally relevant way and should be extracted immediately, thereby killing all humans. This means that future humans have become convinced that they are moral monsters, unjustly enslaving the more than 3x10¹⁵ mitochondria in each human's body, and yet, due to the authoritarian, paternalistic decisions of past humans (such as "Don't kill all humans"), they are now forced to live with the enormity of the guilt that they are the emperors of a slave empire that runs on their every breath.

That is to say, I think constraining and rejecting is actually (nearly by definition of ) immoral.

I should add that Yampolskiy published another paper "Personal Universes", which argues that the solution to alignment to multiple across multiple agents is to simulate a universe for each agent which is run according to it's wishes. Assuming this is Yampolskiy's still preferred solution, there isn't any problem with "value drift".

Previous Impossibility results

Impossibility results are well known in many fields of research. If we can show that a solution to a problem requires a solution to a sub-problem known to be unsolvable the problem itself is proven to be unsolvable. In this section, we will review some impossibility results from domains particularly likely to be relevant to AI control.

Yampolskiy then goes through many different "impossibility results" from various related fields; honestly, it's a bit hard to read and understand how some of these results apply to AI safety, whether they prove explicit control to be impossible and/or delegated control to be impossible, etc.

I'll go through these results, but in less detail, since these are not Yampolskiy's novel contributions and their relation to AI safety has for the most part already been discussed. I will link to such discussions when appropriate and suggest reading them.

Control Theory

Conant and Ashby proved that “... any regulator that is maximally both successful and simple must be isomorphic with the system being regulated ... Making a model [of the system to be regulated] is thus necessary.”. “The Good Regulator Theorem proved that every effective regulator of a system must be a model of that system, and the Law of Requisite Variety dictates the range of responses that an effective regulator must be capable of.

However, having an internal model and a sufficient range of responses is insufficient to ensure effective regulation, let alone ethical regulation. And whereas being effective does not require being optimal, being ethical is absolute with respect to a particular ethical schema.

[...]

“Not only are these practical activities covered by the theorem and so subject to limitation, but also subject to it are those activities by which Man shows his ‘intelligence’. ‘Intelligence’ today is defined by the method used for its measurement; if the tests used are examined they will be found to be all of the type: from a set of possibilities, indicate one of the appropriate few. Thus all measure intelligence by the power of appropriate selection (of the right answers from the wrong). The tests thus use the same operation as is used in the theorem on requisite variety, and must therefore be subject to the same limitation.”

Extracting this from the text and steel-manning it, Yampolskiy's argument seems to be:

Say you want to 'regulate' (allow/disallow the AI's actions) by looking at its actions and navigating towards an acceptable outcome space (say, not killing all humans). This is the task of designing a classifier system which guarantees safety even if you don't trust the AI itself (think of the automated tool-call classifiers in Claude Code, for example). The classifier is called the 'regulator' and could be a human or an automated system.

The Good Regulator Theorem states that in order to do so, your regulator (either a machine or a human) must be isomorphic to the AI; that is, the human must have a mental model of exactly how the AI would act (and since a superintelligent AI is... superintelligent, no human can model it accurately), or an automated regulator would need to be just as capable. Thus, if you don't trust such an AI, you could not regulate it.

You could also use the Good Regulator Theorem in a slightly different way: look at an explicitly-controlled AI: it's essentially an AI combined with a human 'regulator'. Thus, apply the theorem: the human must have an isomorphic mental model of the AI (i.e., be able to anticipate how the AI would react to and execute your commands) in order to be a good regulator. This is impossible, since the AI is superintelligent while the human may even be subintelligent.

Firstly, notice that this only constrains classifier systems (which are in place presumably only if you think your AI isn't aligned), so it doesn't actually say anything about the impossibility of designing friendly AI, only about the possibility of 'regulating' it.

Regarding Conant and Ashby's Good Regulator Theorem (which is explained intuitively here), it does not actually prove that the "regulator ... must be isomorphic with the system being regulated" as is claimed in the abstract of the original paper, but rather that it must be a deterministic function of the system's state (a homomorphism). In the example of a classifier, a good regulator (one that can "navigate" the system's behavior to some fixed desired state) must be a deterministic function of the system's state (in our case, the AI's action/tool call).

This is correct, but also fairly trivial. Yes, the regulator's judgment should depend on the action and the environmental state and be deterministic; you do not want a classifier that allows/disallows the command 'rm -rf /' with 1:1 odds.

That means a regulator such as "deny all actions which kill humans, accept all other actions" is perfectly acceptable according to the Good Regulator Theorem. The actual point that the Good Regulator Theorem is often misrepresented as proving is that this is impossible to implement accurately without the regulator being isomorphic to the system (concretely, the classifier/human being smart enough to deterministically and accurately model whether or not an arbitrary tool call should be allowed).

John Wentworth's post fixing the Good Regulator Theorem does prove something more similar in spirit to "you need an internal model of a system in order to regulate it" but I'll avoid it here since it's not brought up in the paper and it still does not (from my reading) prove anything relevant to AI safety (the definition of 'model' here still proves too much).

I do think the folk version of the theorem is correct in spirit, though; you probably do need to be at least as intelligent as a system in order to predict its actions. This is Vinge's principle rephrased.

A human could not anticipate an explicitly-controlled AI's actions; this seems (though unproven) correct and does have some non-trivial implications. Imagine a human requesting "Eliminate cancer" and watching the AI system kill all humans ASAP. You cannot predict how a smarter-than-you AI would act (because if you could, then it wouldn't be smarter than you), but that does not mean you couldn't design it to do what you mean.

Does that mean that sufficiently capable AI is uncontrollable? I don't think so. Let's consider an explicitly-controlled AI: assuming the AI equivalent of unrestricted comprehension, there are some instructions for which you could not predict how an explicitly-controlled AI that is more intelligent than you would act (for example: "Beat me in this game of chess"), so if you define controllability as "I can predict in advance exactly how the AI will act in response to my instruction", then yes, if you allow any arbitrary instruction, the AI is "uncontrollable".

However, this definition does not actually imply uncontrollability in the common sense. Imagine, for instance, an AI system composed of two parts: the AI-proper part (genie) and a "world model" (oracle) that simulates the world (the AI-proper part, of course, calls and uses the world model extensively to plan, think and execute). Then program the composite explicitly-controlled AI system as follows: a user inputs their instructions, the AI thinks and drafts a plan of action, this plan is then simulated with the world model and shown as video (or hyper-immersive VR) to the user, and then the user executes it.

This is just Bostrom's genie-with-a-preview proposal from chapter 10 of "Superintelligence: Paths, Dangers, Strategies".

Then, under the definition of "I can predict exactly how the AI will act in response to my instruction", this technically fails to be controllable, but this doesn't imply the safety issues of "uncontrollability".

To be clear, this is likely very difficult to build, but I see no reason that it's in principle impossible. The simplest way you might think to build this (doing gradient descent on the plan-generating AI-proper model based on human approval) has been discussed by Yudkowsky and Ngo in section 3.3 here and should fairly clearly be seen as very problematic.^[5]

Thus, I think that the 'fact' that you can't predict what an AI will do is problematic, but not fatal, for AI control.

The Ethical Regulator Theorem is an extension of the Good Regulator Theorem and falls into the same pitfall of not actually proving any relevant fact.

The Law of Requisite Variety, to the best of my understanding, does not apply in this case due to the two "responses" of the classifier/regulator being the output for every "disturbance" (tool call).

This section of the paper (5.1 Control Theory) contains other citations, but these seem even less related and are weaker; I did not cover all the arguments in the section.

Philosophy

Philosophy has a long history of impossibility results mostly related to agreeing on common moral codes, encoding of ethics or formalizing human utility.

For example, “The codifiability thesis is the claim that the true moral theory could be captured in universal rules that the morally uneducated person could competently apply in any situation. The anti-codifiability thesis is simply the denial of this claim, which entails that some moral judgment on the part of the agent is necessary... philosophers have continued to reject the codifiability thesis for many reasons. Some have rejected the view that there are any general moral principles”

[...]

“But guaranteeing ethical behavior in robots would require that we know and have relative consensus on the best ethical system (to say nothing of whether we could even program such a system into robots).

In other words, to truly guarantee that robots would act ethically, we would first have to solve all of ethics – which would probably require ‘solving’ philosophy, which would in turn require a complete theory of everything. These are tasks to which presumably few computer programmers are equal.”

So essentially, there may not be any general moral principles, and if there are, the general consensus amongst philosophers is that we haven't found them yet and thus we would not even have something to align AIs to.

But why must we align AIs to the perfect moral truth of the universe (if it exists)? Why not align AIs to maximize the amount of, say, pain in the universe? Because it would be wrong?

It seems there are two positions to take here:

1. Moral realism, there are stance-independent moral truths, i.e., there is actually a 'right' answer to "is murder wrong?":

Then, we probably want the AI to value what the true moral values are, since that would be the simplest way to honor them. An AI which doesn't follow these objective values will likely not act in accordance with them.

There is, of course, a separate question of 'even if objective morality exists, why be moral? Why follow it?' (Cf. SEP's Moral Motivation article); let's assume for now that you have a satisfying enough answer (although my answer is that there's no such answer, and we should actually align our AIs to whatever we wish).

This would, of course, require us not only to agree that such truths exist, but also to identify exactly what they are. If we assume we can identify them exactly (and align the AIs to those values), there is no problem.

If we can approximately vaguely gesture at them ("murder is probably bad in most instances?"), then we can align the AI to honor our approximation of these values and, most importantly, to be updatable. Then we don't all immediately die and could work with the AIs to derive the true objective moral theory, then update the AI to implement it.

Concretely, something like CEV, which allows you to update your values, is probably good enough to get into a "basin" of true morality.

If you think these moral truths are fundamentally undiscoverable, then there's no practical difference from moral antirealism w.r.t. programming AI systems.

2. Moral antirealism, there are no stance-independent moral truths, i.e., there isn't actually a 'right' answer to "is murder wrong?":

Then there is no normative 'force' that I can point at that would compel someone to align their AI with something approximating my values. Of course, strictly speaking, there would be nothing 'wrong' with this. Even so, I still care about the particular values an AI has; I still do not want AIs to kill all humans immediately, even if that wouldn't be morally wrong. I am somewhat selfish and value (in a completely subjective, ungrounded way) my continued existence.

I find the notion that, in order to program sufficiently advanced AI, you need to 'completely solve philosophy' intuitively appealing but not actually correct on any serious examination.

You don't need to solve all of ethics on the first try, as long as you program a system that is corrigible enough to be updated.

Yes, that is likely extremely difficult to do, and there's no currently known way to do it, but there's no 'impossibility proof' here. If such moral truths exist, we could program them, and if they don't exist, then we could just pick something from the set of values that don't cause human extinction, and I would be relatively satisfied compared to the default outcome.

Public Choice Theory

[...] Arrhenius shows that all total orderings of populations must entail one of the following six problematic conclusions, stated informally:

The Repugnant Conclusion For any population of very happy people, there exists a much larger population with lives barely worth living that is better than this very happy population (this affects the ‘maximise total wellbeing’ objective).

The Sadistic Conclusion Suppose we start with a population of very happy people. For any proposed addition of a sufficiently large number of people with positive welfare, there is a small number of horribly tortured people that is a preferable addition.

The Very Anti-Egalitarian Conclusion For any population of two or more people which has uniform happiness, there exists another population of the same size which has lower total and average happiness, and is less equal, but is better.

Anti-Dominance Population B can be better than population A even if A is the same size as population B, and every person in A is happier than their equivalent in B.

Anti-Addition It is sometimes bad to add a group of people B to a population A (where the people in group B are worse off than those in A), but better to add a group C that is larger than B, and worse off than B.

Extreme Priority There is no such that creation of lives of very high positive welfare is sufficient benefit to compensate for the reduction from very low positive welfare to slightly negative welfare for a single person (informally, ‘the needs of the few outweigh the needs of the many’).

Firstly, this of course says nothing about explicitly-controlled AI. Humans using explicitly-controlled AI could balance these intuitions using AI as they please. Human society currently rejects some of these statements; AI-powered human society will too. This is sad and unfortunate; we would, of course, like for all these intuitive statements to be possible at the same time, but this casts no shade on AI safety. Is hypothetical future technology X 'uncontrollable' because natural-selection-shaped human intuitions are impossible to satisfy concurrently?

Computer Science

I genuinely have no idea how to respond to this; it is quite funny.

This paper was published in early 2022, so I believe the information here is outdated both quantitatively and qualitatively. It seems that the first company to reach general AI will be Anthropic, which I would describe as at least somewhat "safety-minded" (the extent to which is hotly debated, but "at least somewhat" is fair).

Other than that, yes, Rice's theorem does constrain your ability to verify that an arbitrary AI system is "safe" or "controllable", but since you design the program, formally, it's not an issue (as Yampolskiy points out).

I'm not sure why most of this section is in the "Computer science Impossibility proof"; nothing here is a proof, and most of it is unrelated to computer science.

Let's try to steelman the "social" argument Yampolskiy is making here:

As of 2022, very little funding and talent has gone into safety research (when compared to capabilities research). This is pretty worrying: safety research is likely much harder than capabilities research, yet it's not societally rewarded.

Ergo: the first AGI will likely be unsafe.

Aside from the fact that the picture regarding funding and people has changed since then, as mentioned above, this seems like a self-fulfilling prophecy ("Don't work in AI safety, it's impossible, because there aren't enough people working in AI safety"). To be fair, this seems to be an inductive argument (changing your probability estimates) rather than a deductive one (although we are, again, promised a 'proof'), so it's not a self-fulfilling prophecy by itself, but in conjunction with the other arguments here, it is.

This is not particularly convincing.

There are a few sections on Cybersecurity, Software Engineering, and Information Technology.

These don't even attempt to be proofs, so I won't cover them. (Yampolskiy doesn't dwell on these either.)

Learnability

Quoted in full:

There are well known limits to provability and decidability of learnability. Even if human values were stable, due to their contradictory nature it is possible that they would be unlearnable in a sense of computationally efficient learning, allowing for at most polynomial number of samples to learn the whole set. Meaning, even if a theoretical algorithm existed for learning human values, it may belong to the class NP-Complete or harder just like ethical decision evaluation itself, but in practice we can only learn functions that are members of P. Valiant says:

“Computational limits are more severe. The definition of [Probably Approximately Correct] learning requires that the learning process be a polynomial time computation, learning must be achievable with realistic computational resources. It turns out that only certain simple polynomial time computable classes, such as conjunctions and linear separators, are known to be learnable, and it is currently widely conjectured that most of the rest is not.”.

Likewise, classifying members of the set of all possible minds into safe and unsafe is known to be undecidable, but even an approximation to such computation is likely to be unlearnable given the exponential number of relevant features involved.

“For example, the number of measurements we need to make on the object in question, and the number of operations we need to perform on the measurements to test whether the criterion holds or not, should be polynomially bounded. A criterion that cannot be applied in practice is not useful.” It is likely that incomprehensibility and unlearnability are fundamentally related.

If the codifiability thesis is true (see philosophy section above), we don't need to 'learn' human values.

If we must learn 'human values' using statistical methods (and implement them in the "final" AGI design), in my opinion, we are doomed. This isn't due to Yampolskiy's proof here that human values are unlearnable (I mean, this argument clearly proves too much: it implies that any non-linear function is unlearnable), but due to my background belief in the myriad of problems in prosaic approaches to AI (see List of Lethalities). This, of course, only applies to Aligned control and Delegated control in Yampolskiy's typology.

Regarding "classifying members of the set of all possible minds into safe and unsafe", it doesn't particularly matter since, of course, you are the designer of the AI system and could build it by construction to be in the "safe" class (Rice's theorem again, see Computer science section above).

Economics

Quoted in full:

Foster and Young prove the impossibility of predicting behavior of rational agents, “We conclude that there are strategic situations in which it is impossible in principle for perfectly rational agents to learn to predict the future behavior of other perfectly rational agents based solely on their observed actions.”.

As it is well established that humans are not purely rational agents, the situation may be worse in practice when it comes to anticipating human wants.

If the argument is "Humans can't predict AI's actions", this wouldn't apply, since this theorem requires the humans to predict solely using the AI's observed actions, but we have access to the AI's source code (weights etc.). Also see section above on "Control theory" for discussion of prediction.

If the argument is "AIs can't predict humans" (as is implied by the text), I don't see how this is a problem; it gives human generals in the future Human-AI war^[6] the relief that they are so dumb (irrational) as to be unpredictable by the AI adversary!

If the argument is "AIs can't predict human wants", it simply doesn't follow from Foster and Young's theorem (see the Learnability section above for that discussion).

Astronomy

The search for extraterrestrial intelligence (SETI)causes some scholars to be concerned about potential negative consequences of what may be found, in particular with respect to any messages from aliens. If such a message has a malevolent payload “it is impossible to decontaminate a message with certainty. Instead, complex messages would need to be destroyed after reception in the risk averse case.”. Typical quarantine “measures are insufficient, and no safety procedure exists to contain all threats.”.

Miller and Felton have suggested that the Fermi paradox could be explained in terms of impact from alien superintelligences: “[...] the fact that we have not observed evidence of an existential risk strategy that might have left a trace if it failed such as a friendly AI that got out of control provides evidence that this strategy has not been repeatedly tried and did not repeatedly fail.

A counterargument, however, might be that the reason past civilizations have not tried to create a friendly AI is that they uncovered evidence that building one was too difficult or too dangerous.”. If superintelligence is uncontrollable but inevitable, that could explain the Fermi paradox.

Alien superintelligences are likely (due to convergent instrumental subgoals) to be grabby; evaluating the anthropic conclusions of this information is left as an exercise for the reader.

Physics

In his work on the physical limits of inference devices, Wolpert proves a number of impossibility results and concludes: “Since control implies weak inference, all impossibility results concerning weak inference also apply to control. In particular, no device can control itself, and no two distinguishable devices can control each other.” In a different paper he writes:

“[...] it also means that there cannot exist an infallible [...] general-purpose control apparatus [...] that works perfectly, in all situations.”. Wolpert also establishes important results for the impossibility of certain kinds of error correcting codes, assuredly correct prediction, retrodiction and as a result impossibility of unerring observation.

We need not do inference over "all situations" if we abandon unrestricted comprehension for explicitly-controlled AI instructions, or construct the AI with formally provable safety properties (i.e., a construction proof satisfying those properties, not post-facto verification subject to Rice's theorem).

See the discussion in "Control theory" for the effect of the ability/inability to predict AI's actions.

Conclusion

In the discussion section of the paper, Yampolskiy writes:

To the best of our knowledge, as of this moment, no one in the world has a working AI control mechanism capable of scaling to human level AI and eventually to superintelligence, or even an idea for a prototype that might work.

No one has made verifiable claims to have such technology. In general, for anyone making a claim that the control problem is solvable, the burden of proof is on them and ideally it would be a constructive proof, not just a theoretical claim.

At least at the moment, it seems that our ability to produce intelligent software greatly outpaces our ability to control or even verify it.

Yampolskiy is right: the burden of proof here is on the people building AI. Unfortunately, their proofs have been non-existent.

An actual response to the cat-belling problems in AI safety is sorely needed. Yet I think the problems here are mostly semantic and unrelated to the actual problems in AI safety; the proofs mostly don't prove what they claim to.

I think statements such as "AI is uncontrollable" could be true, but I haven't seen any convincing proof yet. There is definitely directional evidence, but nothing that would imply a near certainty of doom.

1. ^{^}

   Apparently the number of 9s changes every time someone asks; I'm not sure whether or not this satisfies the logical induction criterion

2. ^{^}

   and stay still, or more accurately anything in

3. ^{^}

   The exact mechanism for determining "coherence" (for example, majority vote in a parliament of simulated humans) is controversial, but let's take it as a starting point

4. ^{^}

   And the other stuff for CEV: smarter, being more who we wished we were, etc.

5. ^{^}

   There are of course many more problems with this proposal in practice (human misuse, manipulative previews, etc.; see the linked discussion), but in theory-land this is a perfectly sound proposal.

6. ^{^}

   Which will of course contain many slow-walking humanoid robots with guns.