Clickbait: Normative value has high algorithmic complexity: there's no simple way to describe the goals we want AIs to want.

Introduction.

We can see Complexity of Value as being implied by three subpropositions:

The Intrinsic Complexity of Value proposition is that the properties we want AIs to achieve - whatever stands in for the metasyntactic variable 'value' - have a large amount of intrinsic complication in the sense of algorithmic complexity or Kolmogorov complexity. A very bad example that may nonetheless provide an important intuition is that by the time you're done telling an AI what constitutes 'worthwhile happiness' as opposed to e.g. a human with a wire stuck into their pleasure center or tiling the universe with tiny agents experiencing high rewards, you're probably looking at more like thousands or millions of bytes of data Inducted from a large labeled dataset, rather than a 20-byte definition that can be handcoded by the programmers.

The second key proposition is Fragility of Value which says that if you have a 10,000-byte exact specification of worthwhile happiness, and you begin to mutate it, the value created by the corresponding AI with the mutated definition falls off very rapidly. E.g. an AI with only 9,500 bytes of the full definition may end up creating 5%, 0%, or negative value rather than 95% of the value. (E.g., the AI understood all aspects of what makes for a life well-lived... except the part about requiring a conscious observer to experience it.)

The third key proposition is Incompressibility of Value which says that attempts to reduce these complex values into some incredibly simple and elegant principle fail (much like early attempts by e.g. Bentham to reduce all human value to pleasure) and that no simple instruction given an AI will happen to target outcomes of high value either. The core reason to expect a priori that all such attempts will fail, is that most 1000-byte strings aren't compressible down to some incredibly simple pattern no matter how many clever tricks you try to throw at them; at most 1 in 1024 such strings can be compressible to 990 bytes, never mind 20 bytes. Due to the tremendous number of different proposals for why some simple instruction to an AI should end up achieving high-value outcomes or why all human value can be reduced to some simple principle, there is no central absolute demonstration that all these proposals must fail, but there is a sense in which a priori we should strongly expect all such clever attempts to fail.

Together, these propositions imply that to achieve an adequate amount of value (e.g. 90% of potential value, or even 50% of potential value) there may be no simple handcoded rule for the AI that does the job.

Centrality.

Complexity of Value is an extremely central proposition in value alignment theory because many anticipated difficulties revolve around it:

• Complex values can't be hand-coded into an AI, and require value learning or Do What I Mean preference frameworks.
• Complex /fragile values may be hard to learn even by induction because the labeled data may not include distinctions that give all of the 10,000 bytes a chance to cast an unambiguous causal shadow into the data, and it's very bad if 50 bytes are left ambiguous.
• Complex / fragile values require error-recovery mechanisms because of the worry about getting some single subtle part wrong and this being catastrophic. (And since we're working inside of highly intelligent agents, the mechanism has to be a corrigible preference so that the agent goes along with it.)

More generally:

• Complex values tend to be implicated in patch-resistant problems that wouldn't exist if there was some obvious 5-line specification of exactly what to do, or not do.
• Complex values tend to be implicated in the change problems that wouldn't exist if we had a 5-line specification that solved those problems once and for all and that we'd likely run across during the development phase.

Importance.

There are many policy questions with strong dependency on Complexity of Value, mostly having to do with the overall difficulty of developing value-aligned AI, e.g.:

• Should we try to develop Genies or Sovereigns?
• How likely is a moderately safety-aware project to succeed?
• Should we be more worried about malicious actors creating AI, or about well-intentioned errors?
• How difficult is the total problem and how much should we be panicking?
• How attractive would be any genuinely credible game-changing alternative to AI?

To the extent one credits that Complex Value is probably true, one should arguably be very concerned about the number of early assessments of the value alignment problem that seem to rely on Complex Value being false (like just needing to hardcode a particular goal into the AI, or in general treating the value alignment problem as not panic-worthily difficult). It has been advocated that there are biases and mistakes leading to beliefs that directly or by implication deny Complex Value.

Truth condition

The Complexity of Value proposition is true if, relative to viable and acceptable real-world Methodologies for AI development, there isn't any reliably knowable way to specify the AI's [object-level preferences] as a structure of low complexity, such that the result of running that AI is achieving Enough of the possible value, for reasonable and humanly interpersonally persuasive definitions of value.

The caveats above are spelled out below.

Viable and acceptable computation.

Suppose there turns out to exist, in principle, a relatively simple Turing machine (e.g. 100 states) that picks out 'value' by re-running entire evolutionary histories, creating and discarding a hundred billion sapient races in order to pick out one that ended up relatively similar to human. This would both be using a very large amount of computing power and also committing an unacceptable amount of mindcrime.

Todo