(Re)Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin’s note: In the last post, we saw that a good broad value learning approach would need to understand the systematic biases in human planning in order to achieve superhuman performance. Perhaps we can just use machine learning again and learn the biases and reward simultaneously? This post by Stuart Armstrong (original here) and the associated paper say: “Not without more assumptions.”

This post comes from a theoretical perspective that may be alien to ML researchers; in particular, it makes an argument that simplicity priors do not solve the problem pointed out here, where simplicity is based on Kolmogorov complexity (which is an instantiation of the Minimum Description Length principle). The analog in machine learning would be an argument that regularization would not work. The proof used is specific to Kolmogorov complexity and does not clearly generalize to arbitrary regularization techniques; however, I view the argument as being suggestive that regularization techniques would also be insufficient to address the problems raised here.

Humans have no values… nor do any agent. Unless you make strong assumptions about their rationality. And depending on those assumptions, you get humans to have any values.

An agent with no clear preferences

There are three buttons in this world, $B (0)$ , $B (1)$ , and $X$ , and one agent $H$ .

$B (0)$ and $B (1)$ can be operated by $H$ , while $X$ can be operated by an outside observer. $H$ will initially press button $B (0)$ ; if ever $X$ is pressed, the agent will switch to pressing $B (1)$ . If $X$ is pressed again, the agent will switch back to pressing $B (0)$ , and so on. After a large number of turns $N$ , $H$ will shut off. That’s the full algorithm for $H$ .

So the question is, what are the values/preferences/rewards of $H$ ? There are three natural reward functions that are plausible:

$R (0)$ , which is linear in the number of times $B (0)$ is pressed.
$R (1)$ , which is linear in the number of times $B (1)$ is pressed.
$R (2) = I (E, X) R (0) + I (O, X) R (1)$ , where $I (E, X)$ is the indicator function for $X$ being pressed an even number of times, $I (O, X) = 1 - I (E, X)$ being the indicator function for $X$ being pressed an odd number of times.

For $R (0)$ , we can interpret $H$ as an $R (0)$ maximising agent which $X$ overrides. For $R (1)$ , we can interpret $H$ as an $R (1)$ maximising agent which $X$ releases from constraints. And $R (2)$ is the “ $H$ is always ...

Kolmogorov Complexity

See also

Kolmogorov Complexity

See also