During NeurIPS, I had a discussion with Dylan Hadfield-Menell about how to fomalise my methods for extracting human values.

He brought up the issue of how to figure out the goals of a hierarchical system - for example, a simplified corporation. It's well known that not everyone in a corporation acts in the best interests of the corporation. So, a corporation will not act at all like a rational shareholder value maximiser. Despite this, is there a way of somehow deducing, from its behaviour and its internal structure, that its true goals are shareholder maximisation?

I immediately started taking the example too literally, and objected to the premise, mentioning the goals of subagents within the corporation. We were to a large extent talking past each other; I was strongly aware of the fact that even if a system had its origin in one set of preferences, that didn't mean it couldn't have disparate preferences of its own. Dylan, on the other hand, was interested in formalising my ideas in a simple setting. This is especially useful, as the brain itself is a hierarchical system with subagents.

So here is my attempt to fomalise the problem, and present my objection, all at the same time.

The setting and actions

I'm choosing a very simplified version of the hierarchical planning problem presented in this paper.

A robot, in green, is tasked with moving the object A to the empty bottom storage area. The top level of the hierarchical plan formulates the plan of gripping object A.

This plan is then passed down to a lower level of the hierarchy, that realises that first it must get B out of the way. This is passed to a lower level that is tasked with gripping B. It does so, but, for some reason, it first turns on the music:

Then it moves B out of the way (this will involve moving up and down the planning hierarchy a few times, by first gripping B, then moving it, then releasing the grip, etc...):

Then the planning moves up the hierarchy again, which re-issues the "grip A" command. The robot does that, and eventually moves A into storage:

The irrationality of music

Now this seems to be mainly a rational plan for moving A to storage, except for the music part. Is the system trying to move A but is being irrational at one action? Or does the music reflect some sort of preferences for the hierarchical system?

Well, to sort that out, we can look within the agent. Let's look at the level of the hierarchy that is tasked with gripping B, and that turns on the music. In a massively pseudo version of pseudo code, here is one version of this:

$\begin{array}{cc}& \text{Algorithm 1:}\\ \text{1:}& \text{State: Gripped(B)=False}\\ \text{2:}& \text{Goal: G={Gripped(B)=True}}\\ \text{3:}& \text{Suggested plans:}\\ \text{4:}& \text{Radio\_On => Gripped(B)=True}\\ \text{5:}& \text{Grip(B) => Gripped(B)=True}\\ \text{6:}& \text{While G=False:}\\ \text{7:}& \text{Run Suggested plans according to criteria C(G)}\end{array}$

The criteria $\text{C(G)}$ just looks at the plans and gauges whether they will move the state closer to $\text{G}$.

In this situation, it's clear that the algorithm is trying to grip B, but has some delusion that turning the radio on will help it do so (maybe in a previous experiment the vibrations shook B loose?). The algorithm will first try the radio plan, and, when it fails, will then try the second plan and grip B the conventional way.

Contrast that with this code:

$\begin{array}{cc}& \text{Algorithm 2:}\\ \text{1:}& \text{State: Gripped(B)=False AND Enjoyment=False}\\ \text{2:}& \text{Goal: G={Gripped(B)=True AND Enjoyment=True}}\\ \text{3:}& \text{Suggested plans:}\\ \text{4:}& \text{Radio\_On => Enjoyment=True}\\ \text{5:}& \text{Grip(B) => Gripped(B)=True}\\ \text{6:}& \text{While G=False:}\\ \text{7:}& \text{Run Suggested plans according to criteria C(G)}\end{array}$

Now, there is the usual problem with calling a variable "$\text{Enjoyment}$", and that's the reason I've been looking into syntax and semantics.

But even without solving these issues, it's clear that this part of the system has inherited a different goal than the rest of the hierarchical system. It's a subagent with non-aligned preferences. Turning on the music is not an accident, it's a deliberate and rational move on the part of this subagent.

The goals of the system

For Algorithm 1, the system clearly has the goal of of moving A to storage, and has a failure of rationality along the way.

For Algorithm 2, there are several possible goals: moving A to storage, turning the radio on, and various mixes of these goals. Which is the right interpretation of the goal? We can model the hierarchy in different ways, that suggest different intuitions for the "true" goal - for example, Algorithm 2 could actually be a human, acting on instructions from a manager. Or it could be a sub-module of a brain, one that is unendorsed by the rest of the brain.

My take on it is that the "true goal" of the system is underspecified. From the outside, my judgement would be that the closer the subagent resembles an independent being, and the closer that $\text{Enjoyment}$ resembles true enjoyment.

From the inside, though, the system may have many meta-preferences that push it in one direction or another. For example, if there was a module that analysed the performance of the rest of the algorithm, and recommended changes; and if that algorithm systematically changed the system towards becoming a A-storage-mover (such as recommending deleting the $\text{Enjoyment}$ goal), then that would be a strong meta-preference towards only having the storage goal. And if we used the methods here, we'd get moving A the main goal, plus some noise.

Conversely, maybe there are processes that are ok with the subagent's behaviour. A modern corporation, for instance, has legal limits on what it can inflict on its workers, so some of the meta-preferences of the corporation (located for example in HR and the legal department) are geared towards some livable work environments, even if this doesn't increase shareholder value.

(Now, I still might like to intervene, in some cases - even if the goal of a slave state invading their neighbours is clear at the object and meta-level, I can still value subagents even if they don't).

In the absence of some sort of meta-preferences, there are multiple ways of establishing the preferences of a hierarchical system, and many of them are equally valid.