There is a 'no-free-lunch' theorem in value learning; without assuming anything about an agent's rationality, you can't deduce anything about its reward, and vice versa.

Here I'll investigate whether you can deduce more if you start looking into the structure of the algorithm.

Algorithm (in)equivalence

To do this, we'll be violating the principle of algorithmic equivalence: that two algorithms with the same input-output maps should be considered the same algorithm. Here we'll instead be looking inside the algorithm, imagining that we have either the code, a box diagram, an FMRI scan of a brain, or something analogous.

To illustrate the idea, I'll consider a very simple model of the anchoring bias. An agent $H$ (the "Human") is given an object $X$ (in the original experiment, this could be wine, book, chocolates, keyboard, or trackball), an random integer $0 \leq n \leq 99$ , and is asked to output how much they would pay for it.

They will output $H (n, X) = \frac{3}{4} V (X) + \frac{1}{4} n$ , for some valuation subroutine $V$ that is independent of $n$ . This gives a quarter weight to the anchor $n$ .

Assume that $H$ tracks three facts about $X$ : the person's need for $X$ , the emotional valence the person feels at seeing it, and a comparison with objects with similar features. Call these three subroutines Need, Emo, and Sim. For simplicity, we'll assume each subroutine outputs a single number, that then gets averaged.

Now consider four models of $H$ as follows, with arrows showing the input-output flows:

I'd argue that a) and b) imply that the anchoring bias is a bias, c) is neutral, and d) implies (at least weakly) that the anchoring bias is not a bias.

How so? In a) and b), $n$ maps straight into Sim and Need. Since $n$ is random, it has no bearing on how much $X$ is needed, and on how valuable similar objects are. Therefore, it makes sense to see its contribution as noise or error.

In d), on the other hand, it is superficially plausible that a recently heard random input could have some emotional effect (if $n$ was not a number but a scream, we'd expect it to have an emotional impact). So if we wanted to argue that, actually, the anchoring bias is not a bias but that people actually derive pleasure from outputting numbers that are close to numbers they heard recently, then $n$ going into Emo would be the right place for it to go. Setup c) is not informative either wa... ,,,,,,,,,,,,,,,,,,,,,,

Human Values