tl;dr: if an agent has a biased learning process, it may choose actions that are worse (with certainty) for every possible reward function it could be learning.

An agent learns its own reward function if there is a set $R$ of possible reward functions, and there is a learning process $P$ that maps world-histories (and policies) to distributions over $R$ . Thus by interacting with the environment and choosing its own policies, the agent can learn which is the correct reward function it should be maximising.

Given a policy $π$ , a history $h$ , an environment $μ$ ,and a reward $R$ , we can compute the expected probability of $R$ :

E_{π}^{μ} P (R | h) .

Then a learning process is unbiased if that expression is independent of $π$ , and biased otherwise. Biased processes are less desirable, as they allow the agent to manipulate the process through its choice of policy.

Simple biased learning process

The most trivial example of a biased learning process is an agent that completely determines its reward by its actions.

Let $R = {R_{0}, R_{1}}$ , let the agent only act once with two actions available, ${a_{0}, a_{1}}$ , (hence a choice of "policy" is a choice of action), and set

P (R_{0} | a_{0}) = P (R_{1} | a_{1}) = 1.

Thus the agent can simply choose its reward function through its actions.

Note that some designs are a bit more sophisticated, and don't allow the agent to choose its reward function directly through its actions. But this doesn't matter, if the reward function is a consequence of anything that is a predictable consequence of the agent's actions (eg if the agent can trick/coerce/manipulate a human into saying "yes" or "no", and if $P$ is determined by the human's response, it doesn't matter that $P$ is not defined directly through the agent's actions: it is defined indirectly through them).

[Note that all $P$ that involve learning about external facts are unbiased learning processes, so it's not as if unbiased means trivial]

Strictly dominated behaviour

Then an agent with a biased learning process that wants to maximise the expectation of the true reward, can sometimes follow strictly dominated policies.

That means that there is are policies $π_{0}$ and $π_{1}$ , such that for all histories $h_{i}$ possible given $π_{i}$ , and all reward function $R$ in $R$ ,

R (h_{1}) > R (h_{0}) .

And yet the agent will still choose $π_{0}$ to maximise reward.

For example, with $P$ and $R$ defined as above, define $R_{0}$ and $R_{1}$ to be:

\begin{matrix} a_{0} & a_{1} \end{matrix}

... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Reward Functions