A putative new idea for AI control; index here.

This post is just an initial foray into modelling human irrationality, for the purpose of successful value learning. Its purpose is not to be full model, but have enough details that various common situations can be successfully modelled. The important thing is to model humans in ways that humans can understand (as it's our definition which determines what's a bias and what's a preference in humans).

Humans, actions, and joint distributions

The human themselves is simply modelled as their brain (thus various human sense organs can be observed by the AI rather than being part of the description).

Let $R$ be the set of possible reward functions the human may be maximising. Let $H_{π}$ be the set of policies the human may be following. We'll assume that $H_{π}$ is closed under the taking of mixed strategies.

The AI has a joint probability distribution $P$ over $R$ , $H_{π}$ and events in the world. By conditioning on any element $r \in R$ , $P$ defines a map $μ$ from $R$ to probability distributions over $H_{π}$ . Since $H_{π}$ is closed under the taking of mixed strategies, this means that $μ$ can be seen as a map from $R$ to $H_{π}$ .

The map $μ$ and the marginal distribution $P_{R}$ ( $P$ restricted to $R$ ) define $P$ entirely. Note that $μ$ is what relates human actions to their explanation in terms of the reward $R$ .

Basic properties of $P$

Here are a few properties $P$ could have:

#. The distribution $P$ is historical if $P_{R}$ is independent of any action the AI takes. #. An AI's action $a$ overwrites the reward if $μ$ is constant, conditional on $a$ , while $P_{R}$ is still `broad' ("broad" is not fully defined, but $P_{R}$ is certainly broad enough if it assigns non-zero probability to both an $r$ and $- r$ ). #. The distribution $P$ is $Q$ -rational if there exits a prior distribution $Q$ over the universe such that $μ$ maps $r \in R$ to the optimal policy for an $r$ -maximising agent with prior $Q$ .

It's clear that if $P$ is historical, the AI will treat the human's reward function as something it has to discover, and can't influence. An action $a$ that overwrites the reward means that the human's policy is fixed by action $a$ , independently of whatever reward it might have. This is bad because a) the human actions are no longer informative to the AI about their reward, and b) the human actions are likely suboptimal with respect

Note that stratification can be seen as taking a non-historical distribution, and making it historical via coun

...