Crossposted at the Intelligent Agents Forum.

In a previous post, I used the model to show how a value-learning agent might pull a Pascal's mugging on humans: it would change the human reward into something much easier to maximise.

But there's a much easier mugging the agent might want to pull: it might want to make the human fully rational. And this does not depend on low-probability-high reward situations.

Planners and Rewards

In the original mugging, the agent has the ability to let the human continue (event ) with , a "standard" human policy according to some criteria, or coercively change their policy (event ) into , another policy.

Thus the actual human policy is , the policy that follows given , and given .

There are two rewards: , the standard human reward according to some criteria, and a reward that is if the agent chooses (i.e. chooses not to coerce them), and if the agent chooses (coerce them into a new policy), where is a reward rationally maximised by .

Thus the agent has two choices: whether to coercively change the human (), and, if so, which (and hence ) to pick.

There are two planners: , the planner under which human behaviour is rational, given , and , a "reasonable" planner that maps to , the actual human policy. The planer is assumed to note the fact that action is overriding human rewards, since is - presumably - poor at maximising .

There are three compatible pairs: , , and . We'll assume the agent has learnt well, and that the probability of is some .

The full equation: rationalise the human

Let be the expected value, according to the reward , of the human following policy . Let be the expected value of if the human follows the optimal -maximising policy.

Then the reward for the agent for not doing the surgery () is:

  • . (1)

On the other hand, the reward for doing the surgery is (recall how splits):

  • . (2)

In the previous post, I focused on the term, wondering if there could be rewards sufficiently high to overcome the probability.

But there's another way (2) could be higher than (1). What if the term is quite high? Indeed, if is not perfectly rational, setting will set (2) to be , which is higher than (1).

(In practice, there may be a compromise , chosen so that is very high, and is not too low compared with (1)).

So if the human is not perfectly rational, the agent will always choose to transform them into rational maximisers if it can.

Personal identity and reward

What is `setting