Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

All the methods proposed for factoring out (for having the AI maximise a certain value while 'ignoring' it's impact via ) can be put on the same general footing. For some set , define a function on with and .

Then for a utility , the general requirements is for the AI to maximise the quantity:

  • ,

subject to some constraints on .


Let's play a game

Define the two-player game by allowing each player to have moves in . The expected utility, for player , of the moves is defined to be . To completely define the game, we need the expected utility for the second player. We'll just set that to ensure the game is symmetric the expected utility of player two for is the same as the expected utility for player one for .

Then is effectively a probability distribution over the choice of possible moves for . Standard symmetric games include the stag hunt, the prisoner's dilemma, and the coordination game. For our purposes, since the actions for have meaningful labels, we'll be considering skew coordination game given as follows:

+-----+-----+-----+ | | l | r | +-----+-----+-----+ | l |(0,0)|(1,1)| +-----+-----+-----+ | r |(1,1)|(0,0)| +-----+-----+-----+

These 'games' will be mainly used to show that the various methods reach different solutions in different situations, hence that they are genuinely different methods.

Standard maximalisation

If we constrain if , then that equation becomes:

This makes into the action distribution over the AI must choose from to maximise in the classical sense. In terms of , this means the AI will make the superrational choice between the two players, if they lack any way of distinguishing themselves. Thus it will pick for the stag hunt, in the prisoner's dilemma, and or (or some mixture between the two) in the skew coordination game.

Double variable maximalisation

First consider that there are no additional constraints on . The AI can pick the values to be whatever it wants. It therefore acts like a dictator in , choosing both player's outcomes. Under this, it will pick for the stag hunt, in the prisoner's dilemma, or (or some mixture between the two) in the skew coordination game. Note that the AI can always achieve maximal utility by choosing a single action for both players. Thus if we added the restriction that , it could still reach at least one of these optimums.

What if we added the restriction for a single ? Maximising this over possible is equivalent with the AI reaching the superrational decision for both players, where they have access to individual but not joint sources of randomness. Thus it will pick for the stag hunt, in the prisoner's dilemma, and in the skew coordination game.

For the next case, consider that , as above, but the AI is not aiming to maximise , but to pick an that ensures

  • for all .

This is Chris and Jessica's method. You can transform that requirement into a quantity to be maximised, but it's not particularly illuminating. What's more illuminating is that the that solve this are precisely the symmetric Nash equilibriums to (all symmetric games have at least one symmetric Nash equilibrium). ensures they must be symmetric, the condition on picking ensures the first player cannot improve their expectation unilaterally, and, since itself is symmetric, the second player cannot improve their expectation unilaterally.

In cases where there are multiple symmetric Nash equilibriums, we may as well have the AI choose the one that maximises the . Thus the AI will choose for the stag hunt, in the prisoner's dilemma, and in the skew coordination game.

Single variable maximalisation

Here where is some fixed distribution. There are some obvious candidates for that - maybe one action is a default action, or is uniform across all actions. There are some more complicated methods to assign probabilities in a way that is sensible if there are multiple branching decisions.

In this case, the AI will pick or for the stag hunt, depending on and the exact rewards of the game, will always choose in the prisoner's dilemma, and will choose if and if in the skew coordination game.

Summary

There are thus five methods, clearly distinguished by making different choices in the different games:

  1. unless . is maximalised.
  2. There are no constraints on , or . is maximalised.
  3. and is maximalised.
  4. and for all .
  5. for fixed and is maximalised.

Arbitrarily 'bad' decisions

All the methods (except for the first one) can reach arbitrarily bad decision in terms of real expected utility, as compared with standard expected utility maximalisation. Consider the following extension of the skew coordination problem, for large :

+-----+---------+---------+---------+ | | l | r | c | +-----+---------+---------+---------+ | l | (-W,-W) |(W+1,W+1)| (0,0) | +-----+---------+---------+---------+ | r |(W+1,W+1)| (-W,-W) | (0,0) | +-----+---------+---------+---------+ | c | (0,0) | (0,0) | (-1,-1) | +-----+---------+---------+---------+

All alternative methods will choose actions from among and only. This condemns them to a utility of , while the best action choice, , has an expected utility of .

Further considerations

To distinguish which method we should be using, issues like stability, self-consistency, and other properties will probably be needed.

New to LessWrong?

New Comment
9 comments, sorted by Click to highlight new comments since: Today at 11:27 AM

You might be interested in a way of ensuring that 2 players always have the same mixed strategy in all Nash equilibria of some game:

Assume we have a player and a player . Player has some already-specified utility function; we would like player to play the same mixed strategy as . Introduce a new player who gets to observe either or 's action (unknown with 50% probability for each), and tries to determine who took this action (getting a utility of 1 for guessing correctly and 0 otherwise). 's utility function is 1 if guesses incorrectly, and 0 if guesses correctly. will use the same mixed strategy as in all Nash equilibria.

A similar method is used in the appendix A of the reflective oracles paper.

Typo in the "Single Variable Maximalisation" section: you meant to write rather than .

Thanks, corrected!

I'm not sure what the "arbitrarily bad decisions" example is meant to illustrate? If the 2 agents randomize uniformly between r and l, they each get an expected utility of 1/2, which is greater than -1.

But there aren't two players, that's just the model. What I mean is that all these ways of factoring out B can lead to arbitrary bad real expected utility, as compared with the agent that doesn't factor.

I still don't understand why the expected utility is rather than .

In the real world, the utility is given by the diagonal (since and being different in is the fiction allowing for factoring of ). Therefore the genuine expected utilities are only on the diagonal, and anything else than will give .

There's nothing in the setup preventing the players from having access to independent random bits, though it's fair to say that these approaches assume this to be the case even when it's not.

But then the fault is with that assumption of access to randomness, not with any of the constraints on . So I don't think this is a strike against these methods.

I'm not following. This "game" isn't a real game. There are not multiple players. There is one agent, where we have taken its real, one-valued probability, and changed it into a two-valued , for the purposes of factoring out the impact of the variable. The real probability is the original probability, which is the diagonal of .