The Game Theory of Blackmail

by Linda Linsefors 3 min read22nd Mar 201913 comments

23

Ω 6


Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This blog post is composed as following:

  1. Review of Prisoners Dilemma
  2. Explanation of Game of Chicken by comparing it to Prisoners Dilemma
  3. Blackmail is a Game of Chicken
  4. Why we should care about blackmail/Game of Chicken
  5. What to do? Iterated Game of Chicken?

You are encouraged to skip ahead to the part that interests you

1. Review of Prisoners Dilemma

Prisoners Dilemma is a class of two player games which can represent for example mutual beneficial cooperation, or the tragedy of the commons. I don't think it is controversial to say that this class of games are important in almost any multi-agent scenario.

In a Prisoners Dilemma , each player gets to choose between two actions, usually called "cooperate" and "defect". Further more the payoffs haved to fulfill the following:

  • Holding my action constant it is better for me if you cooperate.
  • Holding your action constant, it is better for me if I defect.
  • Cooperate-cooperate is Pareto optimal (even when including mixed strategies).

Example of a payout matrix for Prisoners Dilemma:

In this particular example, cooperate corresponds to spending one of your own utility to give the other player two utility, and defect corresponds to doing nothing. This can represent a situation with the possibility of mutual benefit from cooperation, but where it is possible to win even more (at the other players expense) by cheating.

But we can also consider a negative game:

Here cooperate is doing nothing, while defect corresponds to gaining one utility for yourself while costing the other player two utility. This can represent burning the commons (if the players defect) or not (if they cooperate).

2. Explanation of Game of Chicken by comparing it to Prisoners Dilemma

Just like Prisoners Dilemma, Game of Chicken is a two player game, where each player can choose between two actions. These actions are typically called "swerve" and "straight", but in this blog post I will instead call the two actions "cooperate" and "defect" as to more easily compare with Prisoners Dilemma.

Also the same as Prisoners Dilemma: In Game of Chicken, I get the best payout if I defect and you cooperate (and vice versa). The difference is that conditional on you defecting, it is better for me if i cooperate.

A two action, two player game is a Game of Chicken if:

  • Holding my action constant it is better for me if you cooperate.
  • If you cooperate it is better for me if I defect.
  • If you defect is better for me to cooperate.
  • Cooperate-cooperate is Pareto optimal (even when including mixed strategies).

Furthermore, defect-defect is traditionally super bad for both players. But I would not say that this is a necessary condition for something to be a Game of Chicken.

Example payoff matrix:

The interesting part here is that I can pressure you to cooperate by credibly convincing you that I will defect. In other words, there is a first mover advantage, the first one to precommit to defecting will win against a rational player. However, this fact is of course known by every rational agent, so it might be a rational move to pre-commit to always defect in such games, no mater what. Then again, if two players with such commitments meet, they will both lose.

3. Blackmail is a Game of Chicken

I think that this is easiest explained by just writing out an example payout matrix

If the the blackmailed player gives in, then they pay two utility to give the other player one utility. If the blackmailed player doesn't give in the blackmailer will carry out the threat which is costing both players ten utility. If the the blackmailer doesn't actually blackmails, than nothing happens.

Compare this to the example payout matrix of Game of Chicken. The blackmail payout matrix is not exactly the same, but I claim that in essence this is the same game. If you can handle Game of Chicken then you can handle blackmail both as the blackmailer and the blackmailed.

Not all blackmail is a Game of Chicken. If there is not cost in carrying out the threat then we are in a different type of situation. However I expect this to be rare. It seems unlikely to me that there is no opportunity cost at all in carrying out the threat. Further more, even if costless threats exists in some situations this does not invalidate the argument for considering those blackmail situation where there is a cost to the blackmailer to carryout the threat.

If the blackmailer gains utility by carrying out the threat then I would argue that it is not exactly blackmail anymore. If I have an action that I can take that would help me but hurt you and I ask you for some compensation for refraining from taking this action, then this is more like a value trade than a blackmail.

4. Why we should care about blackmail/Game of Chicken

Prisoners Dilemma receives a lot of attention because this class of games represents an important type of situation in most multiplayer environments. I claim that this is also true for Game of Chicken.

In any situation where one agent (A) has the ability to use up some of its own resources to impose a cost on another agent (B), then A can choose to blackmail B, thus creating a Game of Chicken like situation. And if A thinks that it can win this game, then it will be tempted to engage in blackmail.

If you expect that:

  • It is important to build AI's that can act well in multi-agent situations (e.g. because there will be several simultaneous AIs that are similarly powerful, or there will be acausal trade and threats between agents in different universes simulating each other)

and

  • Toy model such as Prisoners Dilemma are useful

then you should also care about Game of Chicken.

5. What to do? Iterated Game of Chicken?

What should we do about these insights? I am not sure yet. But one possible directions is to study iterated Game of Chicken.

Abram Demski argues that In Logical Time, All Games are Iterated Games. Basically if agents are simulating each other then this is sort of equivalent to the agent playing an iterated game.

Question for the comment section: What would be the winning strategy in iterated Game of Chicken?

I might run a tournament with different strategies.


This post was written with the support of the EA Hotel

23

Ω 6