The first half of this post is a neutral explanation of EDT and CDT. The second half defends EDT and argues against CDT. If you want to learn properly about EDT, you should read Arif Ahmed's Evidence, Decision, and Causality (2014). 

Table of contents

Motivating example: The Prisoners' Dilemma with a copy

Motivating example: The Prisoners' Dilemma with a copy

Imagine you find yourself in the following situation: You and your opponent each are given the choice to either earn $1 or to earn the other player $3. Earning the other player $3 nets you $0. Both of you are asked separately, only once, don't care about each other, and there are also no reputational effects. This problem is called the Prisoners' Dilemma.

This problem seems easy enough. Earning $1 is better than earning $0, so if you don't care about the other person, you should clearly just take the $1.

But what if we introduce a twist: Your opponent is actually a copy of you that some dastardly scientist created. If you're an AI, the other player might just be another instance of you. It seems very likely that whichever choice you make, your opponent will make the same choice! Your actions are correlated with each other.

You would be better off if you both cooperated. Yet, if the other person cooperates, you would be better off if you defect and if the other person defects, you are also better off defecting! How should you decide in this situation?

Defining causal and evidential decision theory

Academics have come up with competing theories with the two most prominent ones giving opposite recommendations. Causal decision theory (CDT) recommends you defect. Evidential decision theory (EDT) recommends you cooperate.

Both causal and evidential decision theory have expected utility maximization as their core: They each say that you should make your decision in three steps: Associate with each action a a probability distribution over outcomes . Then compute expected utilities by multiplying the probabilities with the utilities of outcomes: Then take one of the actions whose expected utility is highest.  They vary on the first step: how to form the probability distributions .

Causal Decision Theory (CDT)

Causal decision theory (CDT) recommends you only consider the causal effects or physical consequences of your actions. In the Prisoners' Dilemma, this leads to the following kind of reasoning: "Well, I can only causally influence my own action. No matter what I do, it won't physically cause my opponent to change their action! So I can just ignore what my opponent does. So, my decision just reduces to gaining $1 or gaining another person $3. I don't care about the other person, so I'll just take $1."

To make the perspective that your action cannot causally change your opponent's action maximally intuitive, imagine the Prisoners' Dilemma with a copy but you know that your opponent already made their choice an hour ago and wrote down their decision. No matter what you do, your opponent's choice is already written down on that piece of paper.

Evidential Decision Theory (EDT)

Evidential decision theory (EDT) recommends you consider the probability of different outcomes if you condition on your action, regardless of whether an outcome is causally downstream of your action. It recommends you ask yourself: "Given everything I know, what do I believe to be the case if I condition on taking action A?” or differently "What evidence does my action provide me?" Evidential effects always include all causal effects and sometimes include additional effects. It's easiest to understand EDT using the Prisoners' Dilemma.

In the Prisoners' Dilemma, EDT would lead to the following reasoning: "If I condition on taking $1, that clearly gives me evidence that I will then own that particular dollar. However, since my opponent is my copy, me taking $1 gives me strong evidence that my opponent also takes $1 and gives me nothing. So, taking $1 will overall result in me having ~$1. On the other hand, if I condition on gifting my copy $3, it would then be very likely that they give me $3. So, gifting $3 will overall result in me having ~$3. So I should cooperate."

EDT still recommends cooperating in the Prisoners' Dilemma if the opponent isn't a copy as long as you think your opponent is sufficiently similar to you, that is, as long as your best guess of what they will choose varies depending on what you choose. If you think that the other agent is 2/3 likely to do the same thing as you regardless of which of the two actions you pick, EDT says you should be indifferent between taking the $1 and giving your opponent $3 since if you take the $1, you gain in expectation:

and if you give the $3, you gain in expectation:

So the two options would be equal in this case.

Further illustrating example: Newcomb's Problem

Another classic example where EDT and CDT come apart is Newcomb's Problem.

Newcomb's Problem as formulated in the LessWrong wiki:

An alien named Omega has come to Earth, and has offered some people the following dilemma.

Before you are two boxes, Box A and Box B.

You may choose to take both boxes ("two-box"), or take only Box B ("one-box").

Box A is transparent and contains $1,000.

Box B is opaque and contains either $1,000,000 or $0.

The alien Omega has already set up the situation and departed, but previously put $1,000,000 into Box B if and only if Omega predicted that you would one-box (take only the opaque Box B and leave Box A and its $1,000 behind).

Omega is an excellent predictor of human behavior. We can assume e.g. that Omega has run 67 previous experiments and not been wrong even once.

CDT recommends two-boxing in this dilemma. Box B is already either filled or not filled. Your action doesn't physically cause any change in the box. And whether Omega predicted one-boxing (left side of the diagram) or two-boxing (right side), you are better off taking both boxes rather than one box.

EDT recommends you one-box. While your action doesn't causally change whether Box B is full or not, your action gives you evidence about whether you are in the branch where Omega predicted you will one-box (the left side of the diagram) or the branch where Omega predicted you will two-box (the right side of the diagram). That's because no matter what you choose, it is very likely that Omega's prediction made the same choice: You correlate with Omega's prediction of you.

Adjudicating between EDT and CDT

There is no agreed-upon metric to compare which decision theory does better. Both decision theories succeed by their own lights: Following CDT leads to the best causal effects and following EDT leads to the best evidential effects. One thing we can do is assess whether the theories give us the intuitively correct results. However, we quickly hit philosophical bedrock when doing so. While one person might find EDT's recommendations intuitive, another might think the opposite or be unsure. Hence, disagreement persists. Which decision theory is correct might be subjective, similar to what some people believe about morality. Emery Cooper argues for decision theory antirealism here.

Despite this, we can still make arguments for one decision theory over another, similar to how we can make moral arguments. I think that the arguments against CDT are very strong and so favour EDT or some hypothetical third decision theory. I will go through an example where EDT purportedly recommends the intuitively wrong answer and the rebuttal. Then I will show that CDT agents are vulnerable to schemes to extract all their money.

EDT solves the Smoking Lesion scenario correctly

Smoking Lesion (as printed in Ahmed, 2014, originally from Egan, 2007)

Susan is debating whether (O1) not to smoke or (O2) to smoke. She believes that smoking is strongly correlated with lung cancer, but only because there is a common cause – a lesion that tends to cause both smoking and cancer. Once we fix the absence (S1) or presence (S2) of this lesion, there is no additional correlation between smoking and cancer. Susan prefers smoking without cancer to not smoking without cancer; and she prefers smoking with cancer to not smoking with cancer. Should Susan smoke?

To be very clear, in the real world, smoking does in fact cause lung cancer. But in the world of Smoking Lesion where it doesn't, it seems clear that Susan should smoke. This is what CDT recommends.

Opponents of EDT argue that EDT recommends Susan not to smoke in Smoking Lesion. Smoking is correlated with cancer, the argument goes, so Susan would gain evidence that she has cancer if she smokes and so smoking is evidentially bad. EDT seems to get this case wrong.

I argue that in all cases where it's clear that smoking is correct, smoking does not actually provide any evidence of cancer. And in cases where smoking does provide evidence of cancer, we just shouldn't smoke (or not smoking in these cases is at least no less intuitive than one-boxing in Newcomb's Problem).

In Smoking Lesion, it's natural to imagine that the lesion causes smoking by causing a desire to smoke. (Presumably if in the real world we found that a gene caused both cancer and some behavior, we would expect some such mechanism.) If this is the case and known to Susan, then Susan already knows whether or not she has the desire to smoke and so actually smoking doesn't give her any additional evidence beyond what the evidence she already has from observing her desire to smoke. This line of reasoning is also called the Tickle Defense.

Alternatively, maybe Susan has some reason to think that the lesion caused smoking by affecting how agents reason about the Smoking Lesion Problem, regardless of their desire to smoke. In this case, I think the situation looks very similar to normal Newcomb's Problem and it's in fact totally reasonable not to smoke.

For further readings on the Smoking Lesion see here. For a detailed defense of EDT in the Smoking Lesion, see Understanding the Tickle Defense in Decision Theory (Oesterheld, 2018) and Evidence, Decision, and Causality (Ahmed, 2014)

Extracting money out of CDT agents using the "Adversarial Offer"

One of the strongest arguments against CDT is that CDT agents are vulnerable to exploitation.

Adversarial Offer (from Oesterheld and Conitzer, 2021)

Adversarial Offer: Two boxes, B1 and B2, are on offer. A (risk-neutral) buyer may purchase one or none of the boxes but not both. Each of the two boxes costs $1. Yesterday, the seller put $3 in each box that she predicted the buyer not to acquire. Both the seller and the buyer believe the seller’s prediction to be accurate with probability 0.75.

Buying a box in this setting results in an expected loss: You pay $1 and in expectation gain 0.25*$3 for an expected value of –$1 + 0.25 * $3 = –$0.25. CDT agents apply this logic when reasoning about what they expect to happen when a different agent (including their future self) buys a box when given the choice. Yet, when actually in the situation, CDT agents buy a box anyway. If they are given adversarial offers repeatedly, they will buy boxes until they run out of money. Here is why.

For the purposes of making a decision, CDT agents treat the state of the world before the decision point as independent from their decision. That makes sense from a CDT perspective because everything before the agent's choice cannot be causally affected by the agent's choice—causality only goes forward.

They first form beliefs about what the world is like in a way that doesn't depend on their choice and then make a choice based on those beliefs. In this case, the CDT agent has to split its credence between the hypotheses "Only B1 has money", "Only B2 has money", and “B1 and B2 both have money”. Since the probabilities have to sum to 1, this means that at least one of B1 or B2 is assigned an at least 0.5 probability of containing money. This means for at least one of the boxes, buying that box has at least expected value of

which is positive and better than buying nothing. From that perspective, the CDT agent should buy one of the boxes. (The agent buys whichever box the agent has higher credence in having the money. If the credences are equal, then the agent is indifferent between buying B1 and B2 but will prefer both to not buying.)

Of course, as soon as the CDT agent makes the decision, the agent can update its credence about which box contains the money. However, this anticipated update does not influence the actual decision-making since CDT agents only consider the causal consequences of their actions and choosing to buy a box doesn't causally change whether it contains money. They can only consider the credence they have in each box containing the money before they make a decision.

Note that CDT agents predict the outcome accurately when reasoning about how another agent will fare when facing the adversarial offer. It is only when reasoning about their present decision that they don't consider the relationship between the choice to buy a particular box and the probability that the box contains money.

EDT agents escape this problem since their decision-relevant credence in whether B1 contains money can depend on their own choice. EDT agents estimate the probability that B1 has money conditional on the EDT agent (not) buying B1 and the same for B2. They correctly conclude that they shouldn't buy a box.

For more details on the adversarial offer, see this paper.

Closing side remarks

Other arguments

The scenarios I discussed only really scratch the surface of the debate between EDT and CDT. For a more detailed treatment in favour of EDT (when the alternative is CDT), see Evidential Decision Theory is correct by Almond (2010) and the aforementioned Evidence, Decision, and Causality by Arif Ahmed (2014).

EDT and CDT don't self-modify to convergence

I just want to get ahead of a common misunderstanding that EDT and CDT self-modify until they converge to the same theory. This is not true and covered in the post in this series about updatelessness. (Not public at the time of publishing, will update with the link.)

On Functional Decision Theory

I don't understand Functional Decision Theory (FDT) as well as the other two theories and am not aware of any instance where FDT diverges from EDT with updatelessness which is discussed in a later post in this series. (Not public at the time of publishing, will update with the link.) Hence I don't cover FDT in this post.