EDT chokes on Simpson's Paradox in general.
Consider:
There's a vaccine that purports to reduce the risk of a certain infection... some of the time.
After doing some investigation, you come up with the following statistics:
Roughly 10% of the population has taken the vaccine. Of those that have taken the vaccine, the infection rate is 25%. Of those that have not taken the vaccine, the infection rate is 10%.
P(Infection|vaccine) > P(Infection|~vaccine), so EDT says don't give your child the vaccine.
You do some more research and discover the following:
The infection rate for unvaccinated heterosexuals is 11%.
The infection rate for vaccinated heterosexuals is 9%.
The infection rate for unvaccinated non-heterosexuals is 40%.
The infection rate for vaccinated non-heterosexuals is 30%.
Non-heterosexuals are a small minority of the population, but are disproportionally represented among all those who are vaccinated.
Getting vaccinated doesn't change a person's sexual orientation.
EDT now comes to the ridiculous conclusion that heterosexuals should take the vaccine and that non-heterosexuals should take the vaccine, but people whose sexual orientation is unknown should not - even though everyone is either heterosexual or non-heterosexual.
I don't think this math is right. There are two general problems I see with it. Number one is that once you find out this,
Non-heterosexuals are a small minority of the population, but are disproportionally represented among all those who are vaccinated.
your P(infection|vaccine) and P(infection|~vaccine) change to values that make more intuitive sense.
Number two is that I don't think the percentages you gave in the problem are actually coherent. I don't believe it's possible to get the 25% infected and vaccinated and 10% infected and unvaccinated with the numbers provided, no matter what values you use for the rate of heterosexuality in the population, and in the vaccinated and unvaccinated groups. I don't really want to discuss that in detail, so for the sake of my post here, I'm going to say that 80% of the population is heterosexual, and that 85% of the individuals who have gotten the vaccine are non-heterosexual.
Using these numbers, I calculate that the initial study would find that 26.85% of the vaccinated individuals are infected, and 14.70555...% of the unvaccinated individuals are infected. Note that given just this information, these are the probabilities that a bayesian should assign to the proposition(s), "If I (do not) get the vaccine, I will become infected". (Unless of course said bayesian managed to pay attention enough to realize that this wasn't a randomized, controlled experiment, and that therefore the results are highly suspect).
However, once we learn that this survey wasn't performed using proper, accurate scientific methods, and we gather more data (presumably paying a bit more attention to the methodology), we can calculate a new P(infected|vaccinated) and a new P(infected|unvaccinated) for our child of unknown sexual orientation. As I calculate it, if you believe that your child is heterosexual with 80% confidence (which is the general rate of heterosexuality in the population, in our hypothetical scenario), you calculate that P(infection|vaccinated) = .132, and P(infection|unvaccinated) = .168. So, EDT says to get the vaccine. Alternatively, let's say you're only 10% confident your child is heterosexual. In this case, P(infection|vaccinated) is .279, and P(infection|unvaccinated) = .371. Definitely get the vaccine. Say you're 90% confident your child is heterosexual. Then P(infection|vaccinated) = .111, while P(infection|unvaccinated) = .139. Still, get the vaccine.
Ultimately, using the raw data from the biased study cited initially as your actual confidence level makes about as much sense as applying Laplace's rule of succession in a case where you know the person drawing the balls is searching through the bag to draw out only the red balls, and concluding that if you draw out a ball from the bag without looking, it will almost certainly be red. It's simply the wrong way for a bayesian to calculate the probability.
If anything, I think this hypothetical scenario is not so much a refutation of EDT, so much as a demonstration of why proper scientific methodology is important.
Well, you're right that I did make up the numbers without checking anything. :(
Here's a version with numbers that work, courtesy of Judea Pearl's book: http://escholarship.org/uc/item/3s62r0d6
Okay, this is going to sound weird. But again, how do I know that getting vaccinated doesn't change a person's sexual orientation? Presumably, the drug was tested, and someone would have noticed if the intervention group turned out "less straight" than the controls. But that doesn't rule out the possibility that something about me choosing to have my child vaccinated will make him/her non-heterosexual. To us, this just contradicts common sense about how the world works, but that common sense isn't represented anywhere in the problem as you stated it.
If all I know about the universe I inhabit are the conditional probabilities you gave, then no, I wouldn't have my child vaccinated. In fact, I would even have to precommit to not having my child vaccinated, in the case that I find out his/her sexual orientation later. On the face of it, this conclusion isn't any more ridiculous than Prisoner's dilemma, where you should defect if your opponent defects, and defect if your opponent cooperates, but should cooperate if you don't know what your opponent will do. Even though your opponent will either cooperate or defect.
UDT solves the apparent contradiction by saying that you should cooperate even if you know what your opponent will choose (as long as you think your opponent is enough like you that you would have predicted his decision to correlate with your own, had you not already known his decision). But this also leads to an intuitively absurd conclusion: that you should care about all worlds that could have been the same way you care about your own world. Sacrificing your own universe for the sake of a universe that could have been may be ethical, in the very broadest sense of altruism, but it doesn't seem rational.
EDT can't come to the conclusion that getting vaccinated doesn't cause, or share a common cause with, homosexuality.
Here, your premise is actually that homosexuality causes vaccinations.
Here, your premise is actually that homosexuality causes vaccinations.
Exactly. They know they're at higher risk, so they're more likely to seek out the vaccine.
No, lower risk people seek out the vaccine: it's right there, people who are vaccinated are less likely to be infected, just like people who are less likely to get lung cancer eschew smoking.
This example is isomorphic to the Smoker's lesion, with being non-heterosexual = having the lesion, infection = cancer, taking the vaccine = stopping smoking. The recipe suggested in the original post can be extended here: separate the decision to take the vaccine using EDT while not knowing about own sexual orientation (A), the state of having taken the vaccine (E) and the outcome of not being infected (O). Although P(O|E) < P(O|~E), it is not true that P(O|A) < P(O|~A).
Your example contains several unstated assumptions:
Hence the decision to take the vaccine with an unknown sexual orientation never occours.
EDT chokes because it ignores the obvious extra controlling principle: those with no sexual experience don't generally have the infection.
Add that, and you are fine. Vaccinate before the time of first expected exposure
I've found that, in practice, most versions of EDT are underspecified, and people use their intuitions to fill the gaps in one direction or the other.
"But if I know that, then, as an evidential decision theorist, I have no reason to turn the sprinklers on."
As an evidential decision theorist, you have no formal language to describe results of interventions as distinct from results of observations. Without this language, you cannot represent the difference between the fact that fast olympic sprinters are likely to wear gold medals, but at the same time putting on a gold medal will not make you run faster. You have no primitive that represents a decision to put on a gold medal as distinct from observing that a gold medal is worn.
In order to have such a primitive you need some notation for something like an intervention: perhaps do(.), or perhaps Neyman's potential outcomes, or perhaps Lewis' counterfactual connective.
"Ultimately, I think that causation is just a convenient short-hand that we use. In practice, we infer causal relations by observing conditional probabilities."
This is wrong, and you need to do some reading: Causality by Pearl, or Causation, Prediction and Search by Spirtes/Scheines/Glymour. It is not possible to infer causal relationships strictly from conditional probabilities without additional assumptions. "No causes in -- no causes out." See also this: http://www.smbc-comics.com/index.php?db=comics&id=1994#comic
Well, why not? The decision to put on a gold medal (D) is an observable event just like noticing you wear one (G) or running fast (F) are. And P(F|D&G) ~= P(F|D) = P(F) < P(F|~D&G). Lacking separate primitives for deciding to take actions and observing the results of these actions is the problem, and it's pretty fixable. This doesn't quite get us causation (we can't tell forks and chains apart) but solves Smoking Lesion.
Let's look at a slightly more complicated example:
A patient comes in to the hospital because he's sick, his vitals are taken (C), and based on C, a doctor prescribes medication A. Sometime later, the patient dies (Y). Say this happens for lots of patients, and we form an empirical distribution p(C,A,Y) from these cases. Things we may want to represent:
marginal probability of death: p(Y)
the policy the doctor followed when prescribing the medicine: p(A|C)
the probability that someone would die given that they were given the medicine: p(Y|A)
the "causal effect" of medicine on death: ???
The issue with the "causal effect" is that the doctor's policy, if it is sensible, will be more likely to prescribe A to people who are already very sick, and not prescribe A to people who are mostly healthy. Thus it may very well turn out that p(Y|A) is higher than the probability p(Y) (this happens for example with HIV drugs). But surely this doesn't mean that medicine doesn't help! What we need is the probability of death given that the doctor arbitrarily decided to give medicine, regardless of background status C. This arbitrary decision "decouples" the influence of the patient's health status and the influence of the medicine in the sense that if we average over health status for patients after such a decision was made, we would get just the effect of the medicine itself.
For this we need to distinguish an "arbitrary" decision to give medicine, divorced from the policy. Call this arbitrary decision A . You may then claim that we can just use rules of conditional probabilities on events Y,C,A, and A , and we do not need to involve anything else. Of course, if you try to write down sensible axioms that relate A with Y,C,A you will notice that essentially you are writing down standard axioms of potential outcomes (one way of thinking about potential outcomes is they are random variables after a hypothetical "arbitrary decision" like A ). For example, if were to arbitrarily decide to prescribe medicine just precisely for the patients the doctor prescribed medicine to, we would get that p(Y|A) = p(Y|A ) (this is known as the consistency axiom). You will find that your A functions as an intervention do(a).
There is a big literature on how intervention events differ from observation events (they have different axiomatizations, for example). You may choose to use a notation for interventions that does not look different from observations, but the underlying math will be different if you wish to be sensible. That is the point. You need new math beyond math about evidence (which is what probability theory is).
It seems to me that we often treat EDT decisions with some sort of hindsight bias. For instance, given that we know that the action A (turning on sprinklers) doesn't increase the probability of the outcome O (rain) it looks very foolish to do A. Likewise, a DT that suggests doing A may look foolish. But isn't the point here that the deciding agent doesn't know that? All he knows is, that P(E|A)>P(E) and P(O|E)>P(O). Of course A still might have no or even a negative causal effect on O, but yet we have more reason the believe otherwise. To illustrate that, consider the following scenario:
Imagine you find yourself in a white room with one red button. You have no idea why you're there and what this is all about. During the first half hour you are undecided whether you should press the button. Finally your curiosity dominates other considerations and you press the button. Immediately you feel a blissful release of happiness hormones. If you use induction it seems plausible to infer that considering certain time intervalls (f.i. of 1 minute) P(bliss|button) > P(bliss). Now the effect has ceased and you wished to be shot up again. Is it now rational to press the button a second time? I would say yes. And I don't think that this is controversial. And since we can repeat the pattern with further variables it should also work with the example above.
From that point of view it doesn't seem to be foolish at all - talking about the sprinkler again - to have a non-zero credence in A (turning sprinkler on) increasing the probability of O (rain). In situations with that few knowledge and no further counter-evidence (which f.i. might suggest that A might have no or a negative influence on O) this should lead an agent to do A.
Considering the doctor, again, I think we have to stay clear about what the doctor actually knows. Let's imagine a doctor who lost all his knowledge about medicine. Now he reads one study which shows that P(Y|A) > P(Y). It seems to me that given that piece of information (and only that!) a rational doctor shouldn't do A. However, once he reads the next study he can figure out that C (the trait "cancer") is confounding the previous assessment because most patients who are treated with A show C as well, whereas ~A don't. This update (depending on the probability values respectively) will then lead to a shift favoring the action A again.
To summarize: I think many objections against EDT fail once we really clarify what the agent knows respectively. In scenarios with few knowledge EDT seems to give the right answers. Once we add further knowledge an EDT updates his beliefs and won’t turn on the sprinkler in order to increase the probability of rain. As we know it from the hindsight bias it might be difficult to really imagine what actually would be different if we didn’t know what we do know now.
Maybe that's all streaked with flaws, so if you find some please hand me over the lottery tickets ; )
I don't get the relevance of this p(Y|A).
In EDT, you contition on both action and observations.
In this case, the EDT doctor prescribes argmin_A p(Y | A & C)
What's the problem with that?
There is no problem with what the doctor is doing. The doctor is trying to minimize the number of deaths given that (s)he measures C, as you said.
The question is, how do we quantify what the effect of medicine A on death is? In other words, how do you answer the question "does medicine A help or hurt?" given that you know p(Y,A,C). This is where you don't want to use p(Y | A). This is because sicker people will die more and get medicine more, hence you might be mislead into thinking that giving people A increases death risk.
This is because sicker people will die more and get medicine more, hence you might be mislead into thinking that giving people A increases death risk.
Only if you ignore the symptoms.
In medicine you want answer questions of the type "given symptoms C, does medicine A help or hurt?"
Often, but not always (one common issue is the size of C can be very large). Even if you measure all the symptoms, and are interested in the effect of the medicine conditional on these symptoms (what they call "effect modification" in epidemiology) there is the question of confounders you did not measure that would prevent p(Y | A, C) from being equal to the effect you want, which is p(Y | do(A), C).
No, this is not Simpson's paradox. Or rather, the reason Simpson's paradox is counterintuitive is precisely the same reason that you should not use conditional probabilities to represent causal effects. That reason is "confounders," that is common causes of both your conditioned on variable and your outcome.
Simpson's paradox is a situation where:
P(E|C) > P(E|not C), but
P(E|C,F) < P(E|not C,F) and P(E|C,not F) < P(E|not C, not F).
If instead of conditioning on C, we use "arbitrary decisions" or do(.), we get that if
P(E|do(C),F) < P(E|do(not C),F), and P(E|do(C),not F) < P(E|do(not C), not F), then
P(E|do(C)) < P(E|do(not C))
which is the intuitive conclusion people like. The issue is that the first set of equations is a perfectly consistent set of constraints on a joint density P(E,C,F). However, people want to interpret that set of constraints causally, e.g. as a guide for making a decision on C. But decisions are not evidence, decisions are do(.). Hence you need the second set of equations, which do have the property of "conserving inequalities by averaging" we want.
See also: http://bayes.cs.ucla.edu/R264.pdf
In my example, the issue was that p(Y|do(A)) was different from p(Y|A) due to the confounding effect of "health status" C. The point is that interventions remove the influence of confounding common causes by "being arbitrary" and not depending on them.
EDT, and more generally standard probability theory, simply fails on causality due to a lack of explicit axiomatization of causal notions like "confounder" or "effect."
Only if you screwed up your primitives definitions in a slightly subtler way, so you can't distinguish between decision to put on a gold medal and decision not to fight the CIO member who's trying to put one around your neck, or if you forgot to condition on things like "cares about running fast".
It is not possible to infer causal relationships strictly from conditional probabilities
Yeah. I stand corrected there. I know Pearl is relevant to this, but I haven't read all of his stuff yet. I guess what I mean is that I'm not sure causation is a real thing that we should care about. If you can't infer causal relations from observations, then in what sense are they real?
There are two answers here:
(1) In one sense, we can infer causal relations from observations. There is quite a large literature on both estimating causal effects from observational data, and on learning causal structure from data. However, for both of these problems you need quite strong assumptions (causal assumptions) that make the game work. You cannot create causality from nothing.
(1a) It is also not quite true that we always need to use observations to infer causality. If you look at a little kid exploring the world, (s)he will always be pushing/prodding things in the environment -- experimenting to discover causal relations. I went to a talk (http://www.alisongopnik.com/) that showed a video of kids aged about 5 figuring out a causal arrow in a little Newtonian gear system by doing experiments. Humans are very good at getting causality from experimental data, even while young!
(2) In another sense, causality is a "useful fiction" like the derivative or real numbers. If you look at the Universe on the scale of the universal wave function, there is no causality, just the evolution equation. But if we look at the scale of "individual agents", it is a useful abstraction to talk about arbitrary interventions, which is a building block from which certain kinds of notions of "causality" spring from.
If you're interested in this stuff, I recommend Judea Pearl's Causality: Models, Reasoning, and Inference.
Basically, by using conditional probabilities you can in fact do the same math as CDT in situations that warrant it. Currently your math is a loosely-defined hodge-podge of what seem like it works, but if you look at Causality you'll see some sweet stuff.
Said sweet stuff still fails on cooperative or anthropic problems, though, so you do need something like UDT.
Wow, I've been cited. Exciting! But I no longer endorse EDT. Here's why:
Anyway, when I posted that, someone pointed out that EDT fails the transparent-box variant of Newcomb's problem, where Omega puts $1 million in box A if he would expect you to 1-box upon seeing box A with $1 million in it, and puts nothing in box A otherwise. An EDT agent who sees a full box A has no reason not to take the $1,000 in box B as well, because that does not provide evidence that he will not be able to get $1 million from box A, since he can already see that he can. But because of this, an EDT agent will never see $1 million in box A.
If all I know about the world I inhabit are the two facts: (1) the probability of rain is higher, given that the ground is wet, and (2) The probability of the ground being wet is higher, given that I turn the sprinklers on - then turning the sprinklers on really is the rational thing to do, if I want it to rain.
That is correct for straightforward models of complete uncertainty about the causal structure underlying (1) and (2), but also irrelevant. CDT can also handle causal uncertainty correctly, and EDT is criticized for acting the same way even when it is known that turning the sprinklers on does not increase the probability of rain enough to be worthwhile. You did address this, but I'm just saying that mentioning the possibility that turning the sprinklers on could be the correct action given some set of partial information doesn't really add anything.
and A is the action of deciding to not smoke for the purpose of avoiding cancer.
People aren't that good and understanding why they do things. It might seem like you decided not to smoke because of EDT, but the more you want to smoke, the less likely you are to follow that line of reasoning.
One of the first things we should expect a self-modifying EDT agent to do, is to make a blanket precommitment for all such problems.
It wouldn't. It would make a blanket precommitment for all such problems that have not already started. It would treat all problems currently in motion differently. Essentially, the ability to modify itself means that it can pick every future choice right now, but if "right now" is half way through the problem, that's not going to matter a whole lot.
It would make a blanket precommitment for all such problems that have not already started.
What do you mean by "already started"? An EDT agent doesn't really care about time, because it doesn't care about causation. So it will act to control events that have already happened chronologically, as long as it doesn't know how they turned out.
People aren't that good at understanding why they do things. It might seem like you decided not to smoke because of EDT, but the more you want to smoke, the less likely you are to follow that line of reasoning.
This is a great point. But it's an issue for us imperfect humans more than for a true EDT agent. This is why my default is to avoid activities associated with higher mortality, until I have good reason to think that, for whatever reason, my participation in the activity wouldn't be inauspicious, after accounting for other things that I know.
What do you mean by "already started"? An EDT agent doesn't really care about time, because it doesn't care about causation.
Take Parfit's hitchhiker. An EDT agent that's stuck in the desert will self-modify in such a way that, when someone offers to pick him up, he knows he'll keep the promise to pay them. If an EDT agent has already been picked up and now has to make the choice to pay the guy (perhaps he became EDT when he had an epiphany during the ride), he won't pay, because he already knows he got the ride and doesn't want to waste the money.
An EDT agent that's stuck in the desert will self-modify in such a way that, when someone offers to pick him up, he knows he'll keep the promise to pay them.
If the agent is able to credibly commit (I assume that's what you mean by self modification), he doesn't have to do that in advance. He can just commit when he's offered the ride.
On a side note, the entry you linked says:
This is the dilemma of Parfit's Hitchhiker, and the above is the standard resolution according to mainstream philosophy's causal decision theory
Is it actually correct that causal decision theory is mainstream? I was under the impression that EDT is mainstream, so much that is usually referred to just as decision theory.
He can just commit when he's offered the ride.
He can in that example. There are others where he can't. For example, the guy picking him up might have other ways of figuring out if he'd pay, and not explain what's going on until the ride, when it's too late to commit.
Is it actually correct that causal decision theory is mainstream?
I don't know. Both of them are major enough to have Wikipedia articles. I've heard that philosophers are split on Newcomb's paradox, which would separate CDTers and EDTers.
In any case, both decision theories give the same answer for Parfit's hitchhiker.
Okay, that's what I thought. So this has nothing to do with time, in the sense of what happens first, but rather the agent's current state of knowledge, in the sense of what it already knows about. Thanks for clarifying. I'm just not convinced that this is a bug, rather than a feature, for an agent that can make arbitrary precommitments.
and A is the action of deciding to not smoke for the purpose of avoiding cancer.
People aren't that good and understanding why they do things. It might seem like you decided not to smoke because of EDT, but the more you want to smoke, the less likely you are to follow that line of reasoning.
This is being used as a proxy for the presence of the gene in question; an easy way around our lack of introspection is to use another proxy: testing for the presence of the gene.
If this were an option, it wouldn't change the problem. An EDT agent that would quit smoking for the good news value, without knowing whether it had the gene, would either avoid getting tested, or precommit to stop smoking regardless of the test results. It would do this for the same reason that, in Newcomb's problem, it wouldn't want to know whether the opaque box was empty before making its decision, even if it could.
It's used as a proxy for a lot of things. This isn't about one specific situation. It's about a class of problems.
I've had a similar nagging feeling about the smoker's lesion paradox for a long time, but have never actually taken the time to research it. I'm going to use the the language of the chewing gum version though.
A common way (I've heard) of describing EDT is that you make the decision that will make your cousin the happiest (assuming your cousin has no particular stake in the matter, other than your own well being). Well, if I am your cousin, and I learn that chewing gum corresponds to a gene, which corresponds to higher rates of throat cancers, I will revise my probability that you will get cancer up. However, if I learn that you have never chewed gum before, and started chewing gum after you read the same study I did, I will revise my probability of cancer down, suggesting that maybe you should chew gum after all. The same argument holds for if you did chew gum before. So it's always seemed to me that EDT gives the wrong answer on the smoker's lesion problem only if you don't take into account the observation that you previously did (or did not) chew gum.
I've had a related doubt about the olympic medalist issue. The probability that someone is a very fast runner, given that they are wearing an olympic gold medalist (for a running event, presumably), is very high. But the probability that someone is a very fast runner, given that they are wearing a gold medal and that they are wearing this medal because they believe it will make them a faster runner is not.
Now, I assume that both of the previous paragraphs are wrong. It seems unlikely to me that very intelligent people would spend entire careers thinking about issues like this, and fail to see flaws that I spotted almost immediately, despite having no training in the field. That's just probably not true. My guess is that I'm getting these errors because my own brain is drawing "obvious" conclusions that weak, simplistic decision theories don't have the intelligence to handle. The only reason it seems obvious to me that chewing gum will decrease the probability that I get cancer is because I don't have introspective access to my brain's decision making. It would be really nice if I could get that confirmed by somebody.
On the Smoking Lesion, P(Cancer|Smoking) != P(Cancer), by hypothesis- since you don't know whether or not you have the gene, your decision to smoke does provide you with new evidence about whether or not you do, and therefore about whether or not you are likely to get cancer. (Crucially, it doesn't cause you to get cancer- all along you either had the gene or didn't. But EDT works based on correlation, not causation.) For example, one way to rig the probabilities is that half of people smoke, and half of those people get cancer, while nobody who doesn't smoke gets cancer. (And, again, smoking doesn't cause cancer.) Then P(Cancer|Smoking)=.5, P(Cancer)=.25, and choosing to smoke is still the right decision.
The one way you can get a correct correlation between your decision and the utility gained is if you refuse to update your belief that you have the gene based on the evidence of your decision. (This is what CDT does.) But this method fails on Newcomb's Problem. EDT wins on Newcomb's Problem precisely because it has you update on who you are based on your decision: P(Kind of Person Who Would One-Box|Decision to One-Box) != P(Kind of Person Who Would One-Box). But if you decide not to update your beliefs about your character traits based on your decision, then this is lost.
One way to solve this is to give the agent access to its own code/knowledge about itself, and have it condition on that knowledge. That is essentially what Wei Dai's Updateless Decision Theory (UDT) does- it doesn't update its knowledge about its code based on its decision (thus the name). UDT still works on the Prisoner's Dilemma, because the agent doesn't know what decision its code will output until it has made the decision, so it still needs to update Omega's (by assumption, accurate) prediction of its output based on its actual output. (Decision theories, in general, can't condition on their output- doing so leads to the five-and-ten problem. So UDT doesn't start out knowing whether it is the kind of agent that one-boxes or two-boxes- though it does start out knowing its own (genetic or computer) code.)
(This comment would probably be significantly improved by having diagrams. I may add some at some point.)
On the Smoking Lesion, P(Cancer|Smoking) != P(Cancer), by hypothesis
Correct: P(Cancer|Smoking) > P(Cancer). When I said P(O|A) = P(O), I was using A to denote "the decision to not smoke, for the purpose of avoiding cancer." And this is given by the hypothesis of Smoker's lesion. The whole premise is that once we correct for the presence of the genetic lesion (which causes both love of smoking and cancer) smoking is not independently associated with cancer. This also suggests that once we correct for love of smoking, smoking is not independently associated with cancer. So if you know that your reason for (not) smoking has nothing to do with how much you like smoking, then the knowledge that you (don't) smoke doesn't make it seem any more (or less) likely that you'll get cancer.
Ah, I see.
Unfortunately, "the decision to not smoke, for the purpose of avoiding cancer" and "the decision not to smoke, for any other reason" are not distinct actions. The actions available are simply "smoke" or "not smoke". EDT doesn't take prior information, like motives or genes, into account.
You can observe your preferences and hence take them into account.
Suppose that most people without the lesion find smoking disgusting, while most people with the lesion find it pleasurable. The lesion doesn't affect your probability of smoking other than by affecting that taste.
The EDT says that should smoke if you find it pleasurable and you shouldn't if you find it disgusting.
EDT doesn't take prior information, like motives or genes, into account.
Is that explicitly forbidden by some EDT axiom? It seems quite natural for an EDT agent to know its own motives for its decision.
Figuring out what your options are is a hard problem for any decision theory, because it goes to the heart of what we mean by "could". In toy problems like this, agents just have their options spoon-fed to them. I was trying to show that EDT makes the sensible decision, if it has the right options spoon-fed to it. This opens up at least the possibility that a general EDT agent (that figures out what its options are for itself) would work, because there's no reason, in principle, why it can't consider whether the statement "I decided to not smoke, for the purpose of avoiding cancer" would be good news or bad news.
Recognizing this as an option in the first place is a much more complicated issue. But recognizing "smoke" as an option isn't trivial either. After all, you can't smoke if there are no cigarettes available. So it seems to me that, if you're a smoker who just found out about the statistics on smoking and cancer, then the relevant choice you have to make is whether to "decide to quit smoking based on this information about smoking and cancer."
(edited for clarity)
With all the exotic decision theories floating around here, it doesn't seem like anyone has tried to defend boring old evidential decision theory since AlexMennen last year. So I thought I'd take a crack at it. I might come off a bit more confident than I am, since I'm defending a minority position (I'll leave it to others to bring up objections). But right now, I really do think that naive EDT, the simplest decision theory, is also the best decision theory.
Everyone agrees that Smoker's lesion is a bad counterexample to EDT, since it turns out that smoking actually does cause cancer. But people seem to think that this is just an unfortunate choice of thought experiment, and that the reasoning is sound if we accept its premise. I'm not so convinced. I think that this "bad example" provides a pretty big clue as to what's wrong with the objections to EDT. (After all, does anyone think it would have been irrational to quit smoking, based only on the correlation between smoking and cancer, before randomized controlled trials were conducted?) I'll explain what I mean with the simplest version of this thought experiment I could come up with.
Suppose that I'm a farmer, hoping it will rain today, to water my crops. I know that the probability of it having rained today, given that my lawn is wet, is higher than otherwise. And I know that my lawn will be wet, if I turn my sprinklers on. Of course, though it waters my lawn, running my sprinklers does nothing for my crops out in the field. Making the ground wet doesn't cause rain; it's the other way around. But if I'm an EDT agent, I know nothing of causation, and base my decisions only on conditional probability. According to the standard criticism of EDT, I stupidly turn my sprinklers on, as if that would make it rain.
Here is where I think the criticism of EDT fails: how do I know, in the first place, that the ground being wet doesn't cause it to rain? One obvious answer is that I've tried it, and observed that the probability of it raining on a given day, given that I turned my sprinklers on, isn't any higher than the prior probability. But if I know that, then, as an evidential decision theorist, I have no reason to turn the sprinklers on. However, if all I know about the world I inhabit are the two facts: (1) the probability of rain is higher, given that the ground is wet, and (2) The probability of the ground being wet is higher, given that I turn the sprinklers on - then turning the sprinklers on really is the rational thing to do, if I want it to rain.
This is more clear written symbolically. If O is the desired Outcome (rain), E is the Evidence (wet ground), and A is the Action (turning on sprinklers), then we have:
(In this case, A implies E, meaning P(E|A) = 1)
It's still possible that P(O|A) = P(O). Or even that P(O|A) < P(O). (For example, the prior probability of rolling a 4 with a fair die is 1/6. Whereas the probability of rolling a 4, given that you rolled an even number, is 1/3. So P(4|even) > P(4). And you'll definitely roll an even number if you roll a 2, since 2 is even. So P(even|2) > P(even). But the probabilty of rolling a 4, given that you roll a 2, is zero, since 4 isn't 2. So P(4|2) < P(4) even though P(4|even) > P(4) and P(even|2) > P(even).) But in this problem, I don't know P(O|A) directly. The best I can do is guess that, since A implies E, therefore P(O|A) = P(O|E) > P(O). So I do A, to make O more likely. But if I happened to know that P(O|A) = P(O), then I'd have no reason to do A.
Of course, "P(O|A) = P(O)" is basically what we mean, when we say that the ground being wet doesn't cause it to rain. We know that making the ground wet (by means other than rain) doesn't make rain any more likely, either because we've observed this directly, or because we can infer it from our model of the world built up from countless observations. The reason that EDT seems to give the wrong answer to this problem is because we know extra facts about the world, that we haven't stipulated in the problem. But EDT gives the correct answer to the problem as stated. It does the best it can do (the best anyone could do) with limited information.
This is the lesson we should take from Smoker's lesion. Yes, from the perspective of people 60 years ago, it's possible that smoking doesn't cause cancer, and rather a third factor predisposes people to both smoking and cancer. But it's also possible that there's a third factor which does the opposite: making people smoke and protecting them from cancer - but smokers are still more likely to get cancer, because smoking is so bad that it outweighs this protective effect. In the absense of evidence one way or the other, the prudent choice is to not smoke.
But if we accept the premise of Smoker's lesion: that smokers are more likely to get cancer, only because people genetically predisposed to like smoking are also genetically predisposed to develop cancer - then EDT still gives us the right answer. Just as with the Sprinkler problem above, we know that P(O|E) > P(O), and P(E|A) > P(E), where O is the desired outcome of avoiding cancer, E is the evidence of not smoking, and A is the action of deciding to not smoke for the purpose of avoiding cancer. But we also just happen to know, by hypothesis, that P(O|A) = P(O). Recognizing A and E as distinct is key, because one of the implications of the premise is that people who stop smoking, despite enjoying smoking, fair just as badly as life-long smokers. So the reason that you choose to not smoke matters. If you choose to not smoke, because you can't stand tobacco, it's good news. But if you choose to not smoke to avoid cancer, it's neutral news. The bottom line is that you, as an evidential decision theorist, should not take cancer into account when deciding whether or not to smoke, because the good news that you decided to not smoke, would be cancelled out by the fact that you did it to avoid cancer.
If this is starting to sound like the tickle defense, rest assured that there is no way to use this kind of reasoning to justify defecting on the Prisoner's dilemma or two-boxing on Newcomb's problem. The reason is that, if you're playing against a copy of yourself in Prisoner's dilemma, it doesn't matter why you decide to do what you do. Because, whatever your reasons are, your duplicate will do the same thing for the same reasons. Similarly, you only need to know that the predictor is accurate in Newcomb's problem, in order for one-boxing to be good news. The predictor might have blind spots that you could exploit, in order to get all the money. But unless you know about those exceptions, your best bet is to one-box. It's only in special cases that your motivation for making a decision can cancel out the auspiciousness of the decision.
The other objection to EDT is that it's temporally inconsistent. But I don't see why that can't be handled with precommitments, because EDT isn't irreparably broken like CDT is. A CDT agent will one-box on Newcomb's problem, only if it has a chance to precommit before the predictor makes its prediction (which could be before the agent is even created). But an EDT agent one-boxes automatically, and pays in Counterfactual Mugging as long as it has a chance to precommit before it finds out whether the coin came up heads. One of the first things we should expect a self-modifying EDT agent to do, is to make a blanket precommitment for all such problems. That is, it self-modifies in such a way that the modification itself is "good news", regardless of whether the decisions it's precommitting to will be good or bad news when they are carried out. This self-modification might be equivalent to designing something like an updateless decision theory agent. The upshot, if you're a self-modifying AI designer, is that your AI can do this by itself, along with its other recursive self-improvements.
Ultimately, I think that causation is just a convenient short-hand that we use. In practice, we infer causal relations by observing conditional probabilities. Then we use those causal relations to inform our decisions. It's a great heuristic, but we shouldn't lose sight of what we're actually trying to do, which is to choose the option such that the probability of a good outcome is highest.