Yeah, it matches wat @mesaoptimizer said, I believe. I was reluctant to post my view, but thought it could be helpful anyway :)
Great question! I'm wondering the same thing now. I, for one, had to be reasoned into it. It does feel like it "clicked", so to speak, but I doubt whether anyone has this intuition naturally.
I would be willing to discuss FDT more, if you'd like (in a separate post, of course).
Regarding (2), I interpret Soares' point as follows: there's a "CDT intuition" and an "FDT intuition" of how to evaluate counterfactuals. Let's just take the Bomb problem as an example.
The CDT intuition deals with which action has the best causal consequences at one given point in time. In Bomb, you can either Left-box or Right-box. There's a bomb in Left, so Left-boxing causes you to die painfully. Right-boxing costs only $100, so right-boxing wins.
The FDT intuition deals with which decision is the best outcome of your decision procedure. Your decision procedure is a function, and could be implemented more than once. In Bomb, it's implemented both in your head and in the predictor's head (she executed it to predict what you would do). Your decision to Left-box or to Right-box therefore happens twice - and, since your decision procedure is a function, it's necessarily the same on both events - and you have to look at the causal consequences of both events. Left-boxing causes the predictor to not put a bomb in Left and you to not lose $100, while Right-boxing causes the predictor to put a bomb in Left and you to lose $100. Left-boxing wins.
P.S. A natural thing to respond here is something like: "But you already see a Bomb in Left, so the FDT intuition makes no sense!" But note that, since the predictor simulates you in order to make her prediction, you don't actually know whether you are the "real you" or the "simulated you", since the simulated you observes the exact same (relevant) things as the real you. (If not, then the simulation would not be accurate and there would be no subjunctive dependence.) So in this intuition, observing the bomb does not actually mean there is a bomb, since you could be in a simulation. In fact, you are, at different points in time, both in the simulation and in the "real" situation, and you have to make a decision that happens in and makes the best of both these situations.
"In your problem description you said you receive the letter"
True, but the problem description also specifies subjunctive dependence between the agent and the predictor. When the predictor made her prediction the letter isn't yet sent. So the agent's decision influences whether or not she gets the letter.
"This intuition is actually false for perfect predictors."
I agree (and have written extensively on the subject). But it's the prediction the agent influences, not the presence of the termite infestation.
Given that you receive the letter, paying is indeed evidence for not having termites and winning $999,000. EDT is elegant, but still can't be correct in my view. I wish it were, and have attempted to "fix" it.
My take is this. Either you have the termite infestation, or you don't.
Say you do. Then
Say you don't. Then
Being a payer has the same result when you do have the termites, but is worse when you don't. So overall, it's worse. Being a payer or a non-payer only influences whether or not you get the letter, and this view is more coherent with the intuition that you can't possibly influence whether or not you have a termite infestation.
XOR Blackmail is (in my view) perhaps the clearest counterexample to EDT:
An agent has been alerted to a rumor that her house has a terrible termite infestation that would cost her $1,000,000 in damages. She doesn’t know whether this rumor is true. A greedy predictor with a strong reputation for honesty learns whether or not it’s true, and drafts a letter:
I know whether or not you have termites, and I have sent you this letter iff exactly one of the following is true: (i) the rumor is false, and you are going to pay me $1,000 upon receiving this letter; or (ii) the rumor is true, and you will not pay me upon receiving this letter.
The predictor then predicts what the agent would do upon receiving the letter, and sends the agent the letter iff exactly one of (i) or (ii) is true. Thus, the claim made by the letter is true. Assume the agent receives the letter. Should she pay up?
(Styling mine, not original.) EDT pays the $1,000 for nothing: it has absolutely no influence on whether or not the agent's house is infested with termites.
Alright, thanks for your answer!
Is it necessary to be able to work during all MIRI office hours, or is it enough if my hours are partially compatible? My time difference with MIRI is 9 hours, but I could work in the evening (my time) every now and then.
Why do you believe this is a scenario in which you are definitely not in a simulation?