TL;DR: Some criticisms aimed at FDT are actually aimed at a self-contradictory "unembedded FDT," and are therefore irrelevant to any refutation of FDT.

Functional Decision Theory (FDT) is a decision theory for a rational agent X who holds the rational belief "the probability that agent Y correctly predicts my actions is very high." Since X is rational, this belief must be grounded in some causal entanglement^[1] — known to X — between X's actions and Y's predictions. But since X's action doesn't precede Y's prediction, both must be caused by something else.

In other words: X's action must be the causal effect of something (call it X's code), and X's code must be knowable to Y, and therefore part of Y's universe, which happens to be the same as X's. In other words: X is an embedded agent.

So FDT only applies to embedded agents. Why does this matter? Because many thought experiments include an unembedded, "God's-eye view" premise. Whenever someone says, "Imagine you're in scenario S, and you know it's not a simulation," they're asking you to imagine a universe that is not itself part of anything bigger. You know there's "nothing else" out there. But knowing that requires you to be "bigger" than the whole thing — which is incompatible with embeddedness.

Does this mean a rational embedded agent must remain agnostic, at all times, about whether they're in a simulation? Not necessarily. The simulations we're familiar with (dreams, etc.) are usually imperfect enough that we can often tell something is off. And we've never heard of anyone who, as in the Hindu story, lived through a fifty-year illusion only to wake up the moment before it began. So when asked, "Do you think you're in a simulation right now?", it isn't absurd to answer, "I'm 99% confident I'm not."^[2]

But the moment we devise a scenario where Omega can predict our actions almost perfectly, the probability that we are inside Omega's simulation goes up. And you can't tack on "but you know you are not in a simulation," or ask "OK, but if you are in the real situation, what do you do?", because the whole point of FDT is to prescribe a policy to an embedded agent who can never know that they are not in a simulation.

A post from yesterday^[3] tries to make FDT palatable to people who, presumably, don't see themselves as embedded agents. This isn't a criticism of the audience! Seeing oneself as an embedded agent is genuinely hard — I fail at it all the time, despite actively trying. But the palatable version of FDT doesn't try to change the audience's understanding of embeddedness, and so it turns out to be itself incompatible with embeddedness.

Here is an embedded reformulation of the Bomb* scenario from that post, along with an embedded-FDT reply:

Facts:

1. You are a rational, embedded agent with the rational beliefs 2, 3, 4, and (5)

2. A hypothetical scenario S is such that doing A maximizes your utility.

3.Omega can predict your behavior in scenario S with a failure rate of one in a trillion trillion.

4. You are faced with an instance of scenario S in which Omega has predicted you won't do A.

There is an implicit premise that needs to be added for this to make sense:

5. You know that Omega isn't changing the rules of the game and/or trying to kill you and/or doing any other non-specified shenanigans.

What do you do?

Embedded FDT's answer: The scenario rules out everything we would normally assume if a similar something like this happened to us in real life: that our estimate of Omega's predictive skill was wrong, that Omega is murdering me for fun, or that we've misunderstood something.

With all of that off the table, the only remaining explanation is that "I" am in a counterfactual simulation — say, a hallucination Omega has induced in me to see what I'd do. So not only does the usual updateless argument for doing A go through, but I might even conclude that my decision is causally connected to what I'll observe when Omega runs the real thing.

1. ^{^}

   Reading "causal entanglement" in an FDT context might come across as strange. Note, however, that I am not talking about how X makes a decision, but about why the statement "Y can predict X's decisions" is true.

2. ^{^}

   I'm not claiming this number is a particularly rational answer to the question — only that it isn't obviously wrong, and that rejecting it would require taking several other things into account.

3. ^{^}

   A post definitely worth reading! I mention it simply because it prompted me to think about this.