Thanks Benja. This is a good objection to the argument I make in the 'Rejecting Good Evidence' section of the paper, but I think I can avoid it by formulating BIP* more carefully.

Suppose I’m in a situation in which it currently appears to me as though f-sim = x. In effect, your suggestion is that, in this situation, my evidence can be characterized by the disjunction (A ∨ B). You then reason as follows:

(1) Conditional on A, my credence in SIM should be >= x.

(2) Conditional on B, my credence in SIM should be 1.

(3) So overall, given that A and B are mutually exclusive, my credence in SIM should be >= x.

I accept that this is a valid argument. The problem with it, in my view, is that (A ∨ B) is not a complete description of what my evidence says.

Let V represent the proposition that my evidence regarding f-sim is veridical (i.e., the true value of f-sim is indeed what it appears to be). If A is true, then V is also true. So a more complete description of what my evidence says is (A ∧ V) ∨ (B ∧ ~V).

Now we need to ask: is it true that, conditional on (A ∧ V), my credence in SIM should be >= x?

BIP doesn’t entail that it should be, since BIP takes no account of the relevance of V. And V is surely relevant, since (on the face of it, at least) V is far more likely to be true if I am not simulated (i.e., Cr (V | ~SIM) >> Cr (V | SIM)).

Indeed, if one were to learn that (A ∧ V) is true, one might well rationally assign credence =< x to SIM. However, it’s not important to my argument that one’s credence should be =< x: all that matters is that there is no compelling reason to think that it should be >= x.

In short, then, your argument shows that, conditional on a certain description of what my evidence indicates, my credence in SIM should be >= x. But that description is the not the most complete description available—and we must always use the most complete description available, because we often find in epistemology that incomplete descriptions of the evidence lead to incorrect inferences.

Nevertheless, I think your response does expose an error in the paper. I should have formulated BIP* like this, explicitly introducing V:

BIP*: Cr [SIM | ((f-sim = x) ∧ V) ∨ ((f-sim ≠ x) ∧ ~V)] >= x

When BIP* is formulated like this, it is not entailed by BIP. Yet this is the modified principle Bostrom actually needs, if he wants to recover his original conclusion while rejecting Good Evidence. So I think the overall argument still stands, once the error you point out is corrected.

Thanks, everyone, for your comments on my paper. It’s great to see that it is generating discussion. I think I ought to take this opportunity to give a brief explanation of the argument I make in the paper, for the benefit of those who haven’t read it.

The basic argument goes like this. In the first section, I point out that the ‘Simulation Argument’ invokes (at different stages) two assumptions that I call Good Evidence (GE) and Impoverished Evidence (IE). GE is the assumption that I possess good evidence regarding the true physical limits of computation. IE is the assumption that my current evidence does not support any empirical claims non-neutral with respect to the hypothesis (SIM) that I am simulated—for example, the empirical claim that I possess two physically real human hands.

Although GE and IE may look in tension with one another, they are not necessarily incompatible. We can generate a genuine incompatibility, however, by introducing a third claim, Parity of Evidence (PE), stating that my epistemic access to the facts about my own physical constitution is at least as good as my epistemic access to the facts about the true physical limits of computation. Since GE, IE and PE are jointly incompatible, at least one of them must be false.

My own view (and a common view, I imagine) is that IE is false, while GE and PE are true. But rejecting IE would fatally compromise the Simulation Argument. So I spend most of the paper considering the two alternatives open to Bostrom: i.e., rejecting GE or rejecting PE. I argue that, if Bostrom rejects GE, the Simulation Argument still fails. I then argue that, if he rejects PE, the Simulation Argument succeeds, but it’s pretty hard to see how PE could be false. So neither of these alternatives is particularly promising.

One common response I’ve encountered focusses on GE, and asks: why does Bostrom actually need GE? Surely all he really needs is the conditional assumption that, if my evidence is veridical, then GE is true. This conditional assumption allows him to say that, if my evidence is veridical, then the Simulation Argument goes through in its original form; whereas if my evidence is not veridical because I’m simulated, then I’m simulated—so we just end up at the same conclusion by a different route.

This is roughly the line of response pressed here by Benja and Eliezer Yudkowsky. It’s a very reasonable response to my argument, but I don’t think it works. The quick explanation is that it’s just not true that, conditional on my evidence being veridical, the Simulation Argument goes through in its original form. This is essentially because conditionalizing on my evidence being veridical makes SIM a lot less likely than it otherwise would be, and this vitiates the indifference-based reasoning on which the Simulation Argument is based. But Benja is right to press me on the formal details here, so I’ll reply to his objection in a separate comment.