By the way, what happens if a billion independent muggers all mug you for 1 dollar, one after another?

The same as if one mugger asks a billion times, I believe. Do you think the probability that a mugger is telling the truth is a billion times as high in the world where 1000 000 000 of them ask the AI versus the world where just 1 asks? If the answer is no, then why would the AI think so?

Why do you think that? What is the probability that the mugger does in fact have exclusive access to 3^^^^3 lives? And what is the probability for 3^^^^^3 lives?

In the section you quoted, I am not saying that other ways of affecting 3^^^^3 lives exist, I am saying that other ways with a non-zero probability of affecting that many lives exist – this is trivial, I think. A way to actually do this does, most likely, not exist.

So there is of course a probability that the mugger does have exclusive access to 3^^^^3 lives. Let's call that $p$ . What I am arguing is that it is wrong to assign a fairly low utility $u$ to 1$ worth of resources and

then conclude "aha, since $p \cdot U ($ 3^^^^3 lives) $> u$ , it must be correct to pay!" And the reason for this is that $u$ is not actually small. Calculating $u$ , the utility of one dollar, does itself include considering various mugging-like scenarios; what if there is just a bit of additional self-improvement necessary to see how 3^^^^3 lives can be saved? It is up to the discretion of the AI to decide when the above formula holds.

So $p$ might be much larger than 1/3^^^^3 but $u$ is actually very large, too. (I am usually a fan of making up specific numbers, but in this case that doesn't seem useful).