Thank you, this is good to know. I'll have to think about this some more.
Hm, I was working under the assumption that the "utility" with paperclips was just the number of paperclips. A universe with X - 10n + 3^^^^3 paperclips is better than a universe with just X paperclips by 3^^^^3 - 10n. Is this not a proper utility function?
The casino version evolved from repeated alterations to Pascal's Mugging, so it retained the 3^^^^3 from there. I had written a paragraph where I mentioned that for one-shot problems, even a more realistic probability could qualify as a Pascal's Mugging, though I had used a 1/million chance of a trillion paperclips instead of 1/100. I ended up editing that paragraph out, though.
Working with a 1/100 probability, it's less obviously a bad idea to pay up, of course. I don't know where to draw the line between "this is a Pascal's Mugging" and "this is good odds", so I'm less confident that you shouldn't pay up for a 1/100 probability. I think it becomes a more obviously bad idea if we up the price of the casino, for example to 1 million paperclips. This still gives positive EU to paying, but has a fairly steep price compared to doing nothing unless you get pretty lucky.
Looking back, I think that one of the factors in my decision to retain such ludicrous numbers was that it seemed more persuasive. I apologise for this.
All that being said, thank you very much for your reply!
Very interesting, thank you!
I think "maximising" still makes sense in one-shot problems. 2>1 and 1000>1, but it's also the case that 1000>2, even without expected utility. The way I see it, EU is a method of comparing choices based on their average utility, but the "average" turns out to be a less useful metric when you only have one chance.
So for cases when an outcome is not a constant amount of paperclips we need more rules than what the object of attention is. So a paperclip maximiser is actually underspecified.
If this is true, it would imply that in a one-shot problem, a utility function is not enough on its own to determine what is the "optimal" choice when you want to "maximise" (get the highest value you can) on that utility function. This would be a pretty big result, I think.
I think that if there is a part that is underspecified, though, it's not the paperclip maximiser, but the word "optimal". What does it mean for a choice to be "optimal" relative to other choices, when it might turn out better or worse depending on luck? I haven't been able to answer that question.
Thank you for your response!
Are you rejecting Pascal’s mugging because of the prospect of relying on uncertain models that you do not expect to confirm?
My intuition is that in a one-shot problem, gambling everything on an extremely low probability event is a bad idea, even when the reward from that low probability event is very high, because you are effectively certain to lose. This is the basis for me not paying up in Pascal's Mugging and in the casino problem in the post.
I'm trying to keep my reasoning simple, so in my examples I always assume that there are no infinities, no unknown unknowns, every outcome of every choice is statistically independent, and all the assigned probabilities are statistically correct (if there is a 1/6 chance of an outcome and you get to repeat the problem, you will get that outcome on average 1/6 of the time).
Is all your intuition captured by maximizing utility over all but the extreme billionth of the distribution?
Honestly, I have no idea how to solve the problem. My intuition is hopelessly muddled on this, and every idea I've been able to come up with seems flawed, including the one you've just asked about.
Here's a one-shot problem for your intuition to answer: You get to design the probability distribution to draw the number of paperclips from, except that its expectation must be at most its negative kolgomorov complexity. What distribution makes for a good choice?
My first thought is 1/googolplex chance of losing 3^^^^3 paperclips, and the rest of the probability giving as many paperclips as the kolmogorov complexity constraint allows. I could do better by increasing the probability of the loss, for example 1/googol would be a better probability. However, I have no idea where to draw the line, at what point it stops being a good idea to increase the probability.
You're absolutely right. I was starting to get at this idea from another of the comments, but you've laid out where I've gone wrong very clearly. Thank you.