somewhat confident of Omega's prediction
51% confidence would suffice.
At some point you'll predictably approach death
I'm pretty sure that decision theories are not designed on that basis. We don't want an AI to start making different decisions based on the probability of an upcoming decommission. We don't want it to become nihilistic and stop making decisions because it predicted the heat death of the universe and decided that all paths have zero value. If death is actually tied to the decision in some way, then sure, take that into account, but otherwise, I don't think a decision theory should have "death is inevitably coming for us all" as a factor.
How do you resolve that tension?
Well, as previously stated, my view is that the scenario as stated (single-shot with no precommitment) is not the most helpful hypothetical for designing a decision theory. An iterated version would actually be more relevant, since we want to design an AI that can make more than one decision. And in the iterated version, the tension is largely resolved, because there is a clear motivation to stick with the decision: we still hope for the next coin to come down heads.
what happens when the consequences grow large? Say 1 person to save 500, or 1 to save 3^^^^3?
If 3^^^^3 lives are at stake, and we assume that we are running on faulty or even hostile hardware, then it becomes all the more important not to rely on potentially-corrupted "seems like this will work".
Well, humans can build calculators. That they can't be the calculators that they create doesn't demand an unusual explanation.
Yes, but don't these articles emphasise how evolution doesn't do miracles, doesn't get everything right at once, and takes a very long time to do anything awesome? The fact that humans can do so much more than the normal evolutionary processes can marks us as a rather significant anomaly.
Your decision is a result of your decision theory
I get that that could work for a computer, because a computer can be bound by an overall decision theory without attempting to think about whether that decision theory still makes sense in the current situation.
I don't mind predictors in eg Newcomb's problem. Effectively, there is a backward causal arrow, because whatever you choose causes the predictor to have already acted differently. Unusual, but reasonable.
However, in this case, yes, your choice affects the predictor's earlier decision - but since the coin never came down heads, who cares any more how the predictor would have acted? Why care about being the kind of person who will pay the counterfactual mugger, if there will never again be any opportunity for it to pay off?
Humans can do things that evolutions probably can't do period over the expected lifetime of the universe.
This does beg the question, How, then, did an evolutionary process produce something so much more efficient than itself?
(And if we are products of evolutionary processes, then all our actions are basically facets of evolution, so isn't that sentence self-contradictory?)
there is no distinction between making the decision ahead of time or not
Except that even if you make the decision, what would motivate you to stick to it once it can no longer pay up?
Your only motivation to pay is the hope of obtaining the $10000. If that hope does not exist, what reason would you have to abide by the decision that you make now?
I didn't mean to suggest that the existence of suffering is evidence that there is a God. What I meant was, the known fact of "shared threat -> people come together" makes the reality of suffering less powerful evidence against the existence of a God.
If this was the only chance you ever get to determine your lifespan - then choose B.
In the real world, it would probably be a better idea to discard both options and use your natural lifespan to search for alternative paths to immortality.