This Can't Go On

The configuration does change, it's just that the change is not about the amount of matter. If there are configurations absurdly more positively or negatively valuable than others, that just makes the ordinary configurations stop being decision relevant, once discerning the important configurations becomes feasible.

Three enigmas at the heart of our reasoning

justify our reasoning

It seems more practical to develop reasoning systems that work well enough to explain what the confusing systems that were never designed happen to be doing. The useful sense of justification is not about describing what's already happening, but about choosing well, making use of what you have available. There is no point in figuring out if what you have available is well-designed, apart from an act of redesign.

"Rational Agents Win"

Consider the distinction between a low level detailed simulation of a world where you are making a decision, and high level reasoning about your decision making. How would you know which one is being applied to you, from within? If there is a way of knowing that, you can act differently in these scenarios, so that the low level simulation won't show the same outcome as the prediction made with high level reasoning. A good process of making predictions by high level reasoning won't allow there to be a difference.

The counterfactual world I'm talking about does not have to exist in any way similar to the real world, such as by being explicitly simulated. It only needs the implied existence of worldbuilding of a fictional story. The difference from a fictional story is that the story is not arbitrary, there is a precise purpose that shapes the story needed for prediction. And for a fictional character, there is no straightforward way of noticing the fictional nature of the world.

"Rational Agents Win"

Therefore, the world where the events take place is not real, it's a figment of past-Omega's imagination, predicting what you'd do. Your actions in the counterfactual just convinced past-Omega not to put the $1M in the box in the real world. Usually you don't have a way of knowing whether the world is real, or what relative weight of reality it holds. Though in this case the reality of the world was under your control.

This Can't Go On

size of the number (representing the amount of value) that can be physically encoded, which is exponential in the number of atoms

The natural numbers that can be physically encoded are not bounded by an exponent of the number of bits if you don't have to be able to encode all smaller numbers as well in the same number of bits. If you define a number, you've encoded it, and it's possible to define very large numbers indeed.

How much should you be willing to pay for an AGI?

Generalizing cousin_it's idea about ems to not-overly-superintelligent AGIs, it's more valuable to run fewer AGIs faster than to run a larger number of AGIs at human speeds. This way AGI-as-a-service might remain in the realm of giant cheesecakes even assuming very slow takeoff.

Oracle predictions don't apply to non-existent worlds

The prediction is why you grab your coat, it's both meaningful and useful to you, a simple counterexample to the sentiment that since correctness scope of predictions is unclear, they are no good. The prediction is not about the coat, but that dependence wasn't mentioned in the arguments against usefulness of predictions above.

Oracle predictions don't apply to non-existent worlds

IMO, either the Oracle is wrong, or the choice is illusory

This is similar to determinism vs. free will, and suggests the following example. The Oracle proclaims: "The world will follow the laws of physics!". But in the counterfactual where an agent takes a decision that won't actually be taken, the fact of taking that counterfactual decision contradicts the agent's cognition following the laws of physics. Yet we want to think about the world within the counterfactual as if the laws of physics are followed.

Oracle predictions don't apply to non-existent worlds

No, I suspect it's a correct ingredient of counterfactuals, one I didn't see discussed before, not an error restricted to a particular decision theory. There is no contradiction in considering each of the counterfactuals as having a given possible decision made by the agent and satisfying the Oracle's prediction, as the agent doesn't know that it won't make this exact decision. And if it does make this exact decision, the prediction is going to be correct, just like the possible decision indexing the counterfactual is going to be the decision actually taken. Most decision theories allow explicitly considering different possible decisions, and adding correctness of the Oracle's prediction into the mix doesn't seem fundamentally different in any way, it's similarly sketchy.

Oracle predictions don't apply to non-existent worlds

Specifically, when you say "It's only guaranteed to be correct on the actual decision", why does the agent not know what "correct" means for the decision?

The agent knows what "correct" means, correctness of a claim is defined for the possible worlds that the agent is considering while making its decision (which by local tradition we confusingly collectively call "counterfactuals", even though one of them is generated by the actual decision and isn't contrary to any fact).

In the post Chris_Leong draws attention to the point that since the Oracle knows which possible world is actual, there is nothing forcing its prediction to be correct on the other possible worlds that the agent foolishly considers, not knowing that they are contrary to fact. And my point in this thread is that despite the uncertainty it seems like we have to magically stipulate correctness of the Oracle on all possible worlds in the same way that we already magically stipulate the possibility of making different decisions in different possible worlds, and this analogy might cast some light on the nature of this magic.

Load More