interstice

Posts

Sorted by New

Comments

UDT might not pay a Counterfactual Mugger

I don’t think Nomega has to simulate you interacting with Omega in order to know how to would react should you encounter it

By 'simulating' I just mean that it's reasoning in some way about your behavior in another universe, it doesn't have to be a literal simulation. But the point remains -- of all the ways that Nomega could choose to act, for some reason it has chosen to simulate/reason about your behavior in a universe containing Omega, and then give away its resources depending on how it predicts you'll act.

What this means is that, from a Kolmogorov complexity perpective, Nomega is strictly more complex than Omega, since the definition of Nomega includes simulating/reasoning about Omega. Worlds containing Nomega will be discounted by a factor proportional to this additional complexity. Say it takes 100 extra bits to specify Nomega. Then worlds containing Nomega have less measure under the Solomonoff prior than worlds with Omega, meaning that UDT cares much less about them.

(My comment above was reasoning as if Nomega could choose to simulate/reason about many different possible universes, not just the ones with Omega. Then, perhaps, its baseline complexity might be comparable to Omega. Either way, the result is that the worlds where Nomega exists and you have influence don't have very high measure)

This argument would also suggest that by the time you see tails, you know you live in the tails world and thus should not pay up.

What I meant by "Nomega world" in that paragraph was a world where Nomega exists but does not simulate/reason about your behavior in the Omega world. The analogous situation to the tails/heads world here is the "Omega"/"Nomega simulating omega" world. I acknowledge that you would have counterfactual influence over this world. The difference is that the heads/tails worlds have equal measure, whereas the "Nomega simulates omega" world has much less measure than the Omega world(under a 'reasonable' measure such as Solomonoff)

UDT might not pay a Counterfactual Mugger

UDT's behavior here is totally determined by its prior. The question is which prior is more reasonable. 'Closeness to Solomonoff induction' is a good proxy for reasonableness here.

I think a prior putting greater weight on Omega, given that one has seen Omega, is much more reasonable. Here's the reasoning. Let's say that the description complexity of both Omega and Nomega is 1000 bits. Before UDT has seen either of them, it assigns a likelihood of to worlds where either of them exist. So it might seem that it should weight them equally, even having seen Omega.

However, the question then becomes -- why is Nomega choosing to simulate the world containing Omega? Nomega could choose to simulate any world. In fact, a complete description of Nomega's behavior must include a specification of which world it is simulating. This means that, while it takes 1000 bits to specify Nomega, specifying that Nomega exists and is simulating the world containing Omega actually takes 2000 bits.[1]

So UDT's full prior ends up looking like:

  • 999/1000: Normal world

  • : Omega exists

  • : Nomega exists

  • : Nomega exists and is simulating the world containing Omega

Thus, in a situation where UDT has seen Omega, it has influence over the Omega world and Nomega/Omega world, but no influence over the normal world and Nomega world. Since the Omega world has so much more weight than the Omega/Nomega world, UDT will effectively act as if it's in the Omega world.


  1. You might object that Nomega is defined by its property of messing with Omega, so it will naturally simulate worlds with Omega. In that case, it's strictly more complex to specify than Omega, probably by several hundred bits due to the complexity of 'messing with' ↩︎

UDT might not pay a Counterfactual Mugger

Although UDT is formally updateless, the 'mathematical intuition module' which it uses to determine the effects of its actions can make it effectively act as though it's updating.

Here's a simple example. Say UDT's prior over worlds is the following:

  • 75% chance: you will see a green and red button, and a sign saying "press the red button for $5"

  • 25% chance: same buttons, but the sign says "press the green button for $5"

Now, imagine the next thing UDT sees is the sign saying that it should press the green button. Of course, what it should do is press the green button(assuming the signs are truthful), even though in expectation the best thing to do would be pressing the red button. So why does it do this? UDT doesn't update -- it still considers the worlds where it sees the red button to be 3X more important -- however, what does change is that, once it sees the green button sign, it no longer has any influence over the worlds where it sees the red button sign. Thus it acts as though it's effectively updated on seeing the green button sign, even though its distribution over worlds remains unchanged.

By analogy, in your scenario, even though Omega and Nomega might be equally likely a priori, UDT's influence over Omega's actions is far greater given that it has actually seen Omega. Or to be more precise -- in the situation where UDT has both seen Omega and the coin comes up heads, it has a lot of predictable influence over Omega's behavior in a(equally valuable by its prior) world where Omega is real and the coin comes up tails. It has no such predictable influence over worlds where Nomega exists.

UDT might not pay a Counterfactual Mugger

That’s exactly what others are saying about priors

It's not the same thing. Other people are correctly pointing out that UDT's behavior here depends on the prior. I'm arguing that a prior similar to the one we use in our day-to-day lives would assign greater probability to Omega than Nomega, given that one has seen Omega. The OP can be seen as implicitly about both issues.

UDT might not pay a Counterfactual Mugger

I think there's a big asymmetry between Omega and Nomega here, namely that Omega actually appears before you, while Nomega does not. This means there's much better reason to think that Omega will actually reward you in an alternate universe than Nomega.

Put another way, the thing you could pre-commit to could be a broad policy of acausally cooperating with beings you have good reason to think exist, in your universe or a closely adjacent one(adjacent in the sense that your actions here actually have a chance of effecting things there) Once you learn that a being such as Omega exists, then you should act as though you had pre-committed to cooperating with them all along.

Misalignment and misuse: whose values are manifest?

The strange new values that were satisfied were those of the AI systems, but the entire outcome only happened because people like Bob chose it knowingly (let’s say). Bob liked it more than the long glorious human future where his business was less good

I think a relevant consideration here is that Bob doesn't actually have the ability to choose between these two futures -- rather his choice seems to be between a world where his business succeeds but AI takes over later, or a world where his business fails but AI takes over anyway(because other people will use AI even if he doesn't). Bob might actually prefer to sign a contract forbidding the use of AI if he knew that everybody else would be in on it. I suspect that this would be the position of most people who actually thought AI would eventually take over, and that most people who would oppose such a contract would not think AI takeover is likely(perhaps via self-deception due to their local incentives, which in some ways is similar to just not valuing the future)

Ethics in Many Worlds

I think you're misusing the word 'real' here. We only think QM is 'real' in the first place because it predicts our experimental results, so it seems backwards to say that those (classical, probabilistic) results are actually not real, while QM is real. What happens if we experimentally discover a deeper layer of physics beneath QM, will you then say "I thought QM was real, but it was actually fake the whole time"? But then, why would you change your notion of what 'real' is in response to something you don't consider real?

Ethics in Many Worlds

I think some notion of prediction/observation has to be included for a theory to qualify as physics. By your definition, studying the results of e.g. particle accelerator experiments wouldn't be part of quantum mechanics, since you need the Born rule to make predictions about them.

Ethics in Many Worlds

I mean that it correctly predicts the results of experiments and our observations -- which, yes, would be different if we were sampled from a different measure. That's the point. I'm taking for granted that we have some pre-theoretical observations to explain here, and saying that the Hilbert measure is needed to explain them.

Ethics in Many Worlds

Why should sampling weight (you're more likely to find yourself as a real vs Boltzmann brain, or 'thick' vs 'arbitrary' computation) imply ethical weight (the experiences of Boltzmann brains matter far less than real brains)?

I think the weights for prediction and moral value should be the same or at least related. Consider, if we're trying to act selfishly, then we should make choices that lead to the best futures according to the sampling weight(conditioned on our experience so far), since the sampling weight is basically defined as our prior on future sense experiences. But then it seems strange to weigh other peoples' experiences differently than our own.

So in order to think that minds matter in proportion to the measure of the world they're in, while recognizing they 'feel' precisely the same, it looks like you end up having to say that something beyond what a conscious experience is subjectively like makes an enormous difference to how much it matters morally

I think of the measure as being a generalization of what it means to 'count' experiences, not a property of the experiences themselves. So this is more like how, in utilitarianism, the value of an experience has to be multiplied by the number of people having it to get the total moral value. Here we're just multiplying by the measure instead.

My understanding was that MWI is something like what you get when you don't add a collapse postulate, or any other definitional gubbins at all, to the bare formalism.

People like to claim that, but fundamentally you need to add some sort of axiom that describes how the wave function cashes out in terms of observations. The best you can get is an argument like "any other way of weighting the branches would be silly/mathematically inelegant". Maybe, but you're still gonna have to put it in if you want to actually predict anything. If you want to think of it in terms of writing a computer program, it simply won't return predictions without adding the Born rule(what I'm calling the 'Hilbert measure' here)

Load More