Wiki Contributions


I see, thanks for this comment. But can humans be considered as possessing an abstract decision making computation? It seems that due to quantum mechanics it's impossible to predict the decision of a human perfectly even if you have the complete initial conditions.

I understand the logic but in a deterministic multiverse the expected utility of any action is the same since the amplitude of the universal wave function is fixed at any given time. No action has any effect on the total utility generated by the multiverse.

I think the fact that the multiverse is deterministic does play a role, since if an agent's utility function covers the entire multiverse and the agent cares about the other branches, its decision theory would suffer paralysis since any action have the same expected utility - the total amount of utility available for the agent within the multiverse, which is predetermined. Utility functions seem to only make sense when constrained to one branch and the agent treats its branch as the sole universe, only in this scenario will different actions have different expected utilities.

But can that really be called acausal "trade"? It's simply the fact that in an infinite multiverse there will be causally independent agents who converge onto the same computation. If I randomly think "if I do X there will exist an agent who does Y and we both benefit in return" and somewhere in the multiverse there will be an agent who does Y in return for me doing X, can I really call that "trade" instead of just a coincidence that necessarily has to occur? But if my actions are determined by a utility function and my utility function extends to other universes/branches then that utility function simply will not work since no matter what action the agent takes, the total amount of utility in the multiverse is conserved. In order for a utility function to give the agent's actions different amounts of expected utility it necessarily has to focus on the single world the agent is in instead of caring about other branches of the multiverse. Therefore shouldn't perfectly rational beings care only about their own branch of the multiverse since that's the only way to have justified actions?

Thanks for the reply! I thought the point of the MWI multiverse is that the wavefunction evolves deterministically according to the Schrodinger equation, so if the utility function takes into account what happens in other universes then it will just output a single fixed constant no matter what the agent experiences, since the amplitude of the universal wave function at any given time is fixed. I think the only way for utility functions to make sense is for the agent to only care about its own branch of the universe and its own possible future observer-moments. Whatever "happens" in the other branches along with their reality measure is predetermined.

Wow. Didn't expect someone from the "rationalist" crowd to do the verbal equivalent of replying clown emojis to tweets you don't like. Your use of all caps really made your arguments so much more convincing. This truly is the pinnacle of human logical discourse: not providing explanations and just ridiculing ideas.

Like I said, "what they want" is irrelevant to the discussion here, you can imagine them wanting virtually anything. The danger lies in understanding the mechanism. You can imagine the alien telling you to order a chocolate ice cream instead of vanilla because that somehow via the butterfly effect yields positive expected utility for them (e.g. by triggering a chain of subtle causal events that makes the AGI we build slightly more aligned with their values or whatever). The problem is that there will also be an alien that wants you to order vanilla instead of chocolate, and who is also fine with applying a negative incentive. Sure, this means you can order whatever flavor of ice cream you want since you will get punished either way, but you're still getting punished (not good). 

The point is "what it wants [us] to do" can essentially be anything we can imagine thanks to the many-gods "refutation" where every possible demand can be imposed on us by some alien on some branch of the quantum multiverse. It can be as ridiculous as leaving your front door open on a Wednesday night or flushing away a straw down a toilet at 3 am, whatever eventually leads to more positive utility to the blackmailer via the butterfly effect (e.g. maybe flushing that straw down the toilet leads to a chain of causal events which makes the utility function of the AGI we build in the future to be slightly more aligned with their goals). "What the alien wants" is irrelevant here, the point is that now you know the mechanism by which aliens can coerce you into doing what they want, and merely knowing so gives other agents increased incentive to acausally extort you. You seem to be hung up on what exactly I'm scared the blackmailer wants me to do, what I am actually worried about is that simply knowing the mechanism imposes danger. The real basilisk is the concept of acausal extortion itself because it opens us up to many dangerous scenarios, not that I am worried about any specific scenario.

The reason why we cannot acausally trade with artificial superintelligences is because we lack the computing power to simulate them accurately, so ASIs would not have any incentive to actually commit to cooperate in a prisoner's dilemma style situation instead of just letting us believe it will while it secretly defects. But we don't have this same problem with non-superintelligences like aliens or even humans who have succeeded in aligning their own AIs, since we can actually simulate such beings in our head. What I am looking for is a concrete argument against this possibility.

The point is that X can essentially be any action, for the sake of the discussion let's say the alien wants you to build an AGI that maximizes the utility function of the alien in our branch of the multiverse.

My main point is that the many-gods refutation is a refutation against taking a specific action, but is not a refutation against the fact that knowing about acausal extortion increases the proportion of bad future observer moments. It in fact makes it worse because, well, now you'll be tortured no matter what you do.

I don't think this would help considering my utter lack of capability to carry out such threats. Are there any logical mistakes in my previous reply or in my concerns regarding the usual refutations as stated in the question? I've yet to hear anyone engage with my points against the usual refutations.

Load More