Lanrian

Comments

The strategy-stealing assumption

My impression of commitment races and logical time is that the amount of computation we use in general doesn't matter; but that things we learn that are relevant to the acausal bargaining problems do matter. Concretely, using computation during a competitive period to e.g. figure out better hardware cooling systems should be innocuous, because it matters very little for bargaining with other civilisations. However, thinking about agents in other worlds, and how to best bargain with them, would be a big step forward in logical time. This would mean that it's fine to put off acausal decisions however long we want to, assuming that we don't learn anything that's relevant to them in the meantime.

More speculatively, this raises the issue of whether some things in the competitive period would be relevant for acausal bargaining. For example, causal bargaining with AIs on Earth could teach us something about acausal bargaining. If so, the competitive period would advance us in logical time. If we thought this was bad (which is definitely not obvious), maybe we could prevent it by making the competitive AI refuse to bargain with other worlds, and precommiting to eventually replacing it with a naive AI that hasn't updated on anything that the competitive AI has learned. The naive AI would be as early in logical time as we were, when we coded it, so it would be as if the competitive period never happened.

How to teach things well

There is some research on knowledge graphs as a data-structure, and as a tool in AI. Wikipedia and a bunch of references.

Do you vote based on what you think total karma should be?

I could see this argument going the other way. If a post is loved by 45% of people, and meh to 55% of people, then if everyone use target karma, the meh voters will downvote it to a meh position. As you say, the final karma will become people's median opinion; and the median opinion does not highlight things that minorities love.

However, if everyone votes solely based on their opinion, 45% will upvote the comment, and 55% won't vote at all. That means that it will end up in an overall quite favorable spot, as long as most comments are upvoted by less than half of readers.

I think both systems would have to rely on some people not always voting on everything. The nonTK system relies on there being large variability in how prone people are to voting (which I think exist; beware the typical mind fallacy... maybe another poll on how often people vote?) whereas the TK system relies on people abstaining if they're uncertain about how valuable something is to other people.

How good is humanity at coordination?

Using examples is neat. I'd characterize the problem as follows (though the numbers are not actually representative of my beliefs, I think it's way less likely that everybody dies). Prior:

  • 50%: Humans are relatively more competent (hypothesis C). The probability that everyone dies is 10%, the probability that 5% survive is 20%, the probability that everyone survive is 70%.
  • 50%: Humans are relatively less competent. The probability that everyone survives is 10%, the probability that only 5% survive is 20%, the probability that everyone dies is 70%.

Assume we are in a finite multiverse (which is probably false) and take our reference class to only include people alive in the current year (whether the nuclear war happened or not). (SIA doesn't care about reference classes, but SSA does.) Then:

  • SSA thinks
    • Notice we're in a world where everyone survived (as opposed to only 5%) ->
      • if C is true, the probability of this is 0.7/(0.7+0.2*0.05)=70/71
      • if C isn't true, the probability of this is 0.1/(0.1+0.2*0.05)=10/11
      • Thus, the odds ratio is 70/71:10/11.
    • Our prior being 1:1, the resulting probability is ~52% that C is true.
  • SIA thinks
    • Notice we're alive ->
      • the world where C is true contains (0.7+0.2*0.05)/(0.1+0.2*0.05)=0.71/0.11 times as many people, so the update in favor of C is 71:11 in favor of C.
    • Notice we're in a world where everyone survived (as opposed to only 5%).
      • The odds ratio is 70/71:10/11, as earlier.
    • So the posterior odds ratio is (71:11) x (70/71:10/11)=70:10, corresponding to a probability of 87.5% that C is true.
    • Note that we could have done this faster by not separating it into two separate updates. The world where C is true contains 70/10 times as many people as the world where C is false, which is exactly the posterior odds. This is what I meant when I said that the updates balance out, and this is why SIA doesn't care about the reference classes.

Note that we only care about the number of people surviving after a nuclear accident because we've included them in SSA's reference class. But I don't know why people want to include those in the reference class, and nobody else. If we include every human who has ever been alive, we have a large number of people alive regardless of whether C is true or not, which makes SSA give relatively similar predictions as SIA. If we include a huge number of non-humans whose existence aren't affected by whether C is true or not, SSA is practically identical to SIA. This arbitrariness of the reference class is another reason to be sceptical about any argument that uses SSA (and to be sceptical of SSA itself).

How good is humanity at coordination?

There's probably some misunderstanding, but I'm not immediately spotting it when rereading. You wrote:

Seems like it's "much weaker" evidence [[for X]] if you buy something like SIA, and only a little weaker evidence if you buy something like SSA.

Going by the parent comment, I'm interpreting this as

  • it = "we didn't observe nukes going off"
  • X = "humans are competent at handling dangerous technology"

I think that

  • SIA thinks that "we didn't observe nukes going off" is relatively stronger evidence for "humans are competent at handling dangerous technology" (because SIA ignores observer selection effects, and updates naively).
  • SSA thinks that "we didn't observe nukes going off" is relatively weaker evidence for "humans are competent at handling dangerous technology" (because SSA doesn't update against hypothesis which would kill everyone).

Which seems to contradict what you wrote?

How good is humanity at coordination?

Hm, interesting. This suggests that, if we're in a simulation, nuclear war is relatively more likely. However, all such simulations are likely to be shortlived, so if we're in a simulation, we shouldn't care about preventing nuclear war for longtermist reasons (only for short-termist ones). And if we think we're sufficiently likely to be outside a simulation to make longterm concerns dominate short-termist ones (obligatory reference), then we should just condition on not being in a simulation, and then I think this point doesn't matter.

How good is humanity at coordination?

This argument sounds like it's SSA-ish (it certainly doesn't work for SIA). I haven't personally looked into this, but I think Anders Sandberg uses SSA for his analysis in this podcast, where he claims that taking taking observer selection effects into account changes the estimated risk of nuclear war by less than a factor of 2 (search for "not even twice"), because of some mathy details making use of near-miss statistics. So if one is willing to trust Anders to be right about this (I don't think the argument is written up anywhere yet?) observer selection effects wouldn't matter much regardless of your anthropics.

How good is humanity at coordination?

Disagree. SIA always updates towards hypotheses that allow more people to exist (the Self Indication Assumption is that your own existence as an observer indicates that there are more observerss), which makes for an update that nuclear war is rare, since there will exist more people in the multiverse if nuclear accidents are rare. This exactly balances out the claim about selection effects – so SIA corresponds to the naive update-rule which says that world-destroying activities must be rare, since we haven't seen them. The argument about observer selection effects only comes from SSA-ish theories.

Note that, in anthropic dilemmans, total consequentialist ethics + UDT makes the same decisions as SIA + CDT, as explained by Stuart Armstrong here. This makes me think that total consequentialists shouldn't care about observer selection effects.

This is complicated by the fact that infinities breaks both anthropic theories and ethical theories. UDASSA might solve this. In practice, I think UDASSA behaves a bit like a combination of SSA and SIA, but that it is a bit closer to SIA, but I haven't thought a lot about this.

The New Frontpage Design & Opening Tag Creation!
Here <INSERT LINK TO TAG GUIDELINES> are some rough tag-guidelines about what makes a good tag.

Is there no guideline yet? Or is there supposed to be a link here?

Load More