entirelyuseless2

I think this post is basically correct. You don't, however, give an argument that most minds would behave this way. However, here is a brief intuitive argument for it. A "utility function" does not mean something that is maximized in the ordinary sense of maximize; it just means "what the thing does in all situations." Look at computers: what do they do? In most situations, they sit there and compute things, and do not attempt to do anything in particular in the world. If you scale up their intelligence, that will not necessarily c... (read more)

This theory is mostly true, but rather than being cynical about people caring about poor people, we should be cynical about the more general concept of people caring about stuff.

This is all basically right.

However, as I said in a recent comment, people do not actually have utility functions. So in that sense, they have neither a bounded nor an unbounded utility function. They can only try to make their preferences less inconsistent. And you have two options: you can pick some crazy consistency very different from normal, or you can try to increase normality at the same time as increasing consistency. The second choice is better. And in this case, the second choice means picking a bounded utility function, and the first choice means choosing an unbounded one, and going insane (because agreeing to be mugged is insane.)

You don't have any such basic list.

I don't think you understood the argument. Let's agree that an electron prefers what it is going to do, over what it is not going to do. But does an electron in China prefer that I write this comment, or a different one?

Obviously, it has no preference at all about that. So even if it has some local preferences, it does not have a coherent preference over all possible things. The same thing is true for human beings, for exactly the same reasons.

I don't know why you think I am assuming this. Regardless of the causes of your opinions, one thing which is not the cause is a coherent set of probabilities. In the same way, regardless of the causes of your actions, one thing which is not the cause is a coherent set of preferences.

This is necessarily true since you are built out of physical things which do not have sets of preferences about the world, and you follow physical laws which do not have sets of preferences about the world. They have something similar to this, e.g. you could metaphorically... (read more)

" It's a list of all our desires and preferences, in order of importance, for every situation ."

This is basically an assertion that we actually have a utility function. This is false. There might be a list of pairings between "situations you might be in" and "things you would do," but it does not correspond to any coherent set of preferences. It corresponds to someone sometimes preferring A to B, and sometimes B to A, without a coherent reason for this.

Asserting that there is such a coherent list would be like asserting t... (read more)

I predict the video was faked (i.e. that everyone in it knows what is happening and that in fact there was not even a test like this.)

Most people, most of the time, state their beliefs as binary propositions, not as probability statements. Furthermore, this is not just leaving out an actually existing detail, but it is a detail missing from reality. If I say, "That man is about 6 feet tall," you can argue that he has an objectively precise height of 6 feet 2 inches or whatever. But if I say "the sky is blue," it is false that there is an objectively precise probability that I have for that statement. If you push me, I might come up with the number. But I am basically ... (read more)

This seems obviously circular, since you depend on using induction based on human languages to conclude that humans were produced by an evolutionary process.

"1) the paperclip maximizer is not a paperclip maximizer but a different kind of unfriendly AI"

Being a paperclip maximizer is about values, not about decision theory. You can want to maximize paperclips but still use some of acausal decison theory that will cooperate with decision makers that would cooperate with paperclippers, as in cousin_it's response.

It's curious that you assume in this discussion that the one is right, and the other is wrong. The opposite happens with equal frequency (this is mathematically necessary, since each is one and each is other.)

" Similarly, this argument for averaging could only make sense if the other person would consider averaging beliefs with you for the same reason. "

No. This is exactly the situation where it would not make sense. If the person has already averaged his belief with others they met, you will end up with a distorted belief. It is percisely when you have two "natural" beliefs that averaging them offers some potential gain.

" I don't see how that conjunction is logically inconsistent?" Suppose you have beliefs A, B, C, and belief D: "At least one of beliefs A, B, C is false." The conjunction of A, B, C, and D is logically inconsistent. They cannot all be true, because if A, B, and C are all true, then D is false, while if D is true, at least one of the others is false. So if you think that you have some false beliefs (and everyone does), then the conjunction of that with the rest of your beliefs is logically inconsistent.

" I think consistency is ... (read more)

" If you are inconsistent, then please fix that. :) "

G.K. Chesterton made a lot of errors which he always managed to state in interesting ways. However, one thing he got right was the idea that a lot of insanity comes from an excessive insistence on consistency.

Consider the process of improving your beliefs. You may find out that they have some inconsistencies between one another, and you might want to fix that. But it is far more important to preserve consistency with reality than interal consistency, and an inordinate insistence on internal con... (read more)

"arbitrary limit"

What would constitute a non-arbitrary limit? A utility function just describes the behavior of a physical being. It is not surprising that a physical being has limits in its behavior -- that follows from the fact that physical law imposes limits on what a thing does. This is why e.g. transhumanist hope for real immortality is absurd. Even if you could find a way around entropy, you will never change the fact that you are following physical laws. The only way you will exist forever is if current physical laws extrapolated from you... (read more)

" Also the lifespan dilemma shows that for many people the answer can’t be just a matter of expected value, otherwise everyone would agree on reducing the probability of success to values near to 0. "

This is a delusion that stems from using the formulation of "expected value" without understanding it. The basic idea of expected value derives from a utility function, which is the effect of being able to give consistent answers to every question of the form, "Would you prefer X to Y, or Y to X? Or does it just not matter?" Onc... (read more)

" that doesn't make their strongest beliefs the most trustworthy -- indeed, they're the ones we ought to downgrade the most"

Not likely. Their strongest beliefs will be the most trustworthy, even though they are downgraded the most, because they start start out higher. It would be a very unlikely calibration graph indeed which assigned a lower probability to 99.9% assigned probabilities than to 95% assigned probabilities.

Great post. I think Robin Hanson would accept that "older people are wiser" should tend to be true, even in relation to himself, but he also accepts that number 9 applies to himself, so it wouldn't bother him that he hasn't corrected his beliefs in particular ways yet.