Nice article! I had always thought that Christianity was my only motivation for caring about the outer world. After I de-converted, I was really confused to find that I still cared. I still wanted to donate to EA charities. This brought me some happiness, but after a lot of introspection, I was pretty sure that it would bring me more happiness if I donated only half of the money, and spent the other half of the money on a trip back to Guatemala to visit close friends. Even realizing this, though, I still didn't want to, and it boggled my mind...

Then I started learning about evolution and natural selection, that altruism is partially genetic, and it all cleared up a bit for me. I came to the conclusion that goodness can be an end in itself, just like happiness... and I wrote my first ever LW article about it here. It has a lot of the same ideas as yours but isn't so well focused.

People will tell you that humans always and only ever do what brings them pleasure.

Actually, Paul Dolan's book "Happiness by Design" offers a better theory: that humans are motivated by two broad classes of feelings that we can dub "pleasure" and "purpose", and that both are required for happiness. In practice, of course, each of these broad classes contains many, many, sub-categories of emotion.

In general, human beings are easier to understand if you don't try to treat them as utility maximizers. We don't have only one stamp counter. (Or rather, we don't have only one future-stamp-count-predictor, which is a good and correct point in your article. That is, that humans try to steer towards desirable states. I'm just pointing out that "desirable" includes a variety of metrics, many of which are better described as "purposeful" rather than pleasurable.)

This is the first time I've seen anyone on LW make this point with quite this level of explicitness, and I appreciated reading it.

Part of why it might be a useful message around these parts is that it has interesting implications for simulationist ethics, depending on how you treat simulated beings.

Caring about "the outer world" in the context of simulation links naturally to thought experiments like Nozick's Experience Machine but depending on one's approach to decision theory, it also has implications for simulated torture threats.

The decision theoretic simulated torture scenario (where your subjective experience of making the decision has lots of simulation measure via your opponent having lots of CPU, with the non-complying answer causing torture in all simulated cases) has been kicking around since at least 2006 or so. My longstanding position (if I were being threatened with simulated torture) has always been to care, as a policy, about only "the substrate" universe.

In terms of making it emotionally plausible that I would stick to my guns on this policy, I find it helpful to think about all my copies (in the substrate and in the simulation) being in solidarity with each other on this point, in advance.

Thus, my expectations are that when I sometimes end up experiencing torture for refusing to comply with such a threat, I will get the minor satisfaction of getting a decisive signal that I'm in a simulation, and "taking one for the team". Conversely, when I sometimes end up not experiencing simulated torture it increments my belief that I'm "really real" (or at least in a sim where the Demon is playing a longer and more complex game) and should really keep my eye on the reality ball so that my sisters in the sims aren't suffering in vain.

The only strong argument against doing this is for the special case where I'm being simmed with relatively high authenticity for the sake of modeling what the real me is likely to do in a hostile situation in the substrate... like a wargame sort of thing... and in that case, it could be argued that acting "normally" so the simulation is very useful is a traitorous act to the version of me that "really matters" (who all my reality-focused-copies, in the sim and in the real world, would presumably prefer to be less predictable).

For the most part I discount the wargame possibility in practice, because it is such a weirdly paranoid setup that it seems to deserve to be very discounted. (Also, it would be ironic if telling me that I might be in an enemy run wargame sim makes the me that counts the most act erratically in case she might be in the sim!)

I feel like the insight that "the outer world matters" has almost entirely healthy implications. Applying the insight to simulationist issues is fun but probably not that pragmatically productive except possibly if one is prone to schizophrenia or some such... and this seems like more of an empiric question that could be settled by psychiatrists than by philosophers ;-)

However, the fact that reality-focused value systems and related decision theories are somewhat determinative for the kinds of simulations that are worth running (and hence somewhat determinative of which simulations are likely to have measure as embeddings within larger systems) seems like a neat trick. Normally the metaphysicians claim to be studying "the most philosophically fundamental thing", but this perspective gives reason to think that the most fundamental thing (even before metaphysics?) might be how decisions about values work :-)

Were you trying to write about psychology or AI? For psychology, you're quite correct, but most naive folk-theories don't make this mistake anyway (because they mistake map for territory, of course). For AI, you're glossing over the yet-to-be-done work of actually designing algorithms that deliberately enforce a correspondence between map and territory with respect to their goals.

I don't disagree that humans can do actions that only benefits others, and that altruism exists. I think there is a better theory than both pleasure-maximizing and "humans are intrinsically nice to others", and that is Evolution. Also, Evolution can be understood as "gene-spread chance maximizing", so I think humans are still better modelled as internal counter maximizer.

Donating to charity can be explained by Signaling, it lets others know that you have an excess of money. Pure altruism alone cannot explain donations because we donate more when we’re being watched. (More detailed explanation of charity can be found in The Elephant in the Brain Chapter 12: Charity.)

Then a mother jumped in front of a car to save her child.

I think that this is a prototypical example in two ways:

1) Descriptive ethics. Describing what people think is right/good/moral. (Actually, I don't think that this is strictly true, but whatever.)

2) Describing how people actually act (cultural anthropology?).

Your main point in this article seems to be related to 2). "People don't only try to seek pleasure."

a) Was that your main point?

b) Do regular people debate this (I'm pretty sure they do, but I'm not positive)? Philosophers? Rationalists? My impression is that rationalists don't debate this, and so I'm not sure who this post is targeting (you did say it's a repost from your blog, so maybe there is indeed a different target audience?).

c) Does this have any implications for what you "should" do? My working conclusion is that "should requires an axiom". That terminal values are arbitrary, and you could only say that you "should" do something to the extent that it leads to a chosen terminal value (or blend of terminal values).

(If this post is only about 2), then the following is tangential, and perhaps isn't the right place for this. But anyway...)

I really don't find "terminal values are arbitrary" to be a comfortable conclusion. I'm not exactly sure why I find it to be so uncomfortable.

• Like most people, I seek "purpose". Some sort of absolute feeling that the goal I'm pursuing is "the right goal". In other words, I have a desire to find and pursue the "right goal", even though I think/understand that terminal values are arbitrary.
• Intellectually/logically, I think I have a good understanding of the ideas of consequentialism. I think I understand the reasoning behind the idea that terminal values are arbitrary. But maybe there are holes in my understanding that are causing the discomfort. Or better yet, maybe my conclusion that terminal values are arbitrary is wrong.
• I'm not sure what my terminal values are. But there's a very large part of me that only cares about my own pleasure (fortunately, acting altruistically brings me a good amount of pleasure). But the implications of that are pretty scary. For example, someone who legitimately only cares about his own pleasure would chose to kill everyone in the world if it meant that he'd survive and be happy. Logically, I don't see a problem here.
  • Consider the question of "What will lead to your terminal goal?". In the hypothetical, this is already answered for us.
  • Consider the question of "Well, is that a good terminal goal?". Logically, it seems to me that there's no such thing as a "good terminal goal".
  • But emotionally, I feel like there's a huge problem here. Unfortunately, when I examine this feeling, I find that there isn't good reason behind it. If you're trying to achieve your terminal goals, then a good reason for an emotion is because it helps you achieve your terminal goals. If the emotion isn't helping you achieve your goals, then I'd say that there isn't good reason behind it. It seems to me that these emotions aren't helping to achieve the terminal goal of personal happiness, and that these emotions are the result of an imperfect brain.
  • I suppose you could argue that those sorts of emotions help you function in society and be happy. That the consequences of such feelings of guilt are decreased likelihood of being shunned and an increased likelihood of being accepted, both of which make it more likely that you survive and live happily. But that doesn't seem to be the case with my feeling guilty for (possibly/hypothetically) being selfish. If I didn't feel this guilt, I highly doubt anyone would know.
  • I suppose that you could subsequently argue that our brains aren't designed for this. That we don't get to be altruistic in the vast majority of circumstances (where other peoples' utility is linked with your own short/long-run utility), but simultaneously be able to feel no guilt for choosing your own life over everyone else's. "But why not?! Doing so seems to be the strategy that would maximize your own utility."

One point I notice here is that "value" is in the map, not the territory. The whole notion that agents have values is a way of modeling real-world things; it isn't a principle on which the world rests.

For instance, people sometimes make big, definitive-sounding claims about "human terminal values", but these are basically (pretty rough) attempts to create a map that might be useful for predicting human behavior. Insofar as that map isn't predictive, it's worth amending or discarding.