Thanks Viliam and weathersystems! Sorry it took me a little while to respond, I wanted to make sure I had read and understood the pointers to the related work you guys provided.

Ya definitely don't do that. If you did that you'd just spend all your time donating sperm or something.

If you liked the article, you might want to try the whole book. It will also answer your question about replacing your goals with inclusive genetic fitness. (Spoiler: bad idea, evolution is not your friend.)

I spent some time digging deeper into inclusive fitness. Social Evolution and Inclusive Fitness Theory: An Introduction by James A. R. Marshall provides a good summary.

There are indeed proofs which show that evolution selects for those individuals that always maximize inclusive genetic fitness in the moment. That said, these proofs assume that maximizing offspring in the moment won't hurt your chances of creating offspring in the future. 

So these proofs don't really apply to the situations that humans find themselves in. Raping the nearest person may give you the highest inclusive genetic fitness in the moment, but you will go to jail and won't be able to reproduce any further, so this behavior won't be favored by evolution in the long run (thank god).

So yeah, definitely don't maximize short term inclusive genetic fitness.

So what about maximizing inclusive genetic fitness in the long term, say 1 billion years? I couldn't find any papers analyzing what evolution would do to a strategy like that, but it intuitively sounds a lot better.  If that was your terminal goal, you would probably want to push science forward, advance technology, spread the human race as far out into the universe as you can, etc. Honestly, those sound like pretty reasonable things I'd be happy to support.

1. You're never gonna be able to do it in a way that covered all edge cases.

I completely agree that this is a possible (even likely) scenario. I really don't know, that's why I'm asking :) 

2. Even if you were able to come up with the "right" system, you wouldn't actually be able to follow it. Because our actual motivational systems aren't simple rule following systems. You're gonna want what you want, even if your predetermined system says to do otherwise.

3.  You don't really get to decide what your terminal values are.  I mean you can fudge it a bit, but you certainly don't have complete control over them (and thank god).

I agree that it's going to be impossible to completely change my behavior, just because I change my value system. e.g. no matter what the terminal goal would be, I'd still spend many hours a day sleeping, I'd still need narcotics to go through surgery, and I still wouldn't be able to eat food that tastes absolutely terrible.

That said, I'd say there's like 80% of my after tax income, 100% of my wealth, and maybe 12 hours a day that I can allocate pretty freely. For example, I think there would be a huge difference between living a life where I donate all my income, where I travel the world, where I play video games all day, where I have 12 kids, where I work for a random company 12 hours a day, or where I live on welfare. All of those are actions that I could absolutely take, if I (and maybe my wife) were convinced that they are the right thing to do.

Do you play any board games? In chess there are a lot of what we can call "values." Better to keep your king safe, control the center, don't double your pawns etc. But there's no "small, principled, consistent set of" rules you can use to negotiate between these. It's always gotta be felt out in each new situation.

Completely agree, even if you know the terminal goal, that doesn't mean you'd have the computational capacity to always act optimally. That said, I'd be very happy if I were in a state like chess, i.e. where my terminal goal is very clear, even if I'm not always 100% able to take the perfect action to achieve that goal.

With problems like "my work or my family", the key is usually to get out of this frame and instead think about specific details. 

I love this sentiment. I generally think people underestimate how often you can have the cake and eat it too if you just think a bit outside the box.

Hi weathersystems and Viliam, these are great ways to think about it! I especially like the link to terminal/intrinsic values, that does clarify a lot!

Using some of the terminology from that link, I would say it feels like my brain's built in values are mostly a big subgoal stomp, of mutually contradictory, inconsistent, and changeable values.

While those built in values work pretty well for many situations, e.g. I definitely wouldn't just randomly jump out a window, it seems like they provide very weak guidance for me when making a lot of important decisions, e.g. valuing work vs valuing family (which I both love).

Instead of relying on this subgoal stomp, it feels like my brain has this longing to find a small, principled, consistent set of terminal values that I could use to make decisions instead. 

By way of analogy, maybe think about it like choosing a phone plan. My brain definitely has built in heuristics that would allow me to choose a phone plan (badly), but my brain is so happy that I know math and can use that to make the decision instead.

It sounds like both of you may have gone through the exercise of find terminal goals that work for you.  Would you mind sharing some of findings you had? I think having some examples of what works for other people, might help me a lot on my search.

The article that Viliam pointed out (thanks again!), talks about inclusive genetic fitness as the goal that humans seem to approximate with their actual goals. Inclusive genetic fitness seems like it may be a reasonable terminal goal to replace the subgoal stomp.  Are there resources that provide more details on this topic? (e.g. having the goal of maximizing inclusive fitness in the short term would look very different from maximizing inclusive fitness in the long term).

If you sell the land and improvements separately, you have to accept that you'll probably end up with different owners for the land and improvements. I believe mobile homes are usually sold with a similar system, as the land and the home are usually owned by different people. Might be interesting to study that. I don't have any experience with that system, but the incentives seem pretty bad, e.g. if you have great improvements on the land, the land owner can increase your rent dramatically, knowing that you won't abandon your improvements.