Useful as an example of the difficulty of building consequentialists- "we can't even solve chess, for crying out loud!"- but I don't see it as particularly useful as an explanation of human values.
Although, sifting it more I think I may see the gemstones that you may be seeing. The value seems to be in saying "the goal of chess is completely described: be in a position where your enemy cannot prevent you from capturing his king" and "the goal of evolution is completely described: maximize inclusive genetic fitness", and then comparing the subgoals explicitly. Status, say, is the analogy of board position- it only leads to higher genetic fitness on average in some broad way, but it's a cheap and effective heuristic for doing so, just like good board position is a cheap and effective (but not guaranteed!) heuristic for winning at chess.
Status, say, is the analogy of board position- it only leads to higher genetic fitness on average in some broad way, but it's a cheap and effective heuristic for doing so, just like good board position is a cheap and effective (but not guaranteed!) heuristic for winning at chess
Yep, basically what I was getting at.
This is just a summary via analogy where I human values come from, as far as I understand it. The expanded version would be Eli's http://lesswrong.com/lw/l3/thou_art_godshatter/.
The basic analogy is to chess-playing programs (at least the basic ones from 40 years ago, the art has progressed since then, but not much). The way they work is basically by examining the branching tree of possible moves; since chess is "too big" to solve completely (find the branch that always leads to winning) by present hardware what these programs do is go to a certain depth and then use heuristics to decide whether the end state is good, such as how many pieces are on its side vs. the enemy side, weighed by their "power" (queen is worth more than pawn) and position (center positions are worth more).
The analogy mapping is as follows: the goal of the game is winning, of evolution is survival of a gene fragment (such as human DNA). Explicit encoding of the goal is not computationally feasible or worthwhile (in terms of the goal itself), so values of certain non-terminal states (in terms of the goal) are explicitly given to the program or to a human; the human/program knows no better than these non-terminal values - they are our values - we are Godshatter.
What do you think?