Complexity of Value

Many human choices can be compressed, by representing them by simple rules - the desire to survive produces innumerable actions and subgoals as we fulfill that desire. But people don'don't just want to survive - although you can compress many human activities to that desire, you cannot compress all of human existence into it. The human equivalents of a utility function, our terminal values, contain many different elements that are not strictly reducible to one another. William Frankena offered this list of things which many cultures and people seem to value (for their own sake rather than strictly for their external consequences):

Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one'one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc.

The "etc.""etc." at the end is the tricky part, because there may be a great many values not included on this list.

SinceOne hypothesis is that natural selection reifies selection pressures as psychological drivesdrives, which then continue to execute independently of any consequentialist reasoning in the organism or. This may also continue without that organism explicitly representing, let alone caring about, the original evolutionary context,context. Under this view, we have no reason to expect these terminal values to be reducible to any one thing, or each other.

Complexity of value also runs into underappreciation in the presence of bad metaethics. The local flavor of metaethics could be characterized as cognitivist, without implying "thick""thick" notions of instrumental rationality; in other words, moral discourse can be about a coherent subject matter, without all possible minds and agents necessarily finding truths about that subject matter to be psychologically compelling. An expected paperclip maximizer doesn'doesn't disagree with you about morality any more than you disagree with it about "which"which action leads to the greatest number of expected paperclips"paperclips", it is just constructed to find the latter subject matter psychologically compelling but not the former. Failure to appreciate that "But it'"But it's just paperclips! What a dumb goal! No sufficiently intelligent agent would pick such a dumb goal!"" is a judgment carried out on a local brain that evaluates paperclips as inherently low-in-the-preference-ordering means that someone will expect all moral judgments to be automatically reproduced in a sufficiently intelligent agent, since, after all, they would not lack the intelligence to see that paperclips are so obviously inherently-low-in-the-preference-ordering. This is a particularly subtle species of anthropomorphism and mind projection fallacy.

As...

Complexity of value poses a problem for AI alignment. If you can't easily compress what humans whatwant into a simple function that can be fed into a computer, it isn't easy to make a powerful AI that does things humans want and doesn't do things humans don't want. Value Learning attempts to address this problem.

Complexity of Valuevalue refersis the thesis that human values have high Kolmogorov complexity; that our preferences, the things we care about, cannot be summed by a few simple rules, or compressed. Fragility of value is the thesis that losing even a small part of the rules that make up our values could lead to the vast diversityresults that most of different things that humans value. If value is also fragile,us would now consider as unacceptable (just like dialing nine out of ten phone digits correctly does not connect you to a person 90% similar to your friend). For example, all of our values except novelty might yield a future full of individuals replaying only one optimal experience through all eternity.

Related: Ethics & Metaethics, Fun Theory, Preference, Wireheading

Many human choices can be compressed, by representing them by simple rules - the desire to survive produces innumerable actions and subgoals as we fulfill that lacks anydesire. But people don't just want to survive - although you can compress many human activities to that desire, you cannot compress all of human existence into it. The human equivalents of a utility function, our terminal values, contain many different elements that are not strictly reducible to one of those things might be almost completely worthless.

Philosopheranother. William Frankenna created a non-comprehensiveFrankena offered this list of things that humanswhich many cultures and people seem to value intrinsically(for their own sake rather than strictly for their external consequences):

Since natural selection reifies selection pressures as psychological drives which then continue to executeindependently of any consequentialist reasoning in the organism or that organism explicitly representing, let alone caring about, the original evolutionary context, we have no reason to expect these terminal values to be reducible to any one thing, or each other.

Taken in conjunction with another LessWrong claim, that all values are morally relevant, this would suggest that those philosophers who seek to do so are mistaken in trying to find cognitively tractable overarching principles of ethics. However, it is coherent to suppose that not all values are morally relevant, and that the morally relevant ones form a tractable subset.

Complexity of value also runs into underappreciation in the presence of bad metaethics. The local flavor of metaethics could be characterized as cognitivist, without implying "thick" notions of instrumental rationality; in other words, moral discourse can be about a coherent subject matter, without all possible minds and agents necessarily finding truths about that subject matter to be psychologically compelling. An expected paperclip maximizer doesn't disagree with you about morality any more than you disagree with it about "which action leads to the greatest number of expected paperclips", it is just constructed to find the latter subject matter psychologically compelling but not...

Complexity of valueValue refers to the vast diversity of different things that humans value. If value is the thesis that human values have high Kolmogorov complexity; that our preferencesalso fragile, the things we care about, cannot be summed by a few simple rules, or compressed. Fragility of value is the thesis that losing even a small part of the rules that make up our values could lead to results that most of us would now consider as unacceptable (just like dialing nine out of ten phone digits correctly does not connect you to a person 90% similar to your friend). For example, all of our values except novelty might yield a future fullthat lacks any one of individuals replaying only one optimal experience through all eternity.those things might be almost completely worthless.

Many human choices can be compressed, by representing them by simple rules - the desire to survive produces innumerable actions and subgoals as we fulfill that desire. But people don't just want to survive - although you can compress many human activities to that desire, you cannot compress all of human existence into it. The human equivalents ofPhilosopher William Frankenna created a utility function, our terminal values, contain many different elements that are not strictly reducible to one another. William Frankena offered thisnon-comprehensive list of things which many cultures and people seem tothat humans value (for their own sake rather than strictly for their external consequences)intrinsically:

"

Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one'one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."

Since natural selection reifies selection pressures as psychological drives which then continue to executeindependently of any consequentialist reasoning inThe "etc." at the organism or that organism explicitly representing, let alone caring about,end is the original evolutionary context, we have no reason to expect these terminaltricky part, because there may be a great many values to be reducible to any one thing, or each other.

Taken in conjunction with another LessWrong claim, that all values are morally relevant,not included on this would suggest that those philosophers who seek to do so are mistaken in trying to find cognitively tractable overarching principles of ethics. However, it is coherent to suppose that not all values are morally relevant, and that the morally relevant ones form a tractable subset.list.

Complexity of value also runsposes a problem for AI alignment. If you can't easily...

Complexity of value is the thesis that human values have high Kolmogorov complexity; that our preferences, the things we care about, cannot be summed by a few simple rules, or compressed. Fragility of value is the thesis that losing even a small part of the rules that make up our values could lead to results that most of us would now consider as unacceptable (just like dialing nine out of ten phone digits correctly does not connect you to a person 90% similar to your friend). For example, all of our values except boredomnovelty might yield a future full of individuals replaying only one optimal experience through all eternity.

• Value is Fragile
• The Hidden Complexity of Wishes
• Fake Morality
• Welcome to Heaven by denisbider
• Complexity of Value ≠ Complexity of Outcome by Wei Dai
• Not for the Sake of Pleasure Alone by lukeprog
• A Non-Comprehensive List of Human Values

Taken in conjunction with another LessWrong claim, that all values are morally relevant, this would suggest that those philosophers who seek to do so are mistaken in trying to find gognitevlycognitively tractable overarching principles of ethics. However, it is coherent to suppose that not all values are morally relevant, and that the morally relevant ones form a tractable subset.

Unfortunately, human moralTaken in conjunction with another LessWrong claim, that all values are morally relevant, this would suggest that those philosophers - both amateurwho seek to do so are mistaken in trying to find gognitevly tractable overarching principles of ethics. However, it is coherent to suppose that not all values are morally relevant, and professional - havethat the morally relevant ones form a tendency to seek out fake utility functions which argue for desiderate arrived at by other psychological means, in terms of their favorite grand moral principle (which typically varies from person to person, far more than their actual preferred conclusions) and this is one force leading people to systematically underestimate the complexity of value.tractable subset.