Because of deception, we don't know how to put a given utility function into a smart agent that has grokked the overall picture of its training environment. Once training finds a smart-enough agent, the model's utility functions ceases to be malleable to us. This suggests that powerful greedy search will find agents with essentially random utility functions.

But, evolution managed to push human values in the rough direction of its own values: inclusive genetic fitness. We don't care about maximizing inclusive genetic fitness, but we do care about having sex, having kids, protecting family, etc.

When the blind idiot god created protein computers, its monomaniacal focus on inclusive genetic fitness was not faithfully transmitted. Its optimization criterion did not successfully quine. We, the handiwork of evolution, are as alien to evolution as our Maker is alien to us. One pure utility function splintered into a thousand shards of desire.

Why? Above all, because evolution is stupid in an absolute sense. But also because the first protein computers weren't anywhere near as general as the blind idiot god, and could only utilize short-term desires.

How come humans don't have a random utility function that's even more out of line with optimizing for inclusive genetic fitness? Because of the exact degree to which our ancestral protein algorithms were stupid. If our ancestors were much smarter, they might have overridden evolution while having just about any utility function. In our world, evolution got to mold our utility function up until it got anatomically modern Homo sapiens, who then -- very quickly from evolution's perspective -- assumed control.