There is no way for human values to magically jump inside the AI, so if it's not specifically created to reflect them, it won't have them, and whatever the AI ends up with won't come close to human values, because human values are too complex to be resembled by any given structure that happens to be formed in the AI.
I'm not convinced by the claim that human values have high Kolmogorov complexity.
In particular, Eliezer's article Not for the Sake of Happiness Alone is totally at odds with my own beliefs. In my mind, it's incoherent to give anything other than subjective experiences ethical consideration. My own preference for real science over imagined science is entirely instrumental and not at all terminal.
Now, maybe Eliezer is confused about what his terminal values are, or maybe I'm confused about what my terminal values are, or maybe our terminal values are incompatible. In any case, it's not obvious that an AI should care about anything other than the subjective experiences of sentient beings.
Suppose that it's okay for an AI to exclude everything but subjective experience from ethical consideration. Is there then still reason to expect that human values have high Kolmogorov complexity?
I don't have a low complexity description to offer, but it seems to me that one can get a lot of mileage out of the principles "if an individual prefers state A to state B whenever he/she/it is in either of state A or state B, then state A is superior for that individual to state B" and "when faced with two alternatives, the moral alternative is the one that you would prefer if you were going to live through the lives of all sentient beings involved."
Of course "sentient being" is ill-defined and one would have to do a fair amount of work frame the things that I just said in more formal terms, but anyway, it's not clear to me that there's a really serious problem here.
The more AI's preference diverges from ours, the more we lose, and this loss is on astronomic scale (even if preference diverges relatively little).
I totally agree that if the creation of a superhuman AI is going to precede all other existential threats then we should focus all of our resources on trying to get the superhuman AI to be as friendly as possible.
Re: "Is there then still reason to expect that human values have high Kolmogorov complexity?"
Human values are mosly a product of their genes and their memes. There is an awful lot of information in those. However, it is true that you can fairly closely approximate human values - or those of any other creature - by the directive to make as many grandchildren as possible - which seems reasonably simple.
Most of the arguments for humans having complex values appear to list a whole bunch of proximate goals - as though that constitutes evidence.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.