I don't think I can persuaded.

I have many good responses to the comments here, and I suppose I could sketch out some of the main arguments against anti-realism, but there are also many serious demands on my time and sadly this doesn't look like a productive discussion. There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying (which is particularly alarming given that many commenters are pointing out that they don't understand what the opponents are saying!).

Given all of this, I fear that writing a post would not be a good use of my time.

Alas. Perhaps some Less Wrongers with more time will write and post a hypothetical apostasy. I invite folk to do so.

1DonGeddis10yI, for one, am interested in hearing arguments against anti-realism. If you don't have personal interest in writing up a sketch, that's fine. Might you have some links to other people who have already done so?
3Paul Crowley10yThis is a little unfair; as soon as you take a deflationary stance on anything, you're saying that the other stance doesn't really have comprehensible content, and it's a mistake to turn that into a general-purpose dismissal of deflationary stances. If you think that's more true here than it is in other discussion forums, we're doing something very wrong. I understand that you're not able to spend time writing for this audience, but for those of us who do want to find out more about what moral realists are saying, every link you can provide to existing essays is valuable.

Complexity of Value ≠ Complexity of Outcome

by Wei_Dai 2 min read30th Jan 2010232 comments


Complexity of value is the thesis that our preferences, the things we care about, don't compress down to one simple rule, or a few simple rules. To review why it's important (by quoting from the wiki):

  • Caricatures of rationalists often have them moved by artificially simplified values - for example, only caring about personal pleasure. This becomes a template for arguing against rationality: X is valuable, but rationality says to only care about Y, in which case we could not value X, therefore do not be rational.
  • Underestimating the complexity of value leads to underestimating the difficulty of Friendly AI; and there are notable cognitive biases and fallacies which lead people to underestimate this complexity.

I certainly agree with both of these points. But I worry that we (at Less Wrong) might have swung a bit too far in the other direction. No, I don't think that we overestimate the complexity of our values, but rather there's a tendency to assume that complexity of value must lead to complexity of outcome, that is, agents who faithfully inherit the full complexity of human values will necessarily create a future that reflects that complexity. I will argue that it is possible for complex values to lead to simple futures, and explain the relevance of this possibility to the project of Friendly AI.

The easiest way to make my argument is to start by considering a hypothetical alien with all of the values of a typical human being, but also an extra one. His fondest desire is to fill the universe with orgasmium, which he considers to have orders of magnitude more utility than realizing any of his other goals. As long as his dominant goal remains infeasible, he's largely indistinguishable from a normal human being. But if he happens to pass his values on to a superintelligent AI, the future of the universe will turn out to be rather simple, despite those values being no less complex than any human's.

The above possibility is easy to reason about, but perhaps does not appear very relevant to our actual situation. I think that it may be, and here's why. All of us have many different values that do not reduce to each other, but most of those values do not appear to scale very well with available resources. In other words, among our manifold desires, there may only be a few that are not easily satiated when we have access to the resources of an entire galaxy or universe. If so, (and assuming we aren't wiped out by an existential risk or fall into a Malthusian scenario) the future of our universe will be shaped largely by those values that do scale. (I should point out that in this case the universe won't necessarily turn out to be mostly simple. Simple values do not necessarily lead to simple outcomes either.)

Now if we were rational agents who had perfect knowledge of our own preferences, then we would already know whether this is the case or not. And if it is, we ought to be able to visualize what the future of the universe will look like, if we had the power to shape it according to our desires. But I find myself uncertain on both questions. Still, I think this possibility is worth investigating further. If it were the case that only a few of our values scale, then we can potentially obtain almost all that we desire by creating a superintelligence with just those values. And perhaps this can be done manually, bypassing an automated preference extraction or extrapolation process with their associated difficulties and dangers. (To head off a potential objection, this does assume that our values interact in an additive way. If there are values that don't scale but interact nonlinearly (multiplicatively, for example) with values that do scale, then those would need to be included as well.)

Whether or not we actually should take this approach would depend on the outcome of such an investigation. Just how much of our desires can feasibly be obtain this way? And how does the loss of value inherent in this approach compare with the expected loss of value due to the potential of errors in the extraction/extrapolation process? These are questions worth trying to answer before committing to any particular path, I think.
P.S., I hesitated a bit in posting this, because underestimating the complexity of human values is arguably a greater danger than overlooking the possibility that I point out here, and this post could conceivably be used by someone to rationalize sticking with their "One Great Moral Principle". But I guess those tempted to do so will tend not to be Less Wrong readers, and seeing how I already got myself sucked into this debate, I might as well clarify and expand on my position.