The point about demarcating individuals is important for ethical theories generally (and decision theories that make use of spooky 'reference classes'). Bostrom's Duplication of experience paper illustrates the problem further.

Also, insofar as CEV is just a sort of idealized deliberative democracy, this points to the problem of emulations with systematically unrepresentative values rapidly becoming the majority when emulation hardware is cheap.

Thinking about this a bit more, and assuming CEV operates on the humans that exist at the time of its application: Why would CEV operate on humans that do exist, and not on humans that could exist? It seems this is what Dr. Evil is taking advantage of, by densely populating identity-space around him and crowding out the rest of humanity. But this could occur for many other reasons: Certain cultures encouraging high birth rates, certain technologies or memes being popular at the time of CEV-activation that affect the wiring of the human brain, or certain historical turns that shape the direction of mankind. A more imaginative scenario: what if another scientist, who knows nothing about FAI and CEV, finds it useful to address a problem by copying himself into trillions of branches, each examining a certain hypothesis, and all the branches are (merged/discarded) when the answer is found. Let's further say that CEV t-zero occurs when the scientist is deep in a problem-solving cycle. Would the FAI take each branch as a separate human/vote? This scenario involves no intent to defraud the system. It also is not manipulation of a proxy, as there is a real definitional problem here whose answer is not easily apparent to a human. Applying CEV to all potential humans that could have existed in identity-space would deal with this, but pushes CEV further and further into uncomputable territory.

One could of course define the CEV in terms of some previous population, say circa the year 2000. But then you might wonder why it is fair to give higher weight to those groups that managed to reproduce from 1900 to 2000, and so defining it in terms of year 1900 people might be better. But then how far back do you go? How is the increase in population that Dr. Evil manages to achieve for his descendants less legitimate than all the prior gains of various groups in previous periods?

Simple solution: Build an FAI to optimize the universe to your own utility function instead of humanity's average utility function. They will be nearly the same thing anyway (remember, you were tempted to have the FAI use the average human utility function instead, so clearly, you sincerely care about other people's wishes). And in weird situations in which the two are radically different (like this one), your own utility function more closely tracks the intended purpose of an FAI.

The thing I've never understood about CEV is how the AI can safely read everyone's brain. The whole point of CEV is that the AI is unsafe unless it has a human value system, but before it can get one, it has to open everyones heads and scan their brains!? That doesn't sound like something I'd trust a UFAI to do properly.

I bring this up because without knowing how the CEV is supposed to occur it is hard to analyse this post. I also agree with JoshuaZ that this didn't deserve a top-level post.

This seems to be another case where explicit, overt reliance on a proxy drives a wedge between the proxy and the target.

One solution is to do the CEV in secret and only later reveal this to the public. Of course, as a member of said public, I would instinctively regard with suspicion any organization that did this, and suspect that the proffered explanation (some nonsense about a hypothetical "Dr. Evil") was a cover for something sinister.

EDIT: Doesn't work, see Wei Dai below.

This isn't a bug in CEV, it's a bug in the universe. Once the majority of conscious beings are Dr. Evil clones, then Dr. Evil becomes a utility monster and it gets genuinely important to give him what he wants.

But allowing Dr. Evil to clone himself is bad; it will reduce the utility of all currently existing humans except Dr. Evil.

If a normal, relatively nice but non-philosopher human ascended to godhood, ve would probably ignore Dr. Evil's clones' wishes. Ve would destroy the clones and imprison the doctor, because ve... (read more)

In a sense, this kind of thing (value drift due to differential reproduction) is already happening. For example, see this article about the changing racial demographics of the USA:

According to an August 2008 report by the U.S. Census Bureau, those groups currently categorized as racial minorities—blacks and Hispanics, East Asians and South Asians—will account for a majority of the U.S. population by the year 2042. Among Americans under the age of 18, this shift is projected to take place in 2023

An increasing Latin-American population in the US seems to... (read more)

What if the influence is weighted by degree of divergence from the already-scanned minds, something like a reverse PageRank? All Dr. Evils would cluster, and therefore count as bit above-1 vote. Also, this could cover the human-spectrum better, less influenced by cultural factors. I guess this would give outliers much more influence but if outliers are in all directions, would they cancel each other out? What else could go terribly wrong with this?

Since Dr. Evil is human, it shouldn't be that bad. Extrapolated volition kicks in, making his current evil intentions irrelevant, possibly even preferring to reverse the voting exploit.

I think this can be dealt with in terms of measure. In a series of articles, "Minds, Measure, Substrate and Value" I have been arguing that copies cannot be considered equally, without regard to substrate: We need to take account of measure for a mind, and the way in which the mind is implemented will affect its measure. (Incidentally, some of you argued against the series: After a long delay [years!], I will be releasing Part 4, in a while, which will deal with a lot of these objections.)

Without trying to present the full argument here, the mini... (read more)

What if we used a two-tiered CEV? A CEV applied to a small, hand selected group of moral philosophers could be used to determine weighting rules and ad hoc exceptions to the CEV that runs on all of humanity to determine the utility function of the FAI.

Then when the CEV encounters the trillion Dr Evil uploads, it will consult what the group of moral philosophers would have wanted to handle it if "they knew more, thought faster, were more the people we wished we were, had grown up farther together", which would be be something like weight them together as one person.

Insofar as the Evil clones are distinct individuals, they seem to be almost entirely potentially distinct. They will need to receive more computing resources before they can really diverge into distinct agents.

I would expect CEV to give the clones votes only to the extent that CEV gives votes to potential individuals. But the number of potential clones of normal humans is even greater than Evil's trillion, even accounting for their slightly greater actuality. So, I think that they would still be outvoted.

That's an interesting point, but I'm having trouble seeing it as worthy of a top-level post. Maybe if you had a solution proposed also.

I think it might be possible to patch around this by weighting people by their projected future cycle count. Otherwise, I fear that you may end up with a Repugnant Conclusion even without an adversary -- a very large number of happy emulated people running very slowly would outweigh a smaller number of equally happy people running at human-brain-speed. Of course, this still gives an advantage to the views of those who can afford more computing power, but it's a smaller advantage. And perhaps our CEV would be to at least somewhat equalize the available computing power per person.

Doesn't the "coherent" aspect of "coherent extrapolated volition" imply, generally speaking, that it's not democracy-of-values, so to speak? That is to say, CEV of humanity is supposed to output something that follows on from the extrapolated values of both the guy on the street corner holding a "Death to all fags" sign who's been arrested twice for assaulting gay men outside bars, and the queer fellow walking past him -- if prior to the implementation of a CEV-using FAI the former should successfully mobilize raise the visibi... (read more)

And then Dr. Evil, forced to compete with 999,999,999,999 copies of himself that all want to rule, is back to square one. Would you see multiplying your competition by 999,999,999,999 as a solution to how to rule the universe? If you were as selfish as Dr. Evil, and intelligent enough to try attempting to take over the universe, wouldn't it occur to you that the copies are all going to want to be the one in charge? Perhaps it won't, but if it did, would you try multiplying your competition, then? If not, then maybe part of the solution to this is makin... (read more)

But of course an AI realizes that satisfying the will of trillion copies of Dr. Evil wasn't what his/her programmers intented.

Pun being, this legendary bad argument is surprisingly strong here. I know, I shouldn't be explaining my jokes.