Coherent extrapolated volition (CEV) asks what humans would want, if they knew more - if their values reached reflective equilibrium. (I don't want to deal with the problems of whether there are "human values" today; for the moment I'll consider the more-plausible idea that a single human who lived forever could get smarter and closer to reflective equilibrium over time.)
This is appealing because it seems compatible with moral progress (see e.g., Muehlhauser & Helm, "The singularity and machine ethics", in press). Morality has been getting better over time, right? And that's because we're getting smarter, and closer to reflective equilibrium as we revise our values in light of our increased understanding, right?
This view makes three claims:
- Morality has improved over time.
- Morality has improved as a result of reflection.
- This improvement brings us closer to equilibrium over time.
There can be no evidence that morality has improved
Intuitively, we feel that morality has definitely improved over time. We are so much better than those 17th-century barbarians who baited bears!
If you have such a strong belief, that must mean you have evidence for it. That must mean you had some hypothesis, and the evidence could have gone either way; and the evidence went in such a way that it supported your hypothesis.
If you believe this, then in the comments below, please describe a scenario that could have happened, in which we would today believe that the values people had hundreds of years ago were superior to the values they have today. Not a scenario in which some conservative sub-group could believe this; but a scenario in which society as a whole could believe it, and keep on believing it for a hundred years, without changing their values.
We can show that values have changed. But we can have no evidence that that change is towards better values, whatever that means, rather than a value-neutral drift. (I don't even know how to express coherently the idea that "values are getting better".)
If society agreed that another set of values were superior, they would adopt those values. In fact, they would already have those values, prior to agreeing. There can be no observed event supporting the hypothesis that morals have improved. No matter how much you feel that they have improved, you cannot have empirical evidence, not even in principle.
Our values do not change as a result of reflection
Values, like biology and culture, evolve. That doesn't mean getting "better" over time. It means becoming more adaptive.
Take any moral advance you like. Study its history, and you'll find people adopted it when it became economically advantageous to those in power do so.
Do unto others as ye would have others do unto you. Turn the other cheek. Slaves, obey your masters.
The Roman Empire was not an empire; it was a forest fire. It burned its way out from Italy and across the continent, using up each new land that it came to, stripping it of resources and funneling them to Rome. When it burned its way out until pillaging the new area on its perimeter (increasing as R) could no longer support the area in its interior (increasing as R squared), it burned out and died. It was not a sustainable economic model. It relied on exploiting conquered peoples, and on suppressing them with armies built from the wealth acquired by conquering other people. (Citation needed. I'm not an expect on ancient Rome.)
With Christianity, you could exploit people without needing large armies to keep them in line. Christianity was the technology that saved the eastern half of the Roman Empire and allowed its survival into the high middle ages; and that enabled the rise of Western European nations. "Slaves, obey your masters" was an economic necessity. (China had discovered Taoism and Buddhism centuries earlier.)
How did Christianity bring us closer to reflective equilibrium? It didn't. It brought us WAY out of reflective equilibrium. The virtues expressed in the Iliad are pretty close to a reflective equilibrium. When we introduced all this stuff about loving your enemy, the cognitive dissonance in Western ethics went up by orders of magnitude. Even today, we've never gotten near to the level of equilibrium we had pre-Christianity. Christianity, as promoted by Jesus, is pacifist, communist, non-materialist, unpatriotic, and anti-family.
Consider an even more significant moral advance: The de-masculinization of the human race. Until a few centuries ago, men were encouraged to fight each other pretty much as often as possible. Excellence in combat was the single greatest virtue in most societies throughout all of history until the 20th century. Beating up weaker boys not only wasn't bad; it was a kind of civic duty.
The destructive technology of the 19th and especially the 20th centuries required changes. Armed conflict was no longer a cost-effective way to make money or resolve disputes. Society had to be reprogrammed. And as population density continued to rise, countries needed to be able to keep a million men in a single city without them turning on each other like rats in a cage.
Again, how did this bring us closer to an equilibrium? It didn't, which is why confused men sometimes feel the need to have steam lodges and drum circles in the woods.
Or take slavery. Was the abolition of slavery in the US the result of reflective equilibrium? The virtuous northern US, which happened to have a lot of textile mills and other industry requiring skilled labor, realized the monstrosity of the institution of slavery, which also happened to give the Southern states enough votes in the House of Representatives to implement tariffs and other economic laws that favored the production of raw materials over the manufacture of goods.
But, you say, the North also had plenty of farmers! Yet these good Presbyterians were never tempted to have their apple orchards or their cranberry bogs tended by slaves.
That's because the northern US is cold. It has a short growing season. It's more economical to hire workers when you need them, than to keep slaves year-round.
The Civil War began just after mechanical reapers and other inventions began to make slavery uneconomical for more and more people, until they reached the tipping point at which the people with these devices could use anti-slavery as a weapon against their competitors. If the War had been delayed fifteen years, the South would have been inundated with labor-saving farm devices that made keeping slaves cost more than it was worth, and would have suddenly seen the error of their ways and renounced slavery on their own. And the North would have missed an opportunity to achieve hegemony and the high moral ground at the same time.
Values shift further from, not closer to, equilibrium over time
The world is not in equilibrium, and hopefully never will be. The trend, historically, has been for cultural change to accelerate, bringing us farther from, not closer to, equilibrium. (This trend may be reversing in the last several decades, a point which would require many additional posts to explore.)
Culture is the sort of thing that you can't predict, you can only simulate. The only way to see how the world is going to develop is to wait for it to develop.
You may think that a super-intelligent AI can simulate this much, much faster than humans can. And you would be right. But the super-intelligent AI is part of the culture - you could say it is the culture - once it exists. In the process of trying to reach reflective equilibrium, it will learn new things, and discover new possibilities, which will require it to re-evaluate all prior beliefs, taking it farther from, not closer to, equilibrium. Is there any reason to think this process will converge, rather than diverge more and more, as it has for all of history? If there is, it has not been articulated.
Values converge and reach equilbrium in the same way that evolution converges and reaches equilbrium: Not at all.