Suppose we were to write down all (input, output) pairs for the ideal "one-place function" described by Eliezer on a oblong stone tablet somewhere. This stone tablet would then contain perfect moral wisdom. It would tell us the right course of action in any possible situation.

This tablet would be the result of computation, but it's computation that nobody can actually do, as we currently only have access to approximations to the ideal Morality(X) function. Thus, as far as we're concerned, this tablet is just a giant look-up table. Its contents are a brute fact about the universe, like the ratio of the masses of the proton and the electron. If we are confronted with a moral dilemna, and our personal ideas of right and wrong contradict the tablet, this will always be a result of our own morality functions poorly approximating the ideal. In such a situation, we should override our instincts and go with the tablet every time.

In other words, according to Eliezer's model, in a universe where this tablet exists morality is given.

This is also true of a universe where the tablet does not exist (such as ours--it wouldn't fit!).

So Eliezer has just rediscovered "morality is the will of God", except he's replacing "God" with a giant block of stone somewhere in a hypothetical universe. It's not clear to me that this is an impovement.

It seems to me that the functional difference is that Eliezer believes he can successfully approximate the will of the Giant Hypothetical Oblong Stone Tablet out of his own head. If George Washington says "Slavery is sometimes just," Eliezer does not take this assertion seriously; he does not start trying to re-work his personal GHOST-approximator to take Washington's views into account. Rather he says, "I know that slavery is wrong, and I approximate the GHOST, so slavery is wrong," ignoring the fact that all men--including Washington--approximate the GHOST as best they can. Worse, by emphasizing the process of making, weighing and pondering moral "arguments", he privileges the verbally and quantitatively quick over the less intelligent, even though the correlation between being good with words and having a good GHOST-approximator is nowhere shown.

Everyone's GHOST-approximator is shaped by his environment. If the modern world encourages people to deny the GHOST in particular ways, and Eliezer indeed does so, then he would not be able to tell. His tool for measuring, his personal GHOST-finder, would have been twisted. His friends' and respected peers' GHOST-approximators might all be twisted in the same way, so nobody would point out his error and he would have no opportunity to correct it. He would use his great skill with words to try to convince everyone that his personal morality was correct. Him and people like him might well succeed. His assertion of moral progress would then merely be the statement that the modern world reflects his personal biases--or perhaps that he reflects the biases of the modern world.

I'm concerned that the metamorality described by Eliezer will encourage self-named rationalists to worship their own egos, placing their personal imperfect GHOST-approximators--all shaped by the moral environment of the modern world--at the same level as those in past ages placed the will of God. Perhaps this is not Eliezer's intention. But to do otherwise, to look beyond the biases of the present day, one would have to acknowledge that the GHOST-readers of our ancestors may have in some ways have been better than ours. This would require humility; and pride cures humility.

Eliezer, Do you think that all public declarations of faith should be met by gasps and frozen shock, or just those which are framed using your particular phrasing? Also, the existence of a social norm implies penalties for flagrant violators. If Jews (for example) persist in encouraging theistical utterances by other Jews, what steps do you think society at large is justified in taking?

Robin:

Discarding false mathematical and scientific conjectures is indeed much easier than discarding false moral conjectures. However, as Eliezer pointed out in an earlier post, a scientist who can come up with a hypothesis that has a 10% chance of being true has already gone most of the way from ignorance to knowledge. I would argue that hypothesis generation is a poorly-understood nonrational process in all three cases. A mathematician who believes he has found truth can undertake the further steps of writing a formal proof and submitting his work to public review, greatly improving his reliability. A man confronted with a moral dilemma must make a decision and move on.

I think that the universal tendency towards religion is indeed evidence in favor of the existence of God, but not very strong evidence. The adaptive advantage of discerning correct metaphysics was minimal in the ancestral environment.

Richard C:

I think if you try to use your "general capacity for systematizing judgments" to make moral decisions, you'll restrict yourself to moral systems which are fully accessible to human reason.

Robin,

I don't know how people are capable of discerning moral truths. I also don't know how people are capable of discerning scientific or mathematical truths. It seems to me that these are similar capabilities, and the one is no more suprising or unlikely than the other.

Robin,

I don't understand your counterfactual.

"Good" and "Evil" are the names for what people perceive with their moral sense. I think we've agreed that this perception correlates to something universally observable (namely, social survival), so these labels are firmly anchored in the physical world. It looks to me like you're trying to assign these names to something else altogether (namely, something which does not correlate with human moral intuitions), and it's not clear to me how this makes sense.

Robin,

Our moral intuitions correspond with moral truths for much the same reason that our rational predictions correspond with more concrete physical truths. A man who ignores reason will stick his hand back in the fire after being burned the first time. Such behavior will kill him, probably sooner rather than later. An man who is blind to good and evil may do quite well for himself, but a society whose citizens ignore virtue will suffer approximately the same fate as the twice-burned fool.

Robin, As Eliezer has pointed out, evolution is a nonhuman optimizer which is in many ways more powerful than the human mind. On the assumption that humans have a moral sense, I don't think we should expect to be able to understand why. That might simply be a problem which is too difficult for people to solve. That aside, a man's virtues benefit the society he lives in; his inclination to punish sin will encourage others to act virtuously as well. If his society is a small tribe of his relatives, then even the weaker forms of kin selection theory can explain the benefit of knowledge of good and evil.

I don't think you have to postulate Space Cannibals in order to imagine rational creatures who don't think murder is wrong. For a recent example, consider Rwanda 1994.

And I think it's quite possible that there might exist moral facts which humans are incapable of perceiving. We aren't just universal Turing machines, after all. Billions of years of evolution might produce creatures with moral blind spots, anologous to the blind spot in the human eye. Just as the squid's eye has no blind spot, a different evolutionary path might produce creatures with a greater or lesser innate capacity to perceive goodness than ourselves.

A duty is half of a contract--it comes from some obligation assumed (perhaps implicitly) in the past. A man may in general assign a very high priority to keeping his promises. He may feel a moral obligation to do so, independent of the specific nature of the promise. Should keeping a promise be difficult or unpleasant, he will balance his desire to avoid unpleasantness with his desire to be the sort of person who repays what was given.

For example, a man who has enjoyed the rights and privileges of a citizen may feel he has a duty to support the interests of his country. Certainly many citizens of the various States felt so, two hundred and thirty-two years ago.