Far more extreme, I would think, to say that zero out of 6.5 billion humans are stable psychopaths.
I wonder if the distinction between 1) something implementing the same dynamic as a typical human but mistaken about what it says and 2) something implementing a completely different dynamic and not mistaken about anything, is the same as the distinction people normally make between 1) immoral and 2) amoral.
Steven: quite possibly related. I don't think they're exactly the same (the classic comic book/high fantasy "I'm evil and I know it" villain fits A2, but I'd describe him as amoral), but it's an interesting parallel.
Eliezer: I'm coming more and more to the conclusion that our main area of disagreement is our willingness to believe that someone who disagrees with us really "embodies a different optimization process." There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processe...
If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all. A General AI should be able to convince it just about anything, right?
Far more extreme, I would think, to say that zero out of 6.5 billion humans are stable psychopaths.
Heck, what about babies? What do they want, and would they be complicated enough to want anything different if they knew more and thought faster?
There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processes; while I believe mine to be the best I've found, I remain aware that if I held any of the others I would believe exactly the same thing.
You would not believe exactly the same thing. If you ...
I must be starting to get it. That unpacked in exactly the way I expected.
On the other hand, this:
If a person's morality is not defined as what they believe about morals, I don't know how it can be considered to meaningfully entail any propositions at all.
makes no sense to me at all.
I think my highest goal in life is to make myself happy. Because I'm not a sociopath making myself happy tends to involve having friends and making them happy. But the ultimate goal is me.If you had a chance to take a pill which would cause you to stop caring about your friends by permanently maxing out that part of your hapiness function regardless of whether you had any friends, would you take it?
Sebastian Hagen:
My intuition is that a good deal of people would take the psychopath pill. At least if the social consequences were minimal, which is besides the point.
Frame it defensively rather than offensively and a whole heck of a lot of people would take that pill. Of course some of us would also take the pill that negates the effects of our friends taking the first pill, hehehe.
"Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob..."
That's right. (I didn't mean to suggest that there's never any disputing what someone's moral commitments are; just that this wasn't supposed to be in dispute in the particular case I was imagining.) I take it that Sally and Bob could disagree even so, and not merely be talking past each other, even if one or both of them was impervious to rational argument. It is at least a significant cost of your theory that it denies this datum. (It doesn't have...
I do not eat steak, because I am uncertain of what my own morality outputs with respect to steak-eating. It seems reasonable to me to imagine that cows are capable of experiencing pain, of fearing death. Of being, and ceasing to be. If you are like the majority of human beings, you do eat steak. The propositions I have suggested do not seem reasonable to you.
Do you imagine that there are facts about the brains of cattle which we could both learn - facts drawn from fMRI scans, or from behavioral science experiments, perhaps - which would bring us into agreement on the issue?
Eliezer: for 'better' vs 'frooter,' of course you're right. I just would have phrased it differently; I've been known to claim that the word 'better' is completely meaningless unless you (are able to) follow it with "better at or for something." So of course, Jadagul_real would say that his worldview is better for fulfilling his values. And Jadagul_hypothetical would say that his worldview is better for achieving his values. And both would (potentially) be correct. (or potentially wrong. I never claimed to be infallible, either in reality o...
Re: "We have nothing to argue about, we are only different optimization processes." That seems to apply in the case when a man wants to rescue copies of his genes from eternal oblivion - by convincing his mate not to abort his prospective offspring. Of course, not many would actually say that under those circumstances.
Eliezer: "But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it."
How sure are you that most human moral disagreements are attributable to
Mike,
"I do not eat steak, because I am uncertain of what my own morality outputs with respect to steak-eating. It seems reasonable to me to imagine that cows are capable of experiencing pain, of fearing death. Of being, and ceasing to be. If you are like the majority of human beings, you do eat steak. The propositions I have suggested do not seem reasonable to you."
Accepting your propositions for the sake of argument, I still find that eating steak seems reasonable.
"Rather, it is essential to the concept of morality that it involves shared standards common to all fully reasonable agents."
Richard,
If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'
Other moral issues where there are a gender differences include: "should prostitution be legalised" and "should there be tighter regulation of pornography".
Again, it seems that part of the effect is due to people's idea of what is right being influenced by their own personal role - i.e. the "different optimization processes" effect.
Gender is the most obvious source of such issues, but I'm sure you can find them in other areas of life. Race politics, for instance.
People in general do not want to be shot. The person doing the shooting, the lethal weapon being fired, the location in which the shooting occurs and the time of day are all pretty much irrelevant. You can ask people if they want to be shot and they'll say no, without even specifying those details. That seems a very different case from Bob, who is considering a moral proposition and outright rejecting it.
Given all of Bacon's idols of the Mind can you ever know definitely if there is an error in your own reasoning, let alone the other persons?
You cannot rely on your moral intuition, nor the cultural norms of your time, nor academic authority, nor your internal reasoning or ability to know the soundness of your argument.
Socialization, severe biases, faulty reasoning can all make you think you are ‘correct’, but can leave you with the incorrect impression of the ‘correctness’ of your thinking. & even if presented with all the correct or relevant information some people still make these errors, so if they can so could you.
Eliezer: "When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about - unless you cover "disagreement" to include differences where two agents have nothing to say to each other.
But this would be an extreme position to take with respect to your fellow...
I should qualify this statement: "As such, we shouldn't expect much more moral agreement from humans than from rational (or approximately rational) AIs."
to instead read:
"As such, on ethical questions that had no precedent in our EEA, we shouldn't expect much more moral agreement from humans than from rational (or approximately rational) AIs, apart, of course, from the fact that most humans share a common set of cognitive biases"
one can see that this is true by looking at the vast disagreements between different moral philosophers consequentialists vs. deontological ethicists, or atheists vs. christians vs. muslims, or libertarians vs. liberals vs. communists.
You have been priming people to think in terms of functions. (Pure) Functions do not change. They map an input to an output, and can be substituted by a list, e.g. a function that tests for primality can can be performed by an (infinitely) long list.
You may want to describe impure functions for people without a functional programming background if you want to keep terminology like morality_john().
I find Roko on-point. The psychological unity of humankind is important, but it can be over-stated. While human beings may occupy a very small area in the space of all possible minds, it is still an area and not a single point. When we shut up and multiply by sufficiently large numbers, very small differences in the starting point are very meaningful. If we are talking about a difference valued at 0.0000001% of a human life, and you extrapolate it over a billion lives, we are talking about life and death matters. Successful AI will affect more than a ...
Virge:
How sure are you that most human moral disagreements are attributable to - lack of veridical information, or - lack of ability/tools to work through that information, or - defects? You talk freely about psychopaths and non-psychopaths as though these were distinct categories of non-defective and defective humans. I know you know this is not so. The arguments about psychological unity of humankind only extend so far. e.g., would you be prepared to tell a homosexual that, if they were fully informed, they would decide to take a pill to change their or...
I must be missing something -- why would you advocate something that you know you can't justify to anyone else?
I said "damaged" not "missing". The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.
Does that work in the other direction? The notion is that you are your current self, but one day I inform you that, r...
Jadagul:
But my moral code does include such statements as "you have no fundamental obligation to help other people." I help people because I like to.While I consider myself an altruist in principle (I have serious akrasia problems in practice), I do agree with this statement. Altruists don't have any obligation to help people, it just often makes sense for them to do so; sometimes it doesn't, and then the proper thing for them is not to do it.
Roko:
In the modern world, people have to make moral choices using their general intelligence, because th...
Re: If there are distinct categories of human transpersonal values, I would expect them to look like "male and female babies", "male children", "male adults", "female children", "female adults", "neurological damage 1", "neurological damage 2", not "Muslims vs. Christians!"
That seems like the position you would get if you thought that cultural evolution could not affect people's values.
Carl - "If you're going to define 'fully reasonable' to mean sharing your moral axioms, so that a superintelligent pencil maximizer with superhuman understanding of human ethics and philosophy is not a 'reasonable agent,' doesn't this just shift the problem a level? Your morality_objectivenorms is only common to all agents with full reasonableness_RichardChappell, and you don't seem to have any compelling reason for the latter (somewhat gerrymandered) account of reasonableness save that it's yours/your culture's/your species.'"
I don't mean to def...
"If there are distinct categories of human transpersonal values, I would expect them to look like [...] 'male adults', [...] 'female adults', [...] not 'Muslims vs. Christians!'"
Really? In the ways that are truly important, don't you think you have more in common with Natasha Vita-More than Osama bin Laden?
Steven, even so, I think the basic question stands. Why should cultural differences and within-sex individual differences wash out of the CEV?
Supposedly genetics allows for people of different ages or sexes to have different mental machinery, whereas individual genetic differences just represent low-complexity differences in tweaking. I'm not sure why that makes Eliezer's point though, if the aforementioned differences in tweaking mean different complex machinery gets activated. Cultural differences I'd expect to wash out just through people learning about different cultures that they could have grown up in.
Zubon: "if you do not think you should be a bit more sociopathic, what are the odds you have exactly the right amounts of empathy and altruism?"
Steven: "Cultural differences I'd expect to wash out just through people learning about different cultures that they could have grown up in."
I suspect a category error here hinging around personal identity. We say "if I had grown up in a different culture ..." when I think we mean "if the baby that grew into me had grown up in a different culture..." If the baby that grew into me had grown up in a radically different culture, I don't think ve'd be me in any meaningful sense, although of course there would be many similarities ...
Re: Roko's "more reasonable position":
Human psychological unity, to the extent that it exists, includes brain speed, brain degree-of-parallelism, brain storage capacity, brain reliability - and other limitations that have little to do with human goals.
If you had a chance to take a pill which would cause you to stop caring about your friends by permanently maxing out that part of your hapiness function regardless of whether you had any friends, would you take it?
I'm not sure this proves anything. I'll take Jadagul's description as my own. I'm maximizing my happiness, but my non-sociopath status means that I like having friends/loved ones, so maximizing my own happiness entails caring about them to some degree or another. Under my existing moral code I wouldn't take the pill, but that's because it will...
I, too, wonder if the "psychological unity of humankind" has been a bit overstated. All [insert brand and model here] computers have identical hardware, but you can install different software on them. We're running different
Consider the case of a "something maximizer". It's given an object, and then maximizes the number of copies of that object.
You give one something maximizer a paperclip, and it becomes a paperclip maximizer. You give another a pencil, and that one becomes a pencil maximizer.
There's no particular reason to expect the p...
Eliezer: "The basic ev-bio necessity behind the psychological unity of human brains is not widely understood."
I agree. And I think you've over-emphasized the unity and ignored evidence of diversity, explaining it away as defects.
Eliezer: "And even more importantly, the portion of our values that we regard as transpersonal, the portion we would intervene to enforce against others, is not all of our values; it's not going to include a taste for pepperoni pizza, or in my case, it's not going to include a notion of heterosexuality or homosexuali...
virge makes a very good point here. The human mind is probably rather flexible in terms of it's ethical views; I suspect that Eli is overplaying our psychological unity.
I think that views are being attributed to me that I do not possess - perhaps on the Gricean notion that if someone attacks me for holding these views, I ought to hold them; but on a blog like this one, that leaves you prey to everyone else's misunderstanding.
I do not assert that all humans end up in the same moral frame of reference (with regard to any particular extrapolation method). I do think that psychological unity is typically underestimated, and I have a hard time taking modern culture at face value (we're the ancient Greeks, guys, not a finished...
"Being a jerk" here means "being a jerk according to other people's notion of morality, but not according to my own notion of morality", right?
I sort of take offense to "we're the ancient Greeks"; I make sure to disagree with Western morality whenever it's wrong, and I have no reason to believe the resulting distribution of errors is biased toward agreement with Western morality. If you meant to say "most of them are the ancient Greeks", then sure.
On second thought I suppose it could mean "being a jerk according to 'ethics'", where "ethics" is conceived not as something intrinsically moral but as a practical way for agents with different moralities to coordinate on a mutually acceptable solution.
The notion is that I am my current self, but one day you inform me that, relative to other humans, my ability to care about others is damaged. Do I want a pill to fix the damage, even though it will change my values? Yes, because I value humanity and want to stay with humanity; I don't want to be off in some lonely unoccupied volume of mindspace. This is one of the arguments that moves me.
Eliezer, relative to other humans, your ability to believe in a personal creative deity is damaged.
Do you want a pill to help you be religious?
Virge: The argument for psychological unity is that, as a sexually reproducing species, it is almost impossible for one gene to rise in relative frequency if the genes it depends on are not already nearly universal. So the all the diversity within any species at any given time consists of only one-step changes; no complex adaptations. The one exception of course is that males can have complex adaptations that females lack, and vice versa.
So, with respect to your specific examples:
Homosexuals: sexual preference certainly is a complex adaptation, but obvio...
Eli: "I do not assert that all humans end up in the same moral frame of reference (with regard to any particular extrapolation method). I do think that psychological unity is typically underestimated,"
- right, thanks for the clarification.
"But if you read "Coherent Extrapolated Volition" you'll see that it's specifically designed to handle, among other problems, the problem of, "What if we don't all want the same thing?" What then can an AI programmer do that does not constitute being a jerk? That was my attempt to answ...
Larry D'Anna: "And it doesn't do any good to say that they aren't defective. They aren't defective from a human, moral point of view, but that's not the point. From evolutions view, there's hardly anything more defective, except perhaps a fox that voluntarily restrains it's own breeding."
Why is it "not the point"? In this discussion we are talking about differences in moral computation as implemented within individual humans. That the blind idiot's global optimization strategy defines homosexuality as a defect is of no relevance.
Larr...
Roko:
I would not extrapolate the volitions of people whose volitions I deem to be particularly dangerous, in fact I would probably only extrapolate the volition of a small subset (perhaps 1 thousand - 1 million) people whose outward philosophical stances on life were at least fairly similar to mine.
Then you are far too confident in your own wisdom. The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI, because when you zoom out to the billion-year view, we may not be all that mu...
The overall FAI strategy has to be one that would have turned out okay if Archimedes of Syracuse had been able to build an FAI, because when you zoom out to the billion-year view, we may not be all that much wiser than they.
"Wiser"? What's that mean?
Your comment makes me think that, as of 12 August 2008, you hadn't yet completely given up on your dream of finding a One True Eternal Morality separate from the computation going on in our heads. Have you changed your opinion in the last two years?
unilaterally label friendly
I love your turn of phrase, it has a Cold War ring to it.
The question why anyone would ever sincerely want to build an AI which extrapolates anything other than their personal volition is still unclear to me. It hinges on the definition of "sincerely want". If Eliezer can task the AI with looking at humanity and inferring its best wishes, why can't he task it with looking at himself and inferring his best idea of how to infer humanity's wishes? How do we determine, in general, which things a document like CEV must spell out and which things can/should be left to the mysterious magic of "intelligence"?
The question why anyone would ever sincerely want to build an AI which extrapolates anything other than their personal volition is still unclear to me. It hinges on the definition of "sincerely want". If Eliezer can task the AI with looking at humanity and inferring its best wishes, why can't he task it with looking at himself and inferring his best idea of how to infer humanity's wishes?
This has been my thought exactly. Barring all but the most explicit convolution any given person would prefer their own personal volition to be extrapolated. If by happenstance I should be altruistically and perfectly infatuated by, say Sally, then that's the FAI's problem. It will turn out that extrapolating my volition will then entail extrapolating Sally's volition. The same applies to caring about 'humanity', whatever that fuzzy concept means when taken in the context of unbounded future potential.
I am also not sure how to handle those who profess an ultimate preference for a possible AI that extrapolates other than their own volition. I mean, clearly they are either lying, crazy or naive. It seems safer to trust someone who says "I would ultimately prefer FAI but I am creatin...
Followup to: Inseparably Right, Sorting Pebbles Into Correct Heaps
Richard Chappell, a pro, writes:
The phenomena of moral disagreement, moral error, and moral progress, on terminal values, are the primary drivers behind my metaethics. Think of how simple Friendly AI would be if there were no moral disagreements, moral errors, or moral progress!
Richard claims, "There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob."
We may not suppose, and there is disputing. Bob does not have direct, unmediated, veridical access to the output of his own morality.
I tried to describe morality as a "computation". In retrospect, I don't think this is functioning as the Word of Power that I thought I was emitting.
Let us read, for "computation", "idealized abstract dynamic"—maybe that will be a more comfortable label to apply to morality.
Even so, I would have thought it obvious that computations may be the subjects of mystery and error. Maybe it's not as obvious outside computer science?
Disagreement has two prerequisites: the possibility of agreement and the possibility of error. For two people to agree on something, there must be something they are agreeing about, a referent held in common. And it must be possible for an "error" to take place, a conflict between "P" in the map and not-P in the territory. Where these two prerequisites are present, Sally can say to Bob: "That thing we were just both talking about—you are in error about it."
Richard's objection would seem in the first place to rule out the possibility of moral error, from which he derives the impossibility of moral agreement.
So: does my metaethics rule out moral error? Is there no disputing that abortion is indeed prohibited by morality_Bob?
This is such a strange idea that I find myself wondering what the heck Richard could be thinking. My best guess is that Richard, perhaps having not read all the posts in this sequence, is taking my notion of morality_Bob to refer to a flat, static list of valuations explicitly asserted by Bob. "Abortion is wrong" would be on Bob's list, and there would be no disputing that.
But on the contrary, I conceive of morality_Bob as something that unfolds into Bob's morality—like the way one can describe in 6 states and 2 symbols a Turing machine that will write 4.640 × 101439 1s to its tape before halting.
So morality_Bob refers to a compact folded specification, and not a flat list of outputs. But still, how could Bob be wrong about the output of his own morality?
In manifold obvious and non-obvious ways:
Bob could be empirically mistaken about the state of fetuses, perhaps believing fetuses to be aware of the outside world. (Correcting this might change Bob's instrumental values but not terminal values.)
Bob could have formed his beliefs about what constituted "personhood" in the presence of confusion about the nature of consciousness, so that if Bob were fully informed about consciousness, Bob would not have been tempted to talk about "the beginning of life" or "the human kind" in order to define personhood. (This changes Bob's expressed terminal values; afterward he will state different general rules about what sort of physical things are ends in themselves.)
So those are the obvious moral errors—instrumental errors driven by empirical mistakes; and erroneous generalizations about terminal values, driven by failure to consider moral arguments that are valid but hard to find in the search space.
Then there are less obvious sources of moral error: Bob could have a list of mind-influencing considerations that he considers morally valid, and a list of other mind-influencing considerations that Bob considers morally invalid. Maybe Bob was raised a Christian and now considers that cultural influence to be invalid. But, unknown to Bob, when he weighs up his values for and against abortion, the influence of his Christian upbringing comes in and distorts his summing of value-weights. So Bob believes that the output of his current validated moral beliefs is to prohibit abortion, but actually this is a leftover of his childhood and not the output of those beliefs at all.
(Note that Robin Hanson and I seem to disagree, in a case like this, as to exactly what degree we should take Bob's word about what his morals are.)
Or Bob could believe that the word of God determines moral truth and that God has prohibited abortion in the Bible. Then Bob is making metaethical mistakes, causing his mind to malfunction in a highly general way, and add moral generalizations to his belief pool, which he would not do if veridical knowledge of the universe destroyed his current and incoherent metaethics.
Now let us turn to the disagreement between Sally and Bob.
You could suggest that Sally is saying to Bob, "Abortion is allowed by morality_Bob", but that seems a bit oversimplified; it is not psychologically or morally realistic.
If Sally and Bob were unrealistically sophisticated, they might describe their dispute as follows:
Now, this is not exactly what most people are explicitly thinking when they engage in a moral dispute—but it is how I would cash out and naturalize their intuitions about transpersonal morality.
Richard also says, "Since there is moral disagreement..." This seems like a prime case of what I call naive philosophical realism—the belief that philosophical intuitions are direct unmediated veridical passports to philosophical truth.
It so happens that I agree that there is such a thing as moral disagreement. Tomorrow I will endeavor to justify, in fuller detail, how this statement can possibly make sense in a reductionistic natural universe. So I am not disputing this particular proposition. But I note, in passing, that Richard cannot justifiably assert the existence of moral disagreement as an irrefutable premise for discussion, though he could consider it as an apparent datum. You cannot take as irrefutable premises, things that you have not explained exactly; for then what is it that is certain to be true?
I cannot help but note the resemblance to Richard's assumption that "there's no disputing" that abortion is indeed prohibited by morality_Bob—the assumption that Bob has direct veridical unmediated access to the final unfolded output of his own morality.
Perhaps Richard means that we could suppose that abortion is indeed prohibited by morality_Bob, and allowed by morality_Sally, there being at least two possible minds for whom this would be true. Then the two minds might be mistaken about believing themselves to disagree. Actually they would simply be directed by different algorithms.
You cannot have a disagreement about which algorithm should direct your actions, without first having the same meaning of should—and no matter how you try to phrase this in terms of "what ought to direct your actions" or "right actions" or "correct heaps of pebbles", in the end you will be left with the empirical fact that it is possible to construct minds directed by any coherent utility function.
When a paperclip maximizer and a pencil maximizer do different things, they are not disagreeing about anything, they are just different optimization processes. You cannot detach should-ness from any specific criterion of should-ness and be left with a pure empty should-ness that the paperclip maximizer and pencil maximizer can be said to disagree about—unless you cover "disagreement" to include differences where two agents have nothing to say to each other.
But this would be an extreme position to take with respect to your fellow humans, and I recommend against doing so. Even a psychopath would still be in a common moral reference frame with you, if, fully informed, they would decide to take a pill that would make them non-psychopaths. If you told me that my ability to care about other people was neurologically damaged, and you offered me a pill to fix it, I would take it. Now, perhaps some psychopaths would not be persuadable in-principle to take the pill that would, by our standards, "fix" them. But I note the possibility to emphasize what an extreme statement it is to say of someone:
"We have nothing to argue about, we are only different optimization processes."
That should be reserved for paperclip maximizers, not used against humans whose arguments you don't like.
Part of The Metaethics Sequence
Next post: "Abstracted Idealized Dynamics"
Previous post: "Sorting Pebbles Into Correct Heaps"