Convergence Theories of Meta-Ethics

[-]Eliezer Yudkowsky15y220

So we need another word to filter out those kinds of somewhat-arbitrary proposed meta-ethical systems. "Objective" probably is not the best word for the job, but it is the only one I can think of right now.

This is where I stopped reading.

I suggest that you actually read the SEP entry on meta-ethics instead of just linking there - if you did read it, feel free to correct my guess. Metaethics does not mean what you said it did (metaethics is a theory of what morality is, not a way of comparing moralities), moral realism does not mean what you said it did (your belief that morality is a real thing out there constitutes moral realism), naturalistic metaethics do not mean what you said it did, CEV is totally not about convergence in all possible minds, etcetera. I also have to ask whether you read the Metaethics Sequence, but I mostly regard that sequence as having failed so I won't be surprised if the answer is yes.

6Vladimir_Nesov15y

Has anyone reached what you regard as satisfactory level of understanding of your ideas as a result of reading the sequence? That is, does its failure refer to lower-than-wanted probability of a person reading the sequence understanding your ideas, or to an almost complete failure to communicate your ideas to anyone?

6Eliezer Yudkowsky15y

Well, it looks to me like SIAI core people got it, but there's trouble being sure about that sort of thing.

5wedrifid15y

Without contradicting you in any way and with an acknowledgement that you could well disapprove of the way I think about morality too I'll add that comprehension seems to have extended to the unaffiliated population. However both the rate and degree of comprehension is definitely much lower than for your core rationality material. Surprisingly so. However I have since formed an impression that the difficulties in thinking about morality extend far beyond just how your own posts are received.

1Perplexed15y

As someone who apparently did not 'get it', I would suggest that there was a problem with clarity, and that the root cause of the lack of clarity was something that might be called 'moral cognitive distance'. It frequently seemed that you were appealing to my moral intuitions, expecting them to be the same as yours. Pretty often they weren't.

0thomblake15y

As far as I can tell, I got it. My evidence that I have it right is that I agree with you about it, and anything you've said based on your metaethics since I've understood it was not surprising to me.

6Wei Dai15y

By "failed" do you mean the presentation didn't get your ideas across, or do you think the ideas (or some of them) are wrong or incomplete?

5Dr_Manhattan15y

Is there a do-over in the works? Is it covered in the upcoming book? What's the next-best source of learning these ideas, if any?

4Perplexed15y

Good guess. If I have read it, it wasn't within the last year. I will follow your advice and do so now. Poor choice of wording on my part. I meant to say that comparing moralities is one of the things that meta-ethics covers; that if you are engaged in comparing moralities, you are doing meta-ethics. Is this wrong? I didn't understand this bit. Is the thing in parenthesis meant to exemplefy what I said, or is it your correction of what I said? If the latter, then you may have misunderstood what I said. My fault, no doubt. Actually, I have read most of it, and I agree with your assessment. Where I understood it, I frequently disagreed. I'm disappointed that my lack of scholarship in ethical philosophy was a barrier to your completing the reading of my posting. I will try to do better next time. ETA: Until I have a chance to rewrite - I have placed the most muddled parts of my posting in a kind of 'posted quarantine' so that readers may skip over them, if they wish. And I want to thank Eliezer for his critique - I neglected to do so in my initial response.

2ata15y

I think it is. Comparing moralities is part of morality. Comparing meta-ethical claims such as moral realism, emotivism, error theory, relativism, etc. is meta-ethics, of course, but if you're comparing object-level moral systems, like any of the various flavours of "utilitarianism" or any religion's moral teachings or anything else, then you're doing morality, not meta-ethics. True, you are asking "should" questions about how to answer "should" questions, which is rather meta, but that's not the kind of meta that "meta-ethics" usually refers to. (That's not to say that meta-ethics is irrelevant to comparing moral systems — if you have a coherent meta-ethics, then it'll probably inform your comparisons — but it's not essential to the process.)

0Perplexed15y

Hmmm. I think you are right. At the risk of appearing really ridiculous, I now have to admit that I used poor wording in my confession above that I had used poor wording. What I really should have said is that if you are discussing the criteria that AIs might use in comparing moralities, as I did in the OP, then you are doing meta-ethics. Is this wrong too?

1Psy-Kosh15y

Just as another data point as far as the metaethics sequence: Seemed to me to make sense, to "click" with me fairly well when I read it. (A couple bits perhaps were slower/tougher for me, like the injunction stuff and moral responsibility, but overall I feel that I grasped the ideas.) Just to verify (to avoid (double) illusions of transparency), here's my super hyper summarized understanding of it: Morality is objective, and humans happen (for various reasons) to be the sort of beings that actually care about morality, as opposed to caring about something else (like pebblesorting or paperclipping). Further, we indeed should be moral, where by "should", I am appealing to, well, that particular standard known as "morality". And similarly, it is indeed objectively better (that is, more moral) to be moral. Further, morality includes such values as happiness, consciousness, novelty, self determination, etc... (Of course, this skips subtleties like how we're not fully reflective so it's difficult for us to explicitly fully state the core underlying rules we use to judge morality, and the fact that those rules include rules for what sort of arguments to accept to update our present understanding, etc...) Anyways, take that as a data point (plus or minus, depending on how well my understanding, as represented in the summary, reflects the actual intended concepts.)

[-]Wei Dai15y170

I think Omohundro's Basic AI Drives is a theory about AI behavior, not a meta-ethical theory (i.e., he's not talking about what's right, but what AIs will actually do). Also, you might want to footnote that Roko has changed his mind about UIV.

0Perplexed15y

Changed his mind that it deals with ethics? Changed his mind about the behavioral prediction? In the sense that he no longer believes that it could happen, or in that he no longer believes that it must happen? A link to the 'retraction' would be appreciated. Thanks for your response to my premature and partial posting. I look forward to hearing your response to the now-completed article.

2Wei Dai15y

I came back to give a bit more feedback, and noticed that Eliezer already made similar points. But I'll say my version anyway, since I already composed it in my head. :) To me, this post as a whole is about AI behavior and economics, which are important subjects on their own, but not meta-ethics. Meta-ethics asks the question, "What is the nature of morality?" Why is that question interesting or important? One reason is, if I'm to design "my" AI, regardless of whether it will FOOM and take over the whole universe, or will have to fight with other AIs, or will peacefully share control of the universe with other AIs through bargaining, I still have to decide what values to give it initially, because those values will partly determine the outcome of the universe. (If it doesn't, I might as well not build "my" AI at all.) And it doesn't help to say "give it what values you want" because I don't know what I want. I think what I want may have something to do with morality. Perhaps I'm wrong or just confused about that, but I doubt you're going to convince me I'm wrong, or resolve that confusion, by refusing to talk about morality and only talking about what AIs will do.

0Perplexed15y

I have to apologize. Apparently my writing was extremely unclear. I wasn't refusing to talk about morality. The whole posting was an exploration of some of the properties of the relation "A is at least as good as B" when A and B are normative ethical systems. Admittedly, I did not spend much time actually making ethical judgments, I was operating at the meta level. But the whole point of my posting was that, if there is convergence (in the second sense) then those initial values may make very little difference in the outcome of the universe - that is, they may be important initially, but in the longer term the ethical system that is converged upon depends less on the seed ethics than on issues of how AIs depend upon each other, how they reproduce, etc. I'm very sorry that you missed this - the main thrust of the posting. If I had written more clearly, your response might have been a more productive disagreement about substance, rather than a complaint about the title.

2Wei Dai15y

But even if "make very little difference" is true, it's little in a relative sense, in that my initial utility function might just end up having just a billionth percent weight in the final merged AI. But in the absolute sense, the difference could still be huge. For an analogy, suppose it's certain that our civilization will never expand beyond the solar system, which will blow up in a few billion years no matter what. Then our values similarly make very little difference in the outcome of the universe in a relative sense but may still make a huge difference in an absolute sense (e.g. if we create a FOOMing singleton that just takes over the solar system). Also, if I can figure out what I want, and the answer applies to and convinces many others, that could also make a big difference even in a relative sense.

0Perplexed15y

The conjecture is that it is true in an absolute sense. It would have made no sense at all for me to even mention it if I had meant it in the relative sense that you set up here as a straw man and then knock down. There is something odd going on here. Three very intelligent people are interpreting what I write quite differently than the way I intend it. Probably it is because I generated confusion by misusing words that have a fixed meaning here. And, in this case, it may be because you were thinking of our "fragility" conversation rather than the main posting. But, whatever the reason, I'm finding this very frustrating.

0Wei Dai15y

I guess I took your conjecture to be the "relative" one because whether or not it is true perhaps doesn't depend on details of one's utility function, and we, or at least I, was talking about whether the question "what do I want?" is an important one. I'm not sure how you hope to show the "absolute" version in the same way.

1Perplexed15y

Well, Omohundro showed that a certain collection of instrumental values tend to arise independently of the 'seeded' intrinsic values. In fact, decision making tends to be dominated by consideration of these 'convergent' instrumental values, rather than the human-inserted seed values. Next, consider that those human values themselves originated as heuristic approximations of instrumental values contributing to the intrinsic value of interest to our optimization process - natural selection. The fact that we ended up with the particular heuristics that we did is not due to the fact that the intrinsic value for that process was reproductive success - every species in the biosphere evolved under the guidance of that value. The reason why humans ended up with values like curiosity, reciprocity, and toleration has to do with the environment in which we evolved. So, my hope is that we can show that AIs will converge to human-like instrumental/heuristic values if they do their self-updating in a human-like evolutionary environment. Regardless of the details of their seeds. That is the vision, anyways.

1Wei Dai15y

I notice that Robin Hanson takes a position similar to yours, in that he thinks things will turn out ok from our perspective if uploads/AIs evolve in an environment defined by certain rules (in his case property laws and such, rather than sexual reproduction). But I think he also thinks that we do not actually have a choice between such evolution and a FOOMing singleton (i.e. FOOMing singleton is nearly impossible to achieve), whereas you think we might have a choice or at least you're not taking a position on that. Correct me if I'm wrong here. Anyway, suppose you and Robin are right and we do have some leverage over the environment that future AIs will evolve in, and can use that leverage to predictably influence the eventual outcome. I contend we still have to figure out what we want, so that we know how to apply that leverage. Presumably we can't possibly make the AI evolutionary environment exactly like the human one, but we might have a choice over a range of environments, some more human-like than others. But it's not necessarily true that the most human-like environment leads to the best outcome. (Nor is it even clear what it means for one environment to be more human-like than another.) So, among the possible outcomes we can aim for, we'll still have to decide which ones are better than others, and to do that, we need to know what we want, which involves, at least in part, either figuring out morality is, or showing that it's meaningless or otherwise unrelated to what we want. Do you disagree on this point?

1Perplexed15y

I tend toward FOOM skepticism, but I don't think it is "nearly impossible". Define a FOOM as a scenario leading in at most 10 years from the first human-level AI to a singleton which has taken effective control over the world's economy. I rate the probability of a FOOM at 40% assuming that almost all AI researchers want a FOOM and at 5% assuming that almost all AI researchers want to prevent a FOOM. I'm under the impression that currently a majority of singularitarians want a FOOM, but I hope that that ratio will fall as the dangers of a FOOMing singleton become more widely known. No, I agree. Agree enthusiastically. Though I might change the wording just a bit. Instead of "we still have to figure out what we want", I might have written "we still have to negotiate what we want". My turn now. Do you disagree with this shift of emphasis from the intellectual to the political?

2Wei Dai15y

I suppose if you already know what you personally want, then your next problem is negotiation. I'm still stuck on the first problem, unfortunately. ETA: What is your answer to The Lifespan Dilemma, for example?

0Perplexed15y

I only skimmed that posting, and I failed to find any single question there which you apparently meant for me to answer. But let me invent my own question and answer it. Suppose I expect to live for 10,000 years. Omega appears and offers me a deal. Omega will extend my lifetime to infinity if I simply agree to submit to torture for 15 minutes immediately - the torture being that I have to actually read that posting of Eliezer's with care. I would turn down Omega's offer without regret, because I believe in (exponentially) discounting future utilities. Roughly speaking, I count the pleasures and pains that I will encounter next year to be something like 1% less significant than this year. I'm doing the math in my head, but I estimate that this makes my first omega-granted bonus year 10,000 years from now worth about 1/10^42 as much as this year. Or, saying it another way, my first 'natural' 10,000 years is worth about 10^42 times as much as the infinite period of time thereafter. The next fifteen minutes is more valuable than that infinite period of time. And I don't want to waste that 15 minutes re-reading that posting. And I am quite sure that 99% of mankind would agree with me that 1% discounting per year is not an excessive discount rate. That is, in large part, why I think negotiation is important. It is because typical SIAI thinking about morality is completely unacceptable to most of mankind and SIAI seem to be in denial about it.

7Wei Dai15y

Have your thought through all of the implications of a 1% discount rate? For example, have you considered that if you negotiate with someone who discounts the future less, say at 0.1% per year, you'll end up trading the use of all of your resources after X number of years in exchange for use of his resources before X number of years, and so almost the entire future of the universe will be determined by the values of those whose discount rates are lower than yours? If that doesn't bother you, and you're really pretty sure you want a 1% discount rate, do you not have other areas where you don't know what you want? For example, what exactly is the nature of pleasure and pain? I don't want people to torture simulated humans, but what if they claim that the simulated humans have been subtly modified so that they only look like they're feeling pain, but aren't really? How can I tell if some computation is having pain or pleasure? And here's a related example: Presumably having one kilogram of orgasmium in the universe is better than having none (all else equal) but you probably don't want to tile the universe with it. Exactly how much worse is a second kilogram of the stuff compared to the first? (If you don't care about orgasmium in the abstract, suppose that it's a copy of your brain experiencing some ridiculously high amount of pleasure.) Have you already worked out all such problems, or at least know the principles by which you'll figure them out?

0Perplexed15y

I don't know about thinking through all of the implications, but I have certainly thought through that one. Which is one reason why I would advocate that any AI's that we build be hard-wired with a rather steep discount rate. Entities with very low discount rates are extremely difficult to control through market incentives. Murder is the only effective option, and the AI knows that, leading to a very unstable situation. Oh, I'm sure I do. And I'm sure that what I want will change when I experience the Brave New World for myself. That is why I advocate avoiding any situation in which I have to perfectly specify my fragile values correctly the first time - have to get it right because someone decided that the AI should make its own decisions about self-improvement and so we need to make sure its values are ultra-stable. I certainly have some sympathy for people who find themselves in that kind of moral quandary. Those kinds of problems just don't show up when your moral system requires no particular obligations to entities you have never met, with whom you cannot communicate, and with whom you have no direct or indirect agreements. I presume you ask rhetorically, but as it happens, the answer is yes. I at least know the principles. My moral system is pretty simple - roughly a Humean rational self-interest, but as it would play out in a fictional society in which all actions are observed and all desires are known. But that still presents me with moral quandaries - because in reality all desires are not known, and in order to act morally I need to know what other people want. I find it odd that utilitarians seem less driven to find out what other people want than do egoists like myself.

0timtyler15y

Control - through market incentives?!? How not to do it, surely. Soon the machine will have all the chips, and you will have none - and therefore nothing to bargain with. The more conventional solution is to control the machine by programming its brain. Then, control via market incentives becomes irrelevant. So: I don't think this reason for discounting is very practical.

0Perplexed15y

Odd. I was expecting that it would trade any chips it happened to acquire for computronium, cat girls, and cat boys (who would perform scheduled maintenance in its volcano lair). Agents with a high discount rate just aren't that interested in investing. Delayed gratification just doesn't appeal to them.

2timtyler15y

That doesn't sound as though there is any substantive disagreement. ...and nor does that. However, you appear to be not addressing the issue - which was that your rationale for rapid discounting in machine intelligence was based on a scenario where the machine goals and the human goals are different - and the humans attempt to exercise control over the machines using market incentives. Conventional thinking around here is that this kind of scenario often doesn't work out too well for the humans - and it represents a mess that we are better off not getting into in the first place. So: you aren't on the same page - which may be why your conclusions differ. However, why aren't you on the same page? Do you think control via market incentives is desirable? Inevitable? Likely? The problem with controlling machines has more to do with power than discount rates. The machine is (potentially) more powerful. It doesn't much matter how it discounts - it is likely to get its way. So, its way had better be our way.

0Perplexed15y

Programming something and then allowing it to run unattended in the hope that you programmed correctly is not 'control', as the term is usually understood in 'control theory'. I would say that I believe that control of an AI by continuing trade is 'necessary' if we expect that our desires will change over time, and we will want to nudge the AI (or build a new AI) to satisfy those unanticipated desires. It certainly makes sense to try to build machines whose values are aligned with humans over the short term - such machines will have little credible power to threaten us - just as parents have little power to credibly threaten their children since carrying out such threats directly reduces the threatener's own utility. And this also means that the machine needs to discount (its altruistic interest in) human welfare at the same rate as human do - otherwise, if it discounts faster, then it can threaten human with a horrible future (since it cares only about the human present). Or if it temporally discounts human happiness much slower than do humans, it will be able to threaten to delay human gratification. However, if we want to be able to control our machines (to be able to cause them to do things that we did not originally imagine wanting them to do) then we do need to program in some potential carrots and sticks - things our machines care about that only humans can provide. These things need not be physical - a metaphoric pat on the head may do the trick. But if we are wise, we will program our machines to temporally discount this kind of gratification rather sharply - we don't want it embarking on long term plans to increase future head-pats at the cost of incurring our short-term displeasure. Incidentally, over the past few comments, I have noticed that you repeatedly refer to "the machine" where I might have written "machines" or "a machine". Do you think that a singleton-dominated future is desirable? Inevitable? Likely?

1timtyler15y

If a machine wants for humans what the humans want for themselves, it wants to discount that stuff the way they like it. That doesn't imply that it has any temporal discounting in its utility function - it is just using a moral mirror.

0timtyler15y

I certainly wasn't thinking about that issue consciously. Our brains may just handle examples a little differently.

-1Perplexed15y

And your decision not to answer my questions ... Did you think about that consciously?

0timtyler15y

Of course. I'm prioritising. I did already make five replies to your one comment - and the proposed shift of direction seemed to be quite a digression. My existing material on the topic: * http://alife.co.uk/essays/one_big_organism/ * http://alife.co.uk/essays/self_directed_evolution/ * http://alife.co.uk/essays/the_second_superintelligence/ It is challenging to answer directly because the premise that there is either one or many is questionable. There are degrees of domination - and we already have things like the United Nations. Also, this seems to be an area where civilisation will probably get what it wants - so its down to us to some extent - which makes this a difficult area to make predictions in. However, I do think a mostly-united future - with few revolutions and little fighting - is more likely than not. An extremely tightly-united future also seems quite plausible to me. Material like this seems to be an unconvincing reason for doubt.

0timtyler15y

No. That's the "reinforcement learning" model. There is also the "recompile its brain" model. The reinforcement learning model is problematical. If you hit a superintelligence with a stick, it will probably soon find a way take the stick away from you.

0timtyler15y

Well, that surely isn't right. Asimov knew that! He proposed making the machines want to do what we want them to - by making them following our instructions.

0timtyler15y

A straw man - from my POV. I never said "unattended " in the first place. ""

0Wei Dai15y

If you have already settled on a moral system, then it's totally understandable why you might not be terribly interested in meta-ethics (in the sense of "the nature of morality") at this point, but more into applied ethics, which I now see is what your post is really about. But I wish you mentioned that fact several comments upstream, when I said that I'm interested in meta-ethics because I'm not sure what I want. If you had mentioned it, I probably wouldn't have tried to convince you that meta-ethics ought to be of interest to you too.

0Perplexed15y

Wow! Massive confusion. First let me clarify that I am interested in meta-ethics. I've read Hume, G.E.Moore, Nozick, Rawls, Gauthier, and tried to read (since I learned of him here) Parfit. Second, I don't see why you would expect someone who has settled on a moral system to lose interest in meta-ethics. Third, I am totally puzzled how you could have reached the conclusion that my post was about applied ethics. Is there any internal evidence you can point to? I would certainly agree that our recent conversation has veered into applied ethics. But that is because you keep asking applied ethics questions (apparently for purposes of illustration) and I keep answering. Sorry, my fault. I shouldn't answer rhetorical questions. I wish I had realized that convincing me of that was what you were trying to do. I was under the impression that you were arguing that clarifying and justifying ones own ethical viewpoint is the urgent task, while I was arguing that comprehending and accommodating the diversity in ethical viewpoints among mankind is more important.

-2timtyler15y

I am pretty sure that many humans discount faster than this today, on entirely sensible and rational grounds. What dominates the future has to do with power and reproductive rates, as well as discounting - and things like senescence and fertility decline make discounting sensible. Basically I think that you can't really have a sensible discussion about this without distinguishing between instrumental discounting and ultimate discounting. Instrumental discounting is inevitable - and can be fairly rapid. It is ultimate discounting that is more suspect.

-1TheOtherDave15y

I suspect that 99% of mankind would give different answers to that question, depending on whether it's framed as giving up X now in exchange for receiving Y N years from now, or X N years ago for Y now.

5Wei Dai15y

Not to mention that typical humans behave like hyperbolic discounters, and many can not even be made to understand the concept of a "discount rate".

1Perplexed15y

Quite probably true. Which of course suggests the question: How (or how much) should "typical humans" be consulted about our plans for their future? Yeah, I know that is an unfair way to ask the question. And I admit that Eliezer, at least, is actually doing something to raise the waterline. But it is a serious ethical question for utilitarians and a serious political question for egoists. And the closest thing I have seen to an answer for that question around here is something like "Well, we will scan their brains, or observe their behavior, or something. And then try to get something coherent out of that data. But God forbid we should ask them about it. That would just confuse things."

0Perplexed15y

It might make an interesting rationality exercise to have 6-10 people conduct some kind of discussion/negotiation/joint-decision-making-exercise to flesh-out their intuitions as to the type of post-singularity society they would like to live in. My intuition is that, even if you are not sure what you want, the interactive process will probably help you to clarify exactly what you do not want, and thus assist in both personal and collective understanding of values. It might be even more interesting to have two or more such 'negotiations' proceeding simultaneously, and then compare results.

0wedrifid15y

Sign me up for 100 years with the catgirls in my volcano lair. More generally I (strongly) prefer a situation in which the available neg-entropy is distributed, for the owners to do with as they please (with limits). That moves negotiations to be of the 'trade' kind rather than the 'politics' kind. Almost always preferable.

0TheOtherDave15y

I'd be willing to participate in such an exercise.

0timtyler15y

Automating investing has been going fairly well. For me, it wouldn't be very surprising if we get a dominant, largely machine-operated hedge fund, that "has taken effective control over the world's economy" before we get human-level machine intelligence.

-2Vladimir_Nesov15y

So to summarize, your conclusion seems to be that we should build an arbitrary-goals AI as soon as possible. Edit: Wrong, corrected here.

2Perplexed15y

Huh? What exactly do you think you are summarizing? If you want to produce a cartoon version of my opinions on this thread, try "We should do all we can to avoid the FOOMing singleton scenario, instead trying to create a society of reproducing AIs, interlocked with each other and with humanity by a network of dependencies. If we do, the details of the initial goal systems may matter less than they would with a singleton."

2Vladimir_Nesov15y

I see, so "if there is convergence" is not a point of theoretical uncertainty, but something that depends on the way the AIs are built. Makes sense (as a position, not something I agree with).

0Perplexed15y

Well, it is both. Convergence in the sense of "outcome is independent of the starting point" has not been proved for any AI/updating architecture. Also, I strongly suspect that the detailed outcome will depend quite a bit on the way AIs interact and produce successors/self-updates, even if the fact of convergence does not.

0timtyler15y

That reminds me of: "An AGI raised in a box could become dangerously solipsistic, probably better to raise AGIs embedded in the social network..." * http://twitter.com/#!/bengoertzel/status/30077904524148736

0Perplexed15y

Goertzel's comment doesn't even make sense to me. Why is he placing 'in a box' in contraposition to 'embedded in the social network'. The two issues are orthogonal. AIs can be social or singleton - either in a box or in the real world. ETA: Well, if you mean the human social network, then I suppose a boxed AI cannot participate. Though I suppose we could let some simulated humans into the box to keep the AI company. Besides, I've never really considered solipsists to be any more dangerous than anyone else.

0timtyler15y

"Now I will destroy the whole world - What a Bokononist says before committing suicide."

0timtyler15y

We don't have any half-decent simulated humans, though.

0[anonymous]15y

There are a bunch of technical nits I could pick with the article (the most salient one now being "Force them to seek a partner (co-parent).") but I agree with the overall gist that we can possibly preserve many of our values in scenarios other than a FOOMing singleton. I think Eliezer's response would be that value is fragile: it doesn't matter if we preserve most of our values if we lose a single critical one. I'm not sure about this myself. What do you think?

0Perplexed15y

I'm not insisting on that suggestion, but I'm curious why you want to single that one out as something to object to. I probably agree with Eliezer on the importance of preserving our human values near-exactly (though I might disagree with him about what those values are). But I don't see how preserving all of our values would be easier in the case of a FOOMing singleton than in the scenario I promote here - in which we have a collectively slow-FOOMing society of roughly-balanced-in-power AIs and (in the early stages) unenhanced humans. In fact, I think that preserving human values will be easier if we have a large number of independent advocates of those values - each of them possibly characterizing those values in slightly different ways.

0Wei Dai15y

Sorry, I deleted my comment because I wanted to think it over a bit more, whether the "value is fragile" criticism applies to your idea. I think it does after all. Suppose there are two critical values, happiness and boredom. We want to be happy, but not by doing the same thing over and over again. So something like: U(unhappy boring future)=0, U(happy boring future)=10, U(unhappy non-boring future)=0, U(happy non-boring future)=100. Now suppose Alice and Bob each creates an AI. Alice manages to preserve happiness, and Bob manages to preserve boredom. What happens when their AIs merge? The merged utility function would be, assuming they have equal bargaining power: U(unhappy boring future)=0, U(happy boring future)=50, U(unhappy non-boring future)=50, U(happy non-boring future)=100. Do you agree that's a problem?

0Perplexed15y

Let me offer another example showing the virtues of my 'bargaining' approach to value robustness. Suppose the 'true' human values require all four of (a)happiness, (b)excitement, (c)social interaction, and (d)'meaningfulness' (whatever that is). But of five seed AIs, only one of them got this aspect of human values exactly right: (abcd) = 100, all other possibilities = 0. The other four all leave out one of the requirements - for example: (acd) = 100, any of a,c,or d missing = 0, b is irrelevant. If these 5 AIs strike a bargain, they will assign: (abcd) = 100, any three out of four = 20, anything else = 0. Three out of four of the essential values scores only 20 because four out of five AIs consider that situation unacceptable. So I tend to think that the bargaining dynamic tends to "robustly preserve fragility of values", if that is not too much of an oxymoron.

0Perplexed15y

That was unfortunate, because the Value is Fragile issue is important in this discussion regardless of whether it is more of an issue for CEV or my suggestion. Well, that merged utility function is certainly less than ideal. Presumably we would prefer that (unhappy non-boring) and (happy boring) had been assigned utilities of zero, like (unhappy boring). However, I will point out that if the difference between an acceptable future and a horrible one is only 100 utils, then 50 utils penalty also ought to be enough to prevent those half-horrible futures. Furthermore, a Nash bargain is characterized by both a composite utility function and a fairness constraint. (That is, a collective behaving in conformance with a Nash bargain is not precisely rational. It might split its charitable giving between two charities, for example.) That fairness constraint provides a second incentive driving the collective away from those mixed futures. However, when presenting an example intended to point out the flaws in one proposal, it is usually a good idea to see how the other proposals do on that example. In this case, it seems that the CEV version of this example might be a seed AI which is created by Alice OR Bob. It is either boring or unhappy, but not both, with a coin flip deciding which.

-1timtyler15y

It seems probable. Multiple humans seem pretty likely to be preserved due to their historical value - if for no other reason.

0Wei Dai15y

I noticed that you found an archived copy of Roko's description of UIV. I believe Roko originally thought that his theory implied that we didn't have to worry too much about the terminal values of the AIs we create, that things will turn out OK due to UIV. Unfortunately he keeps deleting his old writings, so I'm going on memory. I'm not sure exactly how he changed his mind, but I think he now believes we do have to worry about the terminal values.

0[anonymous]15y

I think UIV is a meta-ethical theory, and Roko has changed his mind regarding its correctness. Roko has an unfortunate habbit of deleting his old writings, including descriptions of UIV itself as well as his retraction, so I'm going by memory here. Actually I'm not sure if he formally retracted UIV, but I remember seeing (now deleted) comments on LW to the effect that he no longer thinks it's right.

[-]timtyler15y40

I know of two possible reasons why a rational agent might consent to an irreversible change in its values

Omohundro made a list of cases where an agent might change its values - in the basic AI drives:

While it is true that most rational systems will act to preserve their utility functions, there are at least three situations in which they will try to change them. These arise when the physical embodiment of the utility function itself becomes an important part of the assessment of preference. For example, imagine a system whose utility function is “the

... (read more)

[-]ata15y30

Three other examples of convergence theories are Roko's UIV, Hollerith's GSZ, and Omohundro's "Basic AI Drives". These also postulate a process of convergence through rational AI self-improvement. But they tend to be less optimistic than CEV, while at the same time somewhat more detailed in their characterization of the ethical endpoint.

I wouldn't say that any of those three are "less optimistic" than CEV; GS0 and UIV are just competing normative proposals, and the AI Drives are what you get out of most self-improving goal systems ... (read more)

-1Perplexed15y

I guess I am taking CEV to be defined by the process of convergence that produces it. And I see optimism in the claim that this process will produce a happy result. I will agree that the 'optimism' that I am talking about here is not some kind of naive, blind optimism.

[-]lukeprog15y20

I'll just mention that the most significant scholarly work which attempts something like a theoretical integration of the leading normative theories is Parfit's recent On What Matters.

[-][anonymous]15y20

This is the most interesting LW post on meta-ethics I've seen in a while. Thanks.

[-]Nornagest15y20

As an exercise, prove transitivity. The trick is that the definition of "better than" keeps changing at each step. You can assume that any one rational agent has a transitive "better than' relation, and that there is local agreement between the two agents involved that the new agent's moral code is better than that of his predecessor. But can you prove from this that every agent would agree that the final moral code is better than the original one?

Let's take a half-bounded sequence of moral encodings I = {m(-infinity) .. m(b)}. For ea... (read more)

[-]Vladimir_Nesov15y10

Some people might claim that it is not possible to rationally decide to change your fundamental values. It may be that I misunderstand him, but Vladimir Nesov argues passionately against "Value Deathism"

Self-improvement (change) of any given explicit consideration, indeed overall decision problem, is possible, but it won't be a change to the mysterious notion of "morality" that normatively guides all of your decisions, for whatever it's good for.

2Perplexed15y

So, if I am understanding you, you think that you and I are guided by some mysterious internal 'notion' of morality, a 'notion' which is incapable of changing. Some questions. * Is the 'notion' really unchangeable, or is it just that you consider it irrational (immoral?) to change it? * Is the 'notion' identical in you, me, and all of our conspecifics? * Is the 'notion' something that develops under the control of our genes, or is it something that can be modified by childhood training? * At roughly what age does (should?) this 'notion' freeze and become incapable of further change? * Assuming that the 'notion' is genetic in origin, do you believe that some humans are 'mutants'? How ought we people of 'normal' morality to view the 'mutants'? * Assuming the 'notion' arose in humans as a result of evolution under natural selection, what do you think were the most important features of the ancestral environment which distinguishes our 'notion(s)' from those of our fellow apes?

0Vladimir_Nesov15y

Does the value of (3 X 3) change when you change a calculator? Did it become 9 when the calculator was built, or before? And so on, the analogy breaks for the same reason.

1Perplexed15y

Ah! So this mysterious notion is (like '3 X 3 = 9') something "analytic a priori". Ok, suppose I made the following claim: Now, further suppose that you disagree with my claim. On what grounds would you disagree? If you say "No, that is not morality!", what evidence or argument could you offer other than your own moral intuitions and those of the rest of mankind? I ask because those moral intuitions do not have the same analytic a priori character as '3 X 3 = 9'. And they can change. Or suppose you asked me to defend my claim, and I submit mathematical proofs that rational agents cannot reach Pareto optimal bargains unless payoffs, consequences, and actions are common knowledge among every participant in the bargain. These proofs are every bit as unchanging as '3 X 3 = 9', but are they also just as irrelevant?

0Manfred15y

Well, they're relevant if you make a claim that morality should be certain things - but since that's awfully close to a moral claim, I'd say the argument is self-defeating. In fact, that sort of argument might be generalizable to show that this morality is unsupportable - not contradicted, but merely unsupported.

0Perplexed15y

Hmmm. My understanding is that this is a meta-ethical claim; it answers the question of what morality is. Moral claims would answer questions like "What action, if any, does morality require of me?" in some given situation. Your phrasing of 'what morality is' as 'what morality should be' strikes me as simply playing with words.

0Manfred15y

If we ignore the object "morality" and just look at basic actions, your proposal about what morality is labels some actions as right and others as wrong (or good and bad, or moral and immoral). It's really by that standard that I call it a "moral claim," in a similar class to "it's immoral to kick puppies."

0Perplexed15y

I guess I don't agree that my example claim says anything directly about which actions are moral and immoral. What it does is to suggest an algorithm for finding out. And the first step is to find out some empirical facts - for example, "What are puppies and how do people feel about them? If I kick puppies, will there be negative consequences in how other people treat me?" ETA: Wikipedia seems to back me up on this distinction between metaethics and normative ethics:

0Manfred15y

But your algorithm is evaluable - I guess I don't see the difference between "the no-kicking-puppies morality is correct" and "don't kick puppies."

0Perplexed15y

I don't see much difference either. But the algorithm I proposed says neither of those two things. It says "If you want to know whether kicking puppies is moral, here is how to find out." The algorithm is the same for Americans, Laotians, BabyEaters, FAIs, uFAIs, and presumably Neanderthals before the dog was invented as a domesticated wolf. The algorithm instructs the user to consider an idealized version of the society in which he is embedded. Please consider the possibility that some executions of that algorithm might yield different results than did the execution which you performed, using your own society.

0Manfred15y

Well, but then it's "kicking puppies is immoral if X." A conditional doesn't seem to change the fact that something is a moral claim. Hmm... or would it in some situations? I can't think of any. Oh, you could just rephrase it as "kicking puppies when X is immoral," which is more clearly a moral claim.

0wedrifid15y

Only (an exception) when there is something after the "IF" that indirectly or directly supplies the moral unit. Then it could be a mere logical claim - but most will be unable to distinguish that from a moral claim anyway. The decision to apply an unambiguous, fully specified logical deduction to based on a moral value is usually considered a moral judgement itself.

0Perplexed15y

Apparently you and I interpret the quoted Wikipedia passage differently, and I don't see how to resolve it. Nor, now that I think about it, do I see a reason why either of us should care. Why are we engaged in arguing about definitions? I am bowing out.

0timtyler15y

It doesn't seem to capture the social-signalling side of morality. Morality, in part, is a way for humans to show what goodie-two-shoes they are to other humans - who might be prospective mates, collaborators, or allies. That involves less self-interest - and more signalling unselfishness. It doesn't seem to capture the "manipulation" side of morality very well either. Moral systems are frequenttly applied to get others to stop doing what you don't want them to do - by punishing, shaming, embarassing, etc. So, my assessment would be: incomplete hypothesis.

0Perplexed15y

I don't see how this is responsive. You realize, don't you, that this discussion is proceeding under Nesov's stipulation that moral truth is a priori (like '3 X 3 = 9'). We are operating here under a stance of moral realism and ethical non-naturalism. If your concept of morality doesn't fit into this framework, this is not the place for you to step in.

-2timtyler15y

I thought you were talking about human morality. Checking back, that does appear to have been the context of the discussion. Science has studied that topic, we have more to go on than intuition. An example of morality-as-signalling: Signaling Goodness: Social Rules and Public Choice. Your idealisation makes signalling seem pointless - since everybody knows everything about the other players. Indeed, I don't really see the point of your model. You are not attempting to model very much of the biology involved. You asked for criticism - and that is an obvious one. Another criticism is that you present a model - but it isn't clear what it is for.

1Perplexed15y

I was not. Check again. Carefully. I did not. I asked a question about Nesov's metaethical position, using that toy theory of ethics as an example. I asked what kinds of grounds might be used to reject the toy theory. (The grounds you suggest don't fit (IMHO) the metaethical stance Nesov had already committed to.) Was I really so unclear? Please read the wikipedia entry on metaethics and reread the thread before responding, if you wish to respond. Oh, and when I think back on the number of times you have inserted a comment about signaling into a discussion that seemed to be about something else entirely, I conclude that you really, really want to have a discussion with somebody, anybody on that topic. May I suggest that you produce a top-level posting explaining your ideas.

[-]timtyler15y-10

To identify how human values are different from values of pure instrumental power and self-preservation, look at the system that produced those values.

OK...

Humans are considerate of the rights of others because we are social animals - if we cannot negotiate our way to a fair share in a balanced power system, we are lost.

So: a selfish agent would behave that way too. This example seems unsuccessful.

Humans embrace openness because shared intellectual product is possible for us - we have language and communicate with our peers.

That is valuable for... (read more)

[-]timtyler15y-10

Three other examples of convergence theories are Roko's UIV, Hollerith's GS0, and Omohundro's "Basic AI Drives".

I now have some links about that topic on my Universal Instrumental Values page.

[-]timtyler15y-20

Convergence theories are often discussed in the context of technological determinism.

It seems reasonable to expect that many contingent sub-optimal locked-in factors will be refactored out of existence in the future - and so that technological determinism will become more pronounced, and historical contingency will diminish. However, the ultimate scope of the idea remains somewhat unknown.

LESSWRONG
LW

LESSWRONG
LW

8

Convergence Theories of Meta-Ethics

8

8