138

LESSWRONG
LW

137
Personal Blog

-16

Waser's 3 Goals of Morality

by mwaser
2nd Nov 2010
1 min read
25

-16

Personal Blog

-16

Waser's 3 Goals of Morality
8jimrandomh
-6mwaser
12jimrandomh
-5mwaser
9NihilCredo
-2mwaser
14grouchymusicologist
6Kingreaper
-1mwaser
6NihilCredo
0mwaser
2DaveX
2Kingreaper
1mwaser
2Kingreaper
-7mwaser
10Kingreaper
-5mwaser
14Alicorn
3mwaser
12jimrandomh
5Alicorn
9Kingreaper
6Vladimir_Nesov
4mwaser
New Comment
25 comments, sorted by
top scoring
Click to highlight new comments since: Today at 11:22 AM
[-]jimrandomh15y80

This is too confused to follow as a human, and much too confused to program an AI with.

Also ambiguity aside, (2) is just bad. I'm having trouble imagining a concrete interpretation of "don't over-optimize" that doesn't reduce to "fail to improve things that should be improved". And while short-sightedness is a problem for humans who have trouble modelling the future, I don't think AIs have that problem, and there are some interesting failure modes (of the destroys-humanity variety) that arise when an AI takes too much of a long view.

Reply
[+]mwaser15y-60
[+]mwaser15y-70
Moderation Log
More from mwaser
View more
Curated and popular this week
25Comments

In the spirit of Asimov’s 3 Laws of Robotics

  1. You should not be selfish
  2. You should not be short-sighted or over-optimize
  3. You should maximize the progress towards and fulfillment of all conscious and willed goals, both in terms of numbers and diversity equally, both yours and those of others equally

It is my contention that Yudkowsky’s CEV converges to the following 3 points:

  1. I want what I want
  2. I recognize my obligatorily gregarious nature; realize that ethics and improving the community is the community’s most rational path towards maximizing the progress towards and fulfillment of everyone’s goals; and realize that to be rational and effective the community should punish anyone who is not being ethical or improving the community (even if the punishment is “merely” withholding help and cooperation)
  3. I shall, therefore, be ethical and improve the community in order to obtain assistance, prevent interference, and most effectively achieve my goals

I further contend that, if this CEV is translated to the 3 Goals above and implemented in a Yudkowskian Benevolent Goal Architecture (BGA), that the result would be a Friendly AI.

It should be noted that evolution and history say that cooperation and ethics are stable attractors while submitting to slavery (when you don’t have to) is not.  This formulation expands Singer’s Circles of Morality as far as they’ll go and tries to eliminate irrational Us-Them distinctions based on anything other than optimizing goals for everyone — the same direction that humanity seems headed in and exactly where current SIAI proposals come up short.

Once again, cross-posted here on my blog (unlike my last article, I have no idea whether this will be karma'd out of existence or not ;-)