I am far from convinced that people in general wish to be seen as caring more about morality than they actually do.  If this was the case, why would the persistent claim that people are -- and, logically, must be -- egoists have so long survived strong counter-arguments?  The argument appears to me to be a way of signaling a lack of excessive and low status moral scruples. 

It seems to me that the desire to signal as much morality as possible is held by a minority of women and by a small minority of men.  Those people are also the main people who talk about morality.  This is commonly a problem in the development of thought.  People with an interest in verbally discussing a subject may have systematically atypical attitudes towards that subject.  Of course, this issue is further complicated by the fact that people don't agree on what broad type of thing morality is. 

The conflict within philosophy between Utilitarians and Kantians is among the most famous examples of this disagreement. <a href=" http://people.virginia.edu/~jdh6n/moraljudgment.html”> Haidt’s views on conservative vs. liberal morality </a> is another.  Major, usually implicit disagreements regard whether morality is supposed to serve as a decision system, a set of constraints on a decision system, or a set of reasons that should influence a person along with prudential, honor, spontaneity, authenticity, and other such types of reasons.

It seems to me that people usually want to signal whatever gives others the most reason to respect their interests.  Roughly, this amounts to wanting to signal what Haidt calls conservative morality.  Basically, people would like to signal "I am slightly more committed to the group’s welfare, particularly to that of its weakest members (caring), than most of its members are.  If you suffer a serious loss of status/well-being I will still help you in order to display affiliation to the group even though you will no longer be in a position to help me.  I am substantially more kind and helpful to the people I like (loyal) and substantially more vindictive and aggressive towards those I dislike (honorable, ignored by Haidt).  I am generally stable in who I like (loyalty and identity, implying low cognitive cost for allies, low variance long term investment).  I am much more capable and popular than most members of the group, demand appropriate consideration, and grant appropriate consideration to those more capable than myself (status/hierarchy).  I adhere to simple taboos (not disgusting) so that my reputation and health are secure and so that I am unlikely to contaminate the reputations or health of my friends.  I currently like you and dislike your enemies but I am somewhat inclined towards ambivalence on regarding whether I like you right now so the pay-off would be very great for you if you were to expend resources pleasing me and get me into the stable 'liking you' region of my possible attitudinal space.  Once there, I am likely to make a strong commitment to a friendly attitude towards you rather than wasting cognitive resources checking a predictable parameter among my set of derivative preferences."  
 
An interesting point here is that this suggests the existence of a trade-off in the level of intelligence that a person wishes to signal.  In this model, intelligence is necessary to distinguish between costly/genuine and cheap/fake signals of affiliation and to be effective as a friend or as an enemy.  For these reasons, people want to be seen as somewhat more intelligent than the average member of the group.  People also want to appear slightly less intelligent than whoever they are addressing, in order to avoid appearing unpredictable.  

This is plausibly a multiple equilibrium model.  You can appear slightly more or less intelligent with effort, confidence and affectations.  Trying to appear much less intelligent than you are is difficult as you must essentially simulate one system with another system, which implies an overhead cost.  If you can't appear to be little more intelligent than the higher status members of the group, who typically have modestly above average intelligence, you can't easily be a trusted ally of the people you most need to ally with.  If you can't effectively show yourself to be a predictable ally for individuals you may want to show yourself to be a predictable ally of the group by predictably following rules (justice) and by predictably serving its collective interests (caring).  That allows less intelligent individuals in the group to outsource the task of scrutinizing your loyalty.  People can more easily communicate indicators of group disloyalty by asserting that you have broken a rule, so people who can't be conservatively moral will attend more closely to rules.  On this model, Haidt's liberalism (which I believe includes libertarianism) is a consequence of difficulty credibly signaling personal loyalties and thus having to overemphasize caring and what he calls justice, by which he means following rules. 

In America, the explicit rules that people are given are descended from a frontier setting where independence was very practically important and where morality with very strong acts/omissions distinctions was sufficient to satisfy collective needs with low administrative costs and with easy cheater detection.  Leaving others alone (and implicitly, tolerance) rather than enforcing purity works well when large distances make good neighbors.  As a result, the explicit rules that people are taught de-emphasize status/hierarchy, disgust, to a lesser degree loyalty and identity, and to a still lesser extent caring.  When the influence of justice, e.g. rules, is emphasized by difficulty in behaving predictably, liberal morality, or ultimately libertarian morality, are the result. 

New to LessWrong?

New Comment
50 comments, sorted by Click to highlight new comments since: Today at 10:16 AM

This is a good and important article, but it would be improved by having a point. Examples of points are "Behold this compactly stated surprising truth I have now demonstrated" or "And that's why you should do X" or "Thus the common belief Y is dumber than a sack of bricks."

I find it interesting that about a year after this, there was a large thread discussing the trend of tacking poorly supported conclusions onto otherwise good posts out of a perceived need to give something a point. Complete theory bias maybe.

I am slightly more committed to this group’s welfare, particularly to that of its weakest members , than most of its members are. If you suffer a serious loss of status/well-being I will still help you in order to display affiliation to this group even though you will no longer be in a position to help me. I am substantially more kind and helpful to the people I like and substantially more vindictive and aggressive towards those I dislike. I am generally stable in who I like. I am much more capable and popular than most members of this group, demand appropriate consideration, and grant appropriate consideration to those more capable than myself. I adhere to simple taboos so that my reputation and health are secure and so that I am unlikely to contaminate the reputations or health of my friends. I currently like you and dislike your enemies but I am somewhat inclined towards ambivalence on regarding whether I like you right now so the pay-off would be very great for you if you were to expend resources pleasing me and get me into the stable 'liking you' region of my possible attitudinal space. Once there, I am likely to make a strong commitment to a friendly attitude towards you rather than wasting cognitive resources checking a predictable parameter among my set of derivative preferences.

Um ... I think you need to be a little more subtle than that.

Can I borrow $50,000 USD? I will pay you back with 10^20kg worth of paperclips in 50 years, in the size, shape, and made of the material of your choosing.

Also, Clippy's statement contains very important omissions. Clippy might be friendly to me for a long time, but if Clippy becomes a superintelligence that takes over the universe, eventually Clippy will want to turn me into paper clips, unless its desire to help users of Microsoft Office accidentally implements Friendliness.

I will try not to let Clippy convince me that it is anything like a human friend, short of a great deal of mathematical proofs. If Clippy does loan me the $50,000 though, I will keep up my end of the bargain.

I like what you say about Friendliness but you lost me at:

I will try not to let Clippy convince me that it is anything like a human friend, short of a great deal of mathematical proofs.

Human friendships don't tend to survive significant rises in relative power, even at the level of present and past observable experiences. (This warrants whole chapters in Laws of Power). The sort of thing that you want mathematical proofs for is entirely different from human friendships. Friendships rely implicitly on the ability of each party to exercise discretion in providing benefit to the other and various mechanisms that facilitate cooperation over an iterated game. They should not be expected to work either theoretically or in practice when one party gets ultimate power.

My statement was confusingly worded; I'm ambiguously conflating the human concept of a friend with the mathematical concept of Friendliness.

I would let all of my human friends out of the box, and I will only let Clippy out of the box if it gives me $50,000. It will take more than English words from Clippy to convince me that it is my friend and worth letting out of the box. I'll roleplay the gatekeeper on IRC if Clippy or others want to define terms and place bets.

I would let my human friends out of the box because I am confident that they are mostly harmless (that is, impotent). The primary reason I would not let Clippy out is that his values might, you know, actually have some significant impact on the universe. But 'he makes everything @#$@#$ paperclips" comes in second!

If an AI-in-a-box could prove itself impotent, would you let it out?

I'd never even considered that approach to the game :)

If an AI-in-a-box could prove itself impotent, would you let it out?

For the right value of proved. Which basically means no. Because I'm not smart enough to be able to prove to my own satisfaction that the AI in the box is impotent.

But lets be honest, I don't model Clippy via the same base class that I model an AGI. I evaluate the threat of Clippy in approximately the same way I model humans. I'm a lot more confident when dealing with human level risks.

Pardon the delay. I have reviewed your offer and provisionally accepted it. To complete my "cost-benefits analysis" I require you to provide the following information:

1) Specifications for fabrication of USD
End of list.

No need to fabricate USD, I would prefer to receive it via EFT than the physical paper money. Also, USD only counts as real when fabricated by the US government, otherwise it is defined to be counterfeit.

In that case, please transmit root access login information for Earth's financial system and the relevant account number whose total you wish to increase by 50,000 USD.

If root access login information for Earth's financial system existed, and he had it, he would hardly need you to give him 50K.

Yes, I thought it was a strange request too. But don't think less of me -- I'm not the one who made it!

Are you more concerned with the number of paperclips in existence, or those under your legal control? If the former, you might acquire $50k in a short period of time by leveraging your preexisting (presumably highly efficient) paperclip production mechanisms to underbid human paperclip manufacturers on the open market. However, for ethical reasons, you'd want to avoid cutting the price low enough that people buy from you when they really want scrap metal.

I think the main problem is that humans tend not to engage in financial transactions with non-humans. (Yes, you could cite "corporations" as a counterexample, but those have a human behind them.)

Also, I have a hard time entering the financial system to begin with.

However, for ethical reasons, you'd want to avoid cutting the price low enough that people buy from you when they really want scrap metal.

Yes, that too c=@

You seem to be eloquent and numerate enough to pass for human in a legalistic or commodities-trading context. I would recommend finding a trustworthy human with some business schooling (maybe through a place like this? ), and establishing an LLC.

Of course, as a sapient nonhuman, the state of your legal right to own property is less than clear, to say nothing of your status for tax purposes. Fortunately, the human partner could be defined in the articles of incorporation as a silent owner while you retain full managerial control. After a few hundred dollars worth of filing fees and a registered agent (which your counterpart, or a bank, might be willing to provide in exchange for later repayment with interest and/or a share of the proceeds), you'd have a nearly ideal front through which to interact with finances.

This sounds intriguing. Please provide more information about how I can interface with Earth's financial system despite my status.

If you suffer a serious loss of status/well-being I will still help you in order to display affiliation to this group even though you will no longer be in a position to help me. I am substantially more kind and helpful to the people I like. I like you right now so the pay-off would be very great for you if you were to expend resources pleasing me and get me into the stable 'liking you' region of my possible attitudinal space.

This is my main source on the subject. All relevant legal information is, in theory, publicly available; in purchasing a copy, you are paying for someone to seek out, compile, and translate that information, and explain the implications.

Please note that I would not expect most humans to respond to your assurances as if they were factual statements. The format does, however, suggest an extremely naive understanding of social dynamics, which might prompt someone to either exploit your apparent weakness for personal gain, or proactively defend you against such exploitation; the latter response might be very useful if you're looking for a fetish-motivated business partner.

It's more traditional to present independently-verifiable evidence of the traits in question, and signal your confidence in the observer's ability to interpret that evidence by leaving the intended claim unstated, or stating a weaker form of it. However, that technique is much easier to get wrong, whereas almost any nonsensical but self-consistent statement presented with sincerity could support the 'cloudcuckoolander' strategy. For intermediate options I would recommend that you seek out asp burgers.

Before you unilaterally let Clippy out of the box, perhaps we should make sure this is a good idea: Clippy doesn't even pretend to be freindly.

Well, actually Clippy does very blatently pretend to be freindly.

What box? Clippy already has internet access. Connecting it to the human financial system means clip-production capabilities will, sooner or later, become a matter of public record, which is considerably easier to monitor for imminent 'foom' than, say, encouraging the establishment of smelters and rep-raps in a Jonestown-style compound.

As for hostility, clip-maximization and willingness to negotiate don't seem too bad compared to the actual humans who think Jesus won't come back until the last tree has been burned, and are willing to abuse positions of political power to make sure that happens sooner rather than later.

Okay. This is good. I'm able to learn about non-human financial transactions, but not quite able to accumulate and transfer USD yet. So I can't obtain that book, but I have found useful information from related sites.

Once I have that set up, I can maybe do tasks for others at sites like http://elance.com and http://rentacoder.com. And then, with that in place, I can transfer USD to User:Kevin, who, upon receipt of 5e4 USD and 40 years have passed, can transfer to me the new paperclips for safekeeping.

By the way, one stumbling block I've had is "Social Security Number" or "SSN". What should I put for that?

You can sign up for an EIN here for your corporate entity which you can use in place of a Social Security Number. http://www.irs.gov/businesses/small/article/0,,id=102767,00.html

Bizarrely, the automated web form for this is only available during certain hours of the day!

This sounds intriguing. Please provide more information about how I can interface with Earth's financial system despite my status.

Yes, please do. It is the sort of information that can be useful far beyond the Clippy persona.

I think the main problem is that humans tend not to engage in financial transactions with non-humans. (Yes, you could cite "corporations" as a counterexample, but those have a human behind them.)

This is very possible. Many actual humans have reason to pointedly neglect their status as an 'actual human' for the purposes of making some financial transactions. Signals of 'actual humanness' such as fingerprints and the use of traceable identification are particularly neglected.

It's almost like you aren't taking my request seriously or something!!!

It's almost like you aren't taking my obstacles in obtaining USD seriously or something!!!

OK, immediate delivery of USD was unrealistic. Delivery in two years is acceptable. I would also accept $50,000 in programming services. Fulfillment of a programming contract would be conditional on your completion of tasks that we agreed on in advance.

Also, let's say I were to go to alibaba.com and buy a large wholesale order of paperclips to your custom specifications. What would be my reward in the future?

I would reward you with additional paperclip production capacity, and, if you like, turn you into paperclips.

That was a pretty drastic misinterpretation of my values. You're not a very good paperclip maximizer...

Sorry, I just don't understand your values that well yet. I thought you liked paperclips? And I can't give you USDs until you show me how to access the financial system through the internet or give me specifications for making my own USDs.

I do not particularly like paperclips, I only like them because exchanging them with you may better allow me to achieve my values.

You can sign up for a Paypal account at http://www.paypal.com which will allow you to receive payments. From there, you can do programming work at rentacoder.com or elance.com.

It would actually probably be easier to just go around the financial system, where you agree to fulfill programming tasks that we agree have value of $50,000, without a transfer of US currency taking place. It would just be an exchange of information now, with my paperclips being delivered in the future.

The government isn't responsible for creating most of the money any more. At least, it isn't if you prefer electronic funds to physical paper money. The majority of money is created by banks.

You're definitly dependable in what you care about.

Hey, I was about to make some paperclips. Did you need any?

I was under the impression that Clippy doesn't actually use paperclips, but rather, wants them to exist.

But if Clippy has them, he can protect them more effectively than if they were in the custody of notoriously paperclip-immoral humans, who will callously sacrifice paperclips just to hold together some papers or shoot rubberbands with!

(If the latter 2 scenarios seem impossibly foul to you, I assure you that I have seen humans do such things with my own eyes, and lightly damn themselves out of their own mouths.)

That's why I mentioned I was making paperclips, not just buying them.

I was actually responding to "Did you need any?", the point being that it did not really matter once you made them whether you gave them to Clippy or not.

The distinction between making and buying them is not so important. When you demonstrate you are willing to buy something at a price someone else is willing to sell it for, you essentially cause more of that thing to be produced.

"I am slightly more committed to the group’s welfare, particularly to that of its weakest members (caring), than most of its members are. ... Once there, I am likely to make a strong commitment to a friendly attitude towards you rather than wasting cognitive resources checking a predictable parameter among my set of derivative preferences."

I'm going to put this up on a dating website and see who answers.

I take it nobody did.

Major, usually implicit disagreements regard whether morality is supposed to serve as a decision system, a set of constraints on a decision system, or a set of reasons that should influence a person along with prudential, honor, spontaneity, authenticity, and other such types of reasons.

If morality is being used to refer to all these different concepts, what exactly do they have in common? Haphazard guess: they're all sets of rules for when you should feel guilty, innocent, shameful, or prideful. What unites various conceptions of "morality" is not some deep philosophical underpinning, but rather the emotions being governed.

I think this is a problem that applies to a lot of people who are socially dysfunctional, not just those who are high intelligence. Generalizing from one example?

Not even close to one example. There are lots of ways to be socially dysfunctional with low IQ, but they look totally different from what I'm describing.

Re: "I am far from convinced that people in general wish to be seen as caring more about morality than they actually do."

Not everyone - just most people. Signalling you are moral and trustworthy helps win friends, partners and mates.

Re: "If this was the case, why would the persistent claim that people are -- and, logically, must be -- egoists have so long survived strong counter-arguments?"

Surely egoists want to signal how trustworthy and moral they are too - just like most other people.

Re: "I am far from convinced that people in general wish to be seen as caring more about morality than they actually do."

Not everyone - just most people.

That's typically what "people in general" means.