Rationality Quotes August 2014

  • Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
  • Do not quote yourself.
  • Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here.
  • No more than 5 quotes per person per monthly thread, please.
  • Provide sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Do not quote with only a name.
233 comments, sorted by
magical algorithm
Highlighting new comments since Today at 8:06 PM
Select new highlight date

Hollywood is filled with feel-good messages about how robotic logic is no match for fuzzy, warm, human irrationality, and how the power of love will overcome pesky obstacles such as a malevolent superintelligent computer. Unfortunately there isn’t a great deal of cause to think this is the case, any more than there is that noble gorillas can defeat evil human poachers with the power of chest-beating and the ability to use rudimentary tools.

From the British Newspaper 'The Telegraph', and their article on Nick Bostrom's awesome new book 'Superintelligence'.

I just thought it was a great analogy. Nice to see AI as an X-Risk in the mainstream media too.

Probably true. It's not like Hollywood is an accurate source of information about anything. (Climate change, asteroid impacts, the legal system, the military, romance, sex, business, anything.) But I fail to see how this is a rationality quote. I'm sure there are many more quotes of the form "Group X is wrong about Topic Y."

I would prefer to limit quotes to those that that teach us how to tell whether or how much Group X is right or wrong about Topic Y, and skip quotes that merely turn on the applause lights on a particular topic.

The quote isn't just about Hollywood being wrong, it's about a specific way that it's wrong.

A good rule of thumb to ask yourself in all situations is, “If not now, then when?” Many people delay important habits, work and goals for some hypothetical future. But the future quickly becomes the present and nothing will have changed.

Scott Young

I just bought a book on procrastination. I am going to start reading it tomorrow.

That's why I'm skeptical of people who look at some catastrophic failure of a complex system and say, "Wow, the odds of this happening are astronomical. Five different safety systems had to fail simultaneously!" What they don't realize is that one or two of those systems are failing all the time, and it's up to the other three systems to prevent the failure from turning into a disaster.

-- Raymond Chen

In other words, some of the slices in one's Swiss cheese model are actually missing entirely.

Correlary: if you're running a system for which five simultaneous failures is a disaster, monitor each safety system seperately and treat any three simultaneous failures as if it were a disaster.

Also known as Fundamental Failure Mode. From Systemantics:

System failure

The Fundamental Failure-Mode Theorem (F.F.T.): Complex systems usually operate in failure mode.

A complex system can fail in an infinite number of ways. (If anything can go wrong, it will.) (See Murphy's law.)

The mode of failure of a complex system cannot ordinarily be predicted from its structure.

The crucial variables are discovered by accident.

The larger the system, the greater the probability of unexpected failure.

"Success" or "Function" in any system may be failure in the larger or smaller systems to which the system is connected.

The Fail-Safe Theorem: When a Fail-Safe system fails, it fails by failing to fail safe.

"I want information. I want to understand you. To understand what exactly I'm fighting. You can help me."
"I obviously won't."
"I will kill you if you don't help me. I'm not bluffing, Broadwings. I will kill you and you will die alone and unseen, and frankly you are far too intelligent to simply believe that the stories of ancestral halls are true. You will die and that will probably be it, and nobody will ever know if you talked or not—not that conversing with an enemy in a war you don't support is dishonorable in the first place."
"You'll let me leave if I stonewall, because you don't want to set a precedent of murdering surrendered officers."
"We'll see. Would you like another cup?"
"No."
Derpy smiled deviously. "You know, in that last battle? We didn't fly our cannon up there to the cliffs. Nope. We had Earth ponies drag them. Earth ponies are capable of astounding physical feats, you know. We're probably going to be using more mobility in our artillery deployment going forward, now that they've demonstrated how effective the concept is."
"...why did you tell me that? What would drive you to tell me that?"
"I'll ask again before I continue. Would you like to assist me, Broadwings?"
"I am a gryphon. Telling me your plans will do nothing to change that. I will not barter secrets."
She leaned back, gesturing with a hoof as she talked. "My biggest strengths are that I understand the way crowds think and that I am good at thinking up unexpected ways to solve simple problems. My army's biggest weakness is that my soldiers are inexperienced, and that unexpected developments have an inordinate effect on their morale. Also, my infantry will never be able to stand against a sustained lion charge, so I have to keep finding ways to nullify that disadvantage, and frankly I won't be able to forever."
"I don't understand. What are you doing, Mare? Why are you--"
"--my personal biggest weaknesses," she continued, her smile now malicious, "are my struggles with morality, identity, and my desire to be loved. There's also my relationship with the stallion Macintosh Apple, who is usually called Big Macintosh, with whom I spend upwards of ten hours a day, and on whom I am completely emotionally dependent. If he were to be killed, I'd probably fall apart emotionally. I also have a daughter named Dinky—not by him, mind you—who is in the Southmarch, and who I am very, very guilty about abandoning. If anything were to happen to her I might kill myself. Do you understand yet, Broadwings?"
"Mare, this is insanity. I cannot--"
"--All right then, we'll continue. I also have in this camp Sweetie Belle, Apple Bloom, and Scootaloo, three little fillies, though they're growing quite quickly now. Sweetie Belle is the writer of many propaganda songs, Apple Bloom is Big Mac's sister, who he protects like a daughter, and I believe Scootaloo has no special importance but the other two would defend her to the death. They would be quite easy to kill as well. Do you understand yet?"
"Mare! Are you mad?! Do you have any idea how dangerous it is to tell me these things? Aren't you afraid I would tell--"
"--Good," she nodded. "You're beginning to understand. Let's see. My logistics framework right now is nonexistent. I'm entirely reliant on local villages bringing me food and materiel, and on capturing food and materiel meant for your armies. My army is nowhere near as mobile as it appears, since it can only operate in areas where I have established relationships with each particular village. A bit of simple recon work would let you figure out where I can and cannot go. Do you understand yet?"
Broadwings' eyes opened and his pupils shrank with dawning recognition. "...If I came back to my army, I would use this to defeat you. If I told any other gryphon, they would use it to defeat you. You...you have..."
"Yes. I have sealed your fate; you will not see your home. I can't let you leave now. I absolutely can't. I can now either kill you or keep you prisoner until this war is over—and I don't keep useless prisoners. It's now out of my hooves. One or the other. You pick."

~emkajii, Equestria: Total War

This sounds like something from Schelling's strategy of conflict, although I haven't read it

Yes, that's exactly what I was thinking. General Broadwings thinks General Derpy is bluffing, so Derpy credibly precommits herself to not releasing him by telling him information that would surely doom her army if she did. She gives up the choice of freeing Broadwings, and comes out ahead for it.

It's kind of reminiscent of this, from pages 43-44 of the 1980 edition:

It is not always easy to make a convincing, self-binding, promise. Both the kidnapper who would like to release his prisoner, and the prisoner, may search desperately for a way to commit the latter against informing on his captor, without finding one. If the victim has committed an act whose disclosure could lead to blackmail, he may confess it; if not, he might commit one in the presence of his captor, to create the bond that will ensure his silence. But these extreme possibilities illustrate how difficult, as well as important, it may be to assume a promise.

Compare also Daniel Ellsberg's Kidnap game.

A man is walking on the moon with his eyes turned up toward space And the bright blue world that watches him reflected on his face. The whole world sees the hero there and the module crew also. But few can see the guiding team that guards him from below.

Here's a health to the man who walked the moon, and the module crew above, And the team that watches from the sky with worry, joy, and love. To all who blazed the sky-trail come raise your glasses 'round; And a health to the unknown heroes, too, who never left the ground.

Here's a health to the ship's designers, and the welders of her seams, And all who man the radar-scan to watch our dawning dreams. For all the unknown heroes, sing out to every shore: "What makes one step a giant leap is all the steps before".

Leslie Fish, musically praising the Hufflepuff virtues.

Surgeons finally did upgrade their antiseptic standards at the end of the nineteenth century. But, as is often the case with new ideas, the effort required deeper changes than anyone had anticipated. In their blood-slick, viscera-encrusted black coats, surgeons had seen themselves as warriors doing hemorrhagic battle with little more than their bare hands. A few pioneering Germans, however, seized on the idea of the surgeon as scientist. They traded in their black coats for pristine laboratory whites, refashioned their operating rooms to achieve the exacting sterility of a bacteriological lab, and embraced anatomic precision over speed.

The key message to teach surgeons, it turned out, was not how to stop germs but how to think like a laboratory scientist. Young physicians from America and elsewhere who went to Germany to study with its surgical luminaries became fervent converts to their thinking and their standards. They returned as apostles not only for the use of antiseptic practice (to kill germs) but also for the much more exacting demands of aseptic practice (to prevent germs), such as wearing sterile gloves, gowns, hats, and masks. Proselytizing through their own students and colleagues, they finally spread the ideas worldwide.

-Atul Gawande

It was a gamble: would people really take time out of their busy lives to answer other people’s questions, for nothing more than fake internet points and bragging rights?

It turns out that people will do anything for fake internet points.

Just kidding. At best, the points, and the gamification, and the focused structure of the site did little more than encourage people to keep doing what they were already doing. People came because they wanted to help other people, because they needed to learn something new, or because they wanted to show off the clever way they’d solved a problem.

...

An incredible number of people jumped at the chance to help a stranger

-- Jay Hanlon, Five year retrospective on StackOverflow

On the other hand, a Slashdot comment that's stuck in my mind (and on my hard disks) since I read it years ago:

In one respect the computer industry is exactly like the construction industry: nobody has two minutes to tell you how to do something...but they all have forty-five minutes to tell you why you did it wrong.

When I started working at a tech company, as a lowly new-guy know-nothing, I found that any question starting with "How do I..." or "What's the best way to..." would be ignored; so I had to adopt another strategy. Say I wanted to do X. Research showed me there were (say) about six or seven ways to do X. Which is the best in my situation? I don't know. So I pick an approach at random, though I don't actually use it. Then I wander down to the coffee machine and casually remark, "So, I needed to do X, and I used approach Y." I would then, inevitably, get a half-hour discussion of why that was stupid, and what I should have done was use approach Z, because of this, this, and this. Then I would go off and use approach Z.

In ten years in the tech industry, that strategy has never failed once. I think the key difference is the subtext. In the first strategy, the subtext is, "Hey, can you spend your valuable time helping me do something trivial?" while in the second strategy, the subtext is, "Hey, here's a chance to show off how smart you are." People being what they are, the first subtext will usually fail -- but the second will always succeed.

— fumblebruschi

In addition to the specific advice, this is an excellent example of rationality because it's about getting the best from people as they are rather than being resentful because they aren't behaving as they would if they were ideally rational.

I can't be sure, because I first read that comment so long ago, but I think I took it as an inspiration to be better than the co-workers at the coffee machine. It's repellent to imagine myself as a person who'd spend 45 minutes on a Yer Doin It Rong lecture but wouldn't spend 2 minutes to explain how to do something properly in the first place.

This is known as Cunningham's Law. Another example.The explanation (non-competitive vs. competitive mindsets, the latter of which is more motivating to act) seems quite convincing. In addition, could there also be an analogy to loss aversion (a tendency to prefer avoiding losses to acquiring gains)? Would people feel more urgency to correct what they see as wrong (and thus challenging what they see as correct) rather than explain what is right ("less wrong" vs. "more right", if we are not trying to avoid puns)?

A reply because an upvote doesn't begin to cover it. I might start using this!

Well, did they test popularity of sites without fake internet points vs popularity of sites with, controlling for relevant factors? I skimmed through the post, and there wasn't much actual data on what people do and why, just assertions.

I thought the point of the points was to weed out the people whose "help" you don't want.

That would account for reputation, not badges. (No one says "Hey, I got two answer from people with the same rep, but one has twice as many badges, so I'll go with that one.")

On the actual question, I've seen meta-posts on Stack Exchange complaining that they qualified for a badge and didn't get it, so the stuff does matter somewhat.

Convincing people to offer others programming help on the internet isn't a special accomplishment of SO. From usenet to modern mailing lists to forums to IRC, there are tons and tons of thriving venues for it. The gamification might have helped SO's popularity some, but taking time out of their busy lives to answer others' questions was alive and well.

SO is a dangerous trash heap. It doesn't encourage helping people make good programs; it answers extremely literal questions. Speed of post is important. Style of post is important. Blatantly wrong answers are upvoted by people who don't know what they're looking at when they are early, indicating that vote count isn't telling ever. Doing anything but answering a question completely literally is treated with extreme hostility. These sorts of things have gotten worse with time.

The community relations are bizarre. Active members of the community buy into cheap salesman lines by the owners that are meant to favor the owners. The idea that the community can direct itself is thrown around as if it wasn't blatantly untrue.

Yes, an incredible people jump at the chance to help strangers. SO didn't invent that, they're just one of the more popular current hosts to these people. It's distasteful to act like it started by wondering if such people exist.

It doesn't encourage helping people make good programs; it answers extremely literal questions.

So? That's fine. "Helping people make good programs" is awfully fuzzy and is likely to start by major holy wars breaking out. SO is useful, at least for me, because it offers fast concise answers to very specific and literal questions I have on a regular basis.

I can't say anything about the internal politics of SO since I don't play there.

The amount of energy necessary to refute bullshit is an order of magnitude bigger than to produce it.

-- Alberto Brandolini (via David Brin)

Refuting frequently appearing bullshit could be made more efficient by having a web page with standard explanations which could be linked from the debate. Posting a link (perhaps with a short summary, which could also be provided on the top of that web page) does not require too much energy.

Which would create another problem, of protecting that web page from bullshit created by reversing stupidity, undiscriminating skepticism, or simply affective death spirals about that web page. (Yes, I'm thinking about RationalWiki.) Maybe we could have multiple anti-bullshit websites, which would sometimes explain using their own words, and sometimes merely by linking to another website's explanation they agree with.

http://www.talkorigins.org/indexcc/ is considered a good one on the single issue of creationism vs. evolution.

Yes, it is, and The Counter-Creationism Handbook sits next to Darwin, Dawkins, and Diamond on my shelf. It would be a Good Thing if folks in other bullshit-fighting arenas had the level of scholarship exhibited by Mark Isaak and his collaborators.

(Hell, every time I see a "bingo card" ridiculing an Other Side's arguments, I wish its creators had the time and scholarly dedication of the talk.origins folk.)

undiscriminating skepticism

I think that's a bad description. The kind of people on RationalWiki are very discriminating. When something is said by an Authority they trust they aren't skeptic and when something is said by someone they don't trust they are very "skeptic".

Maybe we could have multiple anti-bullshit websites

I don't think that framing yourself as "anti-bullshit" is helpful. It's makes more sense to frame yourself as being pro-evidence. We already do have multiple websites that do explain issues.

I personally like Skeptics Stackexchange. If I come about a new claim I often simply go and open a question over there.

When it comes to an issue such as vaccination I think Vox has a decent primer: http://www.vox.com/cards/vaccines/what-is-vaccine

How does this differ from religious groups refusing to answer questions that dispute things said by their religion, and instead referring you to scripture passages or Christian apologetics?

Of course it's different in that you will link to refutations that are good arguments, and the religious person will link to apologetics that are bad arguments, but aside from that, how is it different? After all, you can't very well say that certain tactics are acceptable or unacceptable based on whether the associated arguments are good or bad.

Depends on the audience and topic. Also, sometimes the goal is not to convince your opponent, but to convince the bystanders.

Imagine that you are on a web forum where someone comes and writes a long comment about "Isn't it horrible that vaccination causes autism, and yet the government wants us to vaccinate our children? I would do anything to protect my child from autism!" and some information probably copied from some other webpage. It's not just you and them; there are also other readers who don't have a clue and may be frightened by the message. (And they will not use google, because... well, humans are stupid.)

If nobody opposes the message, it seems like their is a clear consensus among the people who care about the topic. If you opposed them, you are wasting your time. -- But if you post a link to a good explanation, then the people frightened by the message can read the explanation and hear a dissenting voice, while you wouldn't have to spend a lot of time... assumming there is a good anti-bullshit page where you just enter "vaccination, autism" in the search box, and it shows you a well-written page about the topic. Where well-written means a short layman-accessible summary at the top, and then detailed arguments and references below.

But by that same reasoning, a fundamentalist Christian could come here, see that someone has written a long comment about, say, evolution, and reply with a link to a prewritten web page listing 100 arguments against evolution. He reasons that if he posts a good explanation, people who are frightened by the idea of fundamentalists being a menace can read the explanation and hear a "dissenting voice",,,.

As far as he is concerned, he has followed your recommendations exactly. Is there something you could say which explains why his behavior is unacceptable, but the behavior you describe is acceptable, that does not involve "our anti-anti-vaccination page is well-written and your anti-evolution page is not"?

(Alternatively, would you find his behavior acceptable? This seems odd.)

A fundamentalist Christian who would post here a link to a page listing arguments against evolution would be more effective than the one who would try to debate, because they would achieve the same (in this situation: zero) effect while spending much less resources. The people who would try to debate them, each of them would waste more of their time by reading the linked page and composing the reply. So, I believe this is a good strategy.

Specifically on LW we have an (unwritten?) norm that if you post a link, you should also provide a summary using your own words. Which probably was designed to counter this strategy. But there are website which don't have this norm, e.g. Facebook.

Specifically on LW we have an (unwritten?) norm that if you post a link, you should also provide a summary using your own words. Which probably was designed to counter this strategy.

It is not specific to LW, but a custom of good practice that personally, I have followed ever since there has been such a thing as a link (and before then, when the equivalent was posting to an email list a cut-and-paste of someone else's words without any words from the person posting). I also practice the custom of ignoring links that come to me without context.

I recommend both parts of this practice to everyone.

How does this differ from religious groups refusing to answer questions that dispute things said by their religion, and instead referring you to scripture passages or Christian apologetics?

How does answering questions at length in one's own words differ from religious groups answering questions at length in their own words?

It doesn't differ. But it doesn't have to, since we consider it acceptable behavior for religious people to come here and answer questions in their own words.

We generally don't consider it acceptable behavior for religious people to come here and respond to posts by giving links to apologetic sites. It should not, then, be acceptable behavior for us except maybe in a few specialized cases (such as where the dispute is purely over facts, like for vaccines).

But you still have to find the proper entry. This just shifts the burden around and in total refutation is probably still much more expensive than creation (esp. as the same BS can be copied but the refutation can't..

Refuting frequently appearing bullshit is more than a matter of making the facts available. After all, anti-vaccination folks appear with enough frequency to be a curious news item (which I admit is a horrendous metric, but let's pretend it means something), and I'm sure that a quick Google search would yield enough facts to disabuse them of their notions. The trick is building up enough credibility and charisma - if such a property could be applied to an argument - to make such a site not just correct, but convincing. That's where the order of magnitude comes in.

a quick Google search would yield enough facts to disabuse them of their notions

Most people are not strategic enough to use google. Or if they get many contradicting results from google search, they are not smart enough to decide.

Also, there is this bias that if an information was brought to you by a person you know, it has much stronger impact. (For most people, good relationships matter more than truth. If someone brings you an information, disbelieving the information is a potential conflict with given person.) You only get an equal opposing force if another person you know opposes the original information. For example, by saying "it's bullshit" and posting a link to refutation.

But if that were the case, then moral philosophers - who reason about ethical principles all day long - should be more virtuous than other people. Are they? The philosopher Eric Schwitzgebel tried to find out. He used surveys and more surreptitious methods to measure how often moral philosophers give to charity, vote, call their mothers, donate blood, donate organs, clean up after themselves at philosophy conferences, and respond to emails purportedly from students. And in none of these ways are moral philosophers better than other philosophers or professors in other fields.

Schwitzgebel even scrounged up the missing-book lists from dozens of libraries and found that academic books on ethics, which are presumably mostly borrowed by ethicists, are more likely to be stolen or just never returned than books in other areas of philosophy. In other words, expertise in moral reasoning does not seem to improve moral behavior, and it might even make it worse (perhaps by making the rider more skilled at post hoc justification). Schwitzgebel still has yet to find a single measure on which moral philosophers behave better than other philosophers.

  • Jonathon Haidt, discussing the idea that ethical reasoning causes good behaviour, in his book 'The Righteous Mind'.

I found the book-stealing thing quite funny, although I imagine that some of the results described could be explained by popularity; if more people get into / like ethics, then there are more people who might steal library books, more antisocial people who don't respond to emails, etc. This hasn't been demonstrated to my knowledge though, and I'm otherwise inclined to believe that people who spend their days thinking about ethics in the abstract, are simply better at coming up with rationales for their instinctive feelings. Joshua Greene says rights are an example of this, where we need a dictum against whatever our emotions are telling us is despicable, even though we can't find any utilitarian justification for it.

There's probably a selection effect at work. Would a highly moral person with a capable and flexible mind become a full-time moral philosopher? Take their sustenance from society's philanthropy budget?

Or would they take the talmudists' advice and learn a trade so they can support themselves, and study moral philosophy in their free time? Or perhaps Givewell's advice and learn the most lucrative art they can and give most of it to charity? Or study whichever field allows them to make the biggest difference in peoples' lives? (Probably medicine, engineering or diplomacy.)

Granted, such a person might think they could make such a large contribution to the field of moral philosophy that it would be comparable in impact to other research fields. This seems unlikely.

The same reasoning would keep highly moral people out of other sorts of philosophy, but people who don't have an interest in moral philosophy per se might not notice the point. It's hard to avoid if you specifically study it.

Granted, such a person might think they could make such a large contribution to the field of moral philosophy that it would be comparable in impact to other research fields. This seems unlikely.

Unlikely that they would make such contribution? Yes. Unlikely that they think they would make such contribution? Maybe no.

But I guess they probably don't even think this way, i.e. don't try to maximize their impact. More likely it is something like: "My contribution to society exceeds my salary, so I am a net benefit to the society". Which is actually possible. Yeah, some people, especially the effective altruists, would consider such thinking an evidence against their competence as a moral philosopher.

Or would they take the talmudists' advice and learn a trade so they can support themselves, and study moral philosophy in their free time?

If someone's studying moral philosophy in their free time, then wouldn't they be taking academic books on ethics out of the library?

This could happen, but I think it's mostly dwarfed by the far larger selection effect that people who are not financially privileged mostly don't attempt to become humanities academics these days -- and for good reason.

Are you saying that financially privileged people tend to be less moral?

While that case has been made in a few isolated studies, I was more generally referring to the fact that people who don't come from money will usually choose careers that make them money, and humanities academia doesn't.

Wasn't sure about that, so I tracked down some research (Goyette & Mullen 2006). Turns out you're right: conditioned on getting into college in the first place, higher socioeconomic status (as proxied by parents' educational achievement) is correlated with going into arts and sciences over vocational fields (engineering, education, business). The paper also finds a nonsignificant trend toward choosing arts and humanities over math and science, within the arts and science category.

(Within the vocational majors, though, engineering is the highest-SES category. Business and education are both significantly lower. I don't know which of those would be most lucrative on average but I suspect it'd be engineering.)

(Within the vocational majors, though, engineering is the highest-SES category. Business and education are both significantly lower. I don't know which of those would be most lucrative on average but I suspect it'd be engineering.)

I think there are several trade-offs there: engineering looks like the highest expected value to us, because we (on LessWrong, mostly) had pre-university educations focused on math, science, and technology. People from lower SES... did not, so fewer of them will survive the weed-out courses taught in "we damn well hope you learned this in AP class" style. And then there's the acclimation to discipline and acclimation to obsessive work-habits (necessary for engineering school) that come from professional parentage... and so on. And then of course, many low-SES people probably want to go into teaching as a helping profession, but that's not a very quantitative explanation and I'm probably just making it up.

On the other hand, engineering colleges tend to have abnormally large quantities of international students and immigrants blatantly focused on careerism. So yeah.

How does that fact impact the morality of moral philosophers as measured?

"In 1971, John Rawls coined the term "reflective equilibrium" to denote "a state of balance or coherence among a set of beliefs arrived at by a process of deliberative mutual adjustment among general principles and particular judgments". In practical terms, reflective equilibrium is about how we identify and resolve logical inconsistencies in our prevailing moral compass. Examples such as the rejection of slavery and of innumerable "isms" (sexism, ageism, etc.) are quite clear: the arguments that worked best were those highlighting the hypocrisy of maintaining acceptance of existing attitudes in the face of already-established contrasting attitudes in matters that were indisputably analogous."

-Aubrey de Grey, The Overdue Demise Of Monogamy

This passage argues that reasoning does impact ethical behavior. Steven Pinker and Peter Singer make similar arguments, which I find convincing.

I actually put up another quote arguing for it, by Joshua Greene, making an analogy between successsful moral argument and the invention of new technology; even though a person rarely invents a whole new piece of technology, our world is defined by technological advance. Similarly, even though it is rare for a moral norm to change as a result of abstract argument, our social norms have change dramatically since times gone by.

Nonetheless, the quote works with empirical evidence, the ultimate arbiter of reality. It looks like, whilst moral argument can change our thoughts (and behaviour) on ethical issues, a lot of the time it doesn't. Like technology, the big changes transform our world, but for the most part we're just playing angry birds.

I find it quite arguable whether or not "reflective equilibrium" is a real thing that actually happens in our cognition, or a little game played by philosophy academics. Actual cognitive dissonance caused by holding mutually contradicting ideas in simultaneous salience is well-evidenced, but that's not exactly an equilibrium across all ideas we hold, merely across the ones we're holding in short-term verbal memory at the time.

This hasn't been demonstrated to my knowledge though, and I'm otherwise inclined to believe that people who spend their days thinking about ethics in the abstract, are simply better at coming up with rationales for their instinctive feelings.

I think it more likely they're better at coming up with rationales to ignore their instinctive feelings.

I think that someone can believe that their instinctive feelings are an approximation to what is ethical, then try to formalize it, then conclude that they have identified areas where the approximation is in error. So their ethics code could be highly based on their instinctive feelings without following them 100% of the time.

That seems unlikely. People's instinctive feelings are generally pretty selfish. (Small sample size, obviously. I think 2 other people where I've spoken with enough about this kind of thing to judge.)

None of your sample were people with children, then?

And there's also the question of what is "instinctive" versus whatever the opposite is. What is this distinction and how do you tell?

No, but I don't see why children should have an effect; favoring your children over strangers is no less selfish than favoring yourself over strangers, and both are strong instincts.

By instinctive I just mean system 1; the judgments made before you take time to think through what you should do.

No, but I don't see why children should have an effect; favoring your children over strangers is no less selfish than favoring yourself over strangers, and both are strong instincts.

I had intended to draw attention to the phenomenon of favouring one's children over oneself. It appears I was right about the test demographic.

And "no less selfish"? At what point would you consider the widening circle to be "less selfish"? To favour your village over others, your country over others, humanity over animals; are these are all no less selfish? Is nothing unselfish but a life of exaninition and unceasing service to everyone and everything but oneself?

By instinctive I just mean system 1; the judgments made before you take time to think through what you should do.

System 1 is susceptible to training -- that is what training is. We may be born with the neurological mechanism, but not its entire content. "Instinct" more usually means (quoting Wikipedia) "performed without being based upon prior experience". A human without prior experience is a baby.

Standard definitions of system 1 describe it as 'instinctive', but if you need a separate definition of instinctive responses, 'untrained system 1 responses' works.

At what point would you consider the widening circle to be "less selfish"? To favour your village over others, your country over others, humanity over animals; are these are all no less selfish? Is nothing unselfish but a life of exaninition and unceasing service to everyone and everything but oneself?

That depends. Any of those things can be unselfish, if you're doing it because you think it's a good thing to do independent of whether it's an outcome/action you like, and the wider the circle the more likely that's the motivation. If it's based on 'I like these people and want them to be happy, therefore I will take this action' that's still selfish.

Lest this sound like I'm saying anything that isn't done for abstract reasons is selfish, I'd contrast it with things done for reasons of compassion. The lines there can get blurry when the people you're feeling compassion for are in your ingroup, but things like the place-quarters-here-for-adorable-sad-children variety of charity are clearly trying to induce compassionate motivation (and it works).

From conversations I have had with my own parents (not as comprehensive or in-depth, but heartfelt), it seemed pretty clear that the parenting instinct is much more 'these kids are mine and I will take care of them come hell or high water' than a compassionate reflex.

Hypothesis: At least some of the people who are interested in ethics are concerned because they have a problem behaving ethically.

Come back with your shield - or on it.

Our kind might not be able to cooperate, but the Spartans certainly could. The Spartans were masters of hoplite phalanx warfare where often every individual would have been better off running away but collectively everyone was better off if none ran away than if all did. The above quote is what Plutarch says Spartan mothers would tell their sons before battle. (Because shields were heavy if you were going to run away you would drop it, and coming back on your shield meant you were dead.) Spreading memes to overcome collective action problems is civilization level rational.

Well... most of what we "know" about the Spartans was written down by their enemies, and may be inaccurate. It is not at all clear that any actual Spartan ever said the words you attribute to them; it may be Plutarch making things up to illustrate how he thought a city ought to work. Which doesn't necessarily make it bad rationality, but does mean it is fictional evidence, not historical.

We have significant amounts written by the Ancient Greek equivalent of a Sparta otaku, Thucydides. He lived there for a significant period (IIRC, he was in exile from Athens at the time) and was firsthand familiar.

An example of the orderly battle of the hellens from Xenophons Anabasis where the enemy has ten-fold numeric superiority:

Clearchus, though he could see the compact body at the centre, and had been told by Cyrus that the king lay outside the Hellenic left (for, owing to numerical superiority, the king, while holding his own centre, could well overlap Cyrus's extreme left), still hesitated to draw off his right wing from the river, for fear of being turned on both flanks; and he simply replied, assuring Cyrus that he would take care all went well.

...

At this time the barbarian army was evenly advancing, and the Hellenic division was still riveted to the spot, completing its formation as the various contingents came up.

...

And now the two battle lines were no more than three or four furlongs apart, when the Hellenes began chanting the paean, and at the same time advanced against the enemy. But with the forward movement a certain portion of the line curved onwards in advance, with wave-like sinuosity, and the portion left behind quickened to a run; and simultaneously a thrilling cry burst from all lips, like that in honour of the war-god—eleleu! eleleu! and the running became general. Some say they clashed their shields and spears, thereby causing terror to the horses (4); and before they had got within arrowshot the barbarians swerved and took to flight. And now the Hellenes gave chase with might and main, checked only by shouts to one another not to race, but to keep their ranks. The enemy's chariots, reft of their charioteers, swept onwards, some through the enemy themselves, others past the Hellenes. They, as they saw them coming, opened a gap and let them pass. One fellow, like some dumbfoundered mortal on a racecourse, was caught by the heels, but even he, they said, received no hurt, nor indeed, with the single exception of some one on the left wing who was said to have been wounded by an arrow, did any Hellene in this battle suffer a single hurt.

...according to Xenophon, at any rate. I don't see what that has to do with the alleged Spartan quote.

The hellenes mentioned in the quote were likely spartans.

Some of the commanders were Spartans, yes; and it does seem likely that the mercenaries segregated themselves at least somewhat by city of origin, so the Spartan commanders probably had Spartan troops. But the tactics described are standard Hellenic ones; there is nothing about them that is special to Sparta, as far as I can see.

I'm neither istoriean nor expert on acient warfare. My quote was intended to substantiate the claim

Our kind might not be able to cooperate, but the Spartans certainly could. The Spartans were masters of hoplite phalanx warfare where often every individual would have been better off running away but collectively everyone was better off if none ran away than if all did.

My given quote indeed doesn't distinguish between Spartans and Athens... but that isn't needed as it appears that all hellenes were able to much better cooperate than their enemies. And from my reading of Anabasis this is substantiated. And my given quote is no bad one at that.

The reason it's relevant is that some of us consider the Athenians to be "our kind", or at least the closest thing at the time.

Plaudits for actually explaining and justifying your rationality quote. May others follow your example!

"Our kind cannot cooperate" is a common meme for which I've seen comparatively little evidence. Mailing lists are not the real world, and while most people might start flame wars over the tiniest bullshit on mailing lists, their real-world behavior is largely cooperative and prosocial.

Would those be the same people you characterised by these words?

(ChristianKl) Normal civilized humans don't really want to kill other humans.

(eli_sennesh) Well, certainly not nearby humans who have similar skin coloration and evince membership in the same tribe. Those people, on the other hand, are disgusting, and the lot of them simply have to go.

I find it ironic that you use a military example to illustrate how we can achieve collective action at the civilization level.

Isn't the fact the Spartans were willing to "come back with their shields - or on it" the epitome of our kind not being able to cooperate?

I always interpreted "our kind" as the whole of humanity, so for me one sub-set of humanity banding together to destroy another subset (or die trying) isn't a good example of civilization-level cooperation, or the kind of meme that would be useful to spread.

I always interpreted "our kind" as the whole of humanity,

Did you read the linked article? In it Eliezer is contrasting rationalist and religious institutions. You may also want to read this to get an idea for the problem James Miller is trying to address. Here is a relevant quote:

Suppose that a country of rationalists is attacked by a country of Evil Barbarians who know nothing of probability theory or decision theory.

Now there's a certain viewpoint on "rationality" or "rationalism" which would say something like this:

"Obviously, the rationalists will lose. The Barbarians believe in an afterlife where they'll be rewarded for courage; so they'll throw themselves into battle without hesitation or remorse. Thanks to their affective death spirals around their Cause and Great Leader Bob, their warriors will obey orders, and their citizens at home will produce enthusiastically and at full capacity for the war; anyone caught skimming or holding back will be burned at the stake in accordance with Barbarian tradition. They'll believe in each other's goodness and hate the enemy more strongly than any sane person would, binding themselves into a tight group. Meanwhile, the rationalists will realize that there's no conceivable reward to be had from dying in battle; they'll wish that others would fight, but not want to fight themselves. Even if they can find soldiers, their civilians won't be as cooperative: So long as any one sausage almost certainly doesn't lead to the collapse of the war effort, they'll want to keep that sausage for themselves, and so not contribute as much as they could. No matter how refined, elegant, civilized, productive, and nonviolent their culture was to start with, they won't be able to resist the Barbarian invasion; sane discussion is no match for a frothing lunatic armed with a gun. In the end, the Barbarians will win because they want to fight, they want to hurt the rationalists, they want to conquer and their whole society is united around conquest; they care about that more than any sane person would."

And that's assuming the rationalists don't simply surrender without a fight on the grounds that "war is a zero sum game".

I didn't read the linked article--it certainly seems to frame the issue as rationalists vs. barbarians, not humanity vs. the environment (and the flaws of humanity), so thanks for pointing that out.

I do think fundamentalists/extremists/terrorists have an asymmetrical advantage in the short term in that it's always easier to cause damage/disorder than improvement/order. This quote above seems to be a particular example of this phenomenon.

However, I have to agree with Jiro's comment. Extremists may be able to destroy things and kill people, but I wouldn't say they've been able to conquer anything. To me, "conquer" implies taking control of a country, making its economy work for you, dominating the native population, building a palace, etc. Modern extremists commit suicide and then their mastermind hides silently for a decade until helicopters fly in and soldiers kill him.

When referring to actual barbarians, the description of the barbarians seems to lie by omission--even if all the things described above are mostly true, the barbarians have wrecked their economy because central planning doesn't work no matter how many orders they give, burning people at the stake is bad for investment, their belief in an afterlife is associated with other beliefs that prevent them from making or even efficiently using scientific advances, and their inability to have sane discussion means they can't make tactical decisions or really plan anything well at all. (Etc.) That sort of thing is pretty much the reason that the West hasn't been conquered by Muslim fundamentalists yet:

Also, barbarism doesn't arise at random. Some social structures are more conducive to barbarism than others and they may have inherent flaws which reduce the efficiency of conquest even as their encouragement of barbarism increases it.

the barbarians have wrecked their economy because central planning doesn't work no matter how many orders they give

Not all barbarians do that. The communists did that, but they also considered themselves rationalists and were considered such by many people at the time. Muslim fundamentalists generally don't.

burning people at the stake is bad for investment

Depends on whose being burned and why. Having the highest per capita rate of capital punishment doesn't seem to have hurt Singapore's ability to get investment.

their belief in an afterlife is associated with other beliefs that prevent them from making or even efficiently using scientific advances

I don't think so. It might hurt their ability to make scientific advancements, but they're perfectly capable of using them once someone else makes them.

Also, barbarism doesn't arise at random. Some social structures are more conducive to barbarism than others and they may have inherent flaws which reduce the efficiency of conquest even as their encouragement of barbarism increases it.

'Rationalist' societies can also have inherent flaws, like say problems solving the collective action problems associated with wars.

This feels to me like a just world fallacy, or perhaps choosing the most convenient world. Yes, if the barbarians are completely stupid, they are probably not so much of a danger these days. If they are completely anti-science, we probably have better guns.

Now imagine somewhat smarter barbarians, who by themselves are unable to do sophisticated science, but have no problem kidnapping a few scientists and telling them to produce a lot of weapons for them, otherwise their families will be burned at stake. (Even if their religion prevents them from doing science, they may compartmentalize and believe it is okay to use the devil's tools against the devil himself.) Suddenly, the barbarians have good guns, too.

Maybe the reason why the West hasn't been conquered by Muslim fundamentalists yet is that Muslims don't have an equivalent of Genghis Khan. Someone who would have the courage to conquer the nearest territory, kill horribly everyone who opposed them, let live those who didn't (and make this fact publicly known), take some men and weapons from the conquered territory and use them to attack the next territory immediately, et cetera, spreading like a wildfire. First attacking some smaller but civilized countries to get better weapons for attacking the next ones. With multiple leaders, so that dropping a bomb won't stop the war. (Maybe one Osama hiding in secret, giving commands to dozen wannabe Genghis Khans who don't mind getting to paradise too soon.)

Jiro's fallacy is not in saying that the world is or has been just in this respect, but rather in implicitly saying it must be. I don't think it's a coincidence that liberal/secular/enlightenment nations are the most powerful today, but that fact doesn't negate the point of the barbarian hypothetical.

I seriously doubt the viability of your Genghis Khan plan for modern fundamentalist Islam, seeing as that same M.O. was tried recently except starting with one of the world's top industrial and scientific powers. But that's a fact about our world, and the point of the barbarian example is more universal than that.