..fragments of a book that would never be written...

*      *      *

Captain Selena, late of the pirate ship Nemesis, quietly extended the very tip of her blade around the corner, staring at the tiny reflection on the metal.  At once, but still silently, she pulled back the sword; and with her other hand made a complex gesture.

The translation spell told Hirou that the handsigns meant:  "Orcs.  Seven."

Dolf looked at Hirou.  "My Prince," the wizard signed, "do not waste yourself against mundane opponents.  Do not draw the Sword of Good as yet.  Leave these to Selena."

Hirou's mouth was very dry.  He didn't know if the translation spell could understand the difference between wanting to talk and wanting to make gestures; and so Hirou simply nodded.

Not for the first time, the thought occurred to Hirou that if he'd actually known he was going to be transported into a magical universe, informed he was the long-lost heir to the Throne of Bronze, handed the legendary Sword of Good, and told to fight evil, he would have spent less time reading fantasy novels.  Joined the army, maybe.  Taken fencing lessons, at least.  If there was one thing that didn't prepare you for fantasy real life, it was sitting at home reading fantasy fiction.

Dolf and Selena were looking at Hirou, as if waiting for something more.

Oh.  That's right.  I'm the prince.

Hirou raised a finger and pointed it around the corner, trying to indicate that they should go ahead -

With a sudden burst of motion Selena plunged around the corner, Dolf following hard on her heels, and Hirou, startled and hardly thinking, moving after.

(This story ended up too long for a single LW post, so I put it on yudkowsky.net.
Do read the rest of the story there, before continuing to the Acknowledgments below.)

 


 

Acknowledgments:

I had the idea for this story during a conversation with Nick Bostrom and Robin Hanson about an awful little facet of human nature I call "suspension of moral disbelief".  The archetypal case in my mind will always be the Passover Seder, watching my parents and family and sometimes friends reciting the Ten Plagues that God is supposed to have visited on Egypt.  You take drops from the wine glass - or grape juice in my case - and drip them onto the plate, to symbolize your sadness at God slaughtering the first-born male children of the Egyptians.  So the Seder actually points out the awfulness, and yet no one says:  "This is wrong; God should not have done that to innocent families in retaliation for the actions of an unelected Pharaoh."  I forget when I first realized how horrible that was - the real horror being not the Plagues, of course, since they never happened; the real horror is watching your family not notice that they're swearing allegiance to an evil God in a happy wholesome family Cthulhu-worshiping ceremony.  Arbitrarily hideous evils can be wholly concealed by a social atmosphere in which no one is expected to point them out and it would seem awkward and out-of-place to do so.

In writing it's even simpler - the author gets to create the whole social universe, and the readers are immersed in the hero's own internal perspective.  And so anything the heroes do, which no character notices as wrong, won't be noticed by the readers as unheroic.  Genocide, mind-rape, eternal torture, anything.

Explicit inspiration was taken from this XKCD (warning: spoilers for The Princess Bride), this Boat Crime, and this Monty Python, not to mention that essay by David Brin and the entire Goblins webcomic.  This Looking For Group helped inspire the story's title, and everything else flowed downhill from there.

292 comments, sorted by
magical algorithm
Highlighting new comments since Today at 9:08 PM
Select new highlight date
Moderation Guidelines: Reign of Terror - I delete anything I judge to be annoying or counterproductiveexpand_more

I wonder if you might have seen this essay by David Brin...

Now ponder something that comes through even the party-line demonization of a crushed enemy -- this clear-cut and undeniable fact: Sauron's army was the one that included every species and race on Middle Earth, including all the despised colors of humanity, and all the lower classes.

Hmm. Did they all leave their homes and march to war thinking, "Oh, goody, let's go serve an evil Dark Lord"?

Or might they instead have thought they were the "good guys," with a justifiable grievance worth fighting for, rebelling against an ancient, rigid, pyramid-shaped, feudal hierarchy topped by invader-alien elfs and their Numenorean-colonialist human lackeys?

Picture, for a moment, Sauron the Eternal Rebel, relentlessly maligned by the victors of the War of the Ring -- the royalists who control the bards and scribes (and moviemakers). Sauron, champion of the common Middle Earthling! Vanquished but still revered by the innumerable poor and oppressed who sit in their squalid huts, wary of the royal secret police with their magical spy-eyes, yet continuing to whisper stories, secretly dreaming and hoping that someday he will return ... bringing more rings.

My guess would be that Mordor is a totalitarian communist state, formed on promises of empowerment of the People, and then turned into a horrible labor camp with collective farms by lake Nurnen and armies of expendable mooks kept in line by harsh superiors (think Commissars), along with heavy racist and nationalistic propaganda so they hate their enemies more than they hate their own rulers. Remember the communist revolution that happened in the Shire while our heroes were out destroying the One Ring? It started out as a ham-fisted attempt at social justice, and before long people were disappearing for being enemies of the state. Imagine that, but on a larger scale, and many times worse, and festering for generations.

The orcs (et al) don't have to be inherently evil for Sauronland to be an evil nation.

Right! I forgot this. Will add to acknowledgments.

When Brin (via RH) invoked his article on Overcoming Bias, Brian Moore (and Eliezer) invoked Jacqueline Carey’s "Sundering.” I'm surprised that Carey didn't show up in the acknowledgements. Brin & Carey are mentioned in another (ex-)OB thread.

Carey's book is even more powerful but it sends a totally different message - the idea that both sides have their reasons.

Nick Perumov wrote a huge fan-sequel to LOTR in exactly this vein. In the end the new rebel leader (who started out pretty good and gathered races with legitimate grievances) zbecuf vagb n zbafgre orpnhfr ur'q hfrq gur anmthyf' yrsgbire evatf gb tnva fgeratgu, naq hcba ernyvmvat gung ur fheeraqref gb gur cebgntbavfg gb trg xvyyrq.

EDIT: rot13'd the spoilers. Which doesn't mean I recommend reading the book!

Please edit this to rot13 the spoilers. You don't say: "X wrote a wonderful story and here's the ending", just "X wrote a wonderful story and here's the link".

I don't advise you to, and anyway who'd translate a Russian fanfic into English and put it online?

I think Sauron did enough explicitly evil stuff to make himself the bad guy. Tricking the Numenorians into destroying themselves out of spite is pretty hard to justify.

There's also the fact that orcs don't have free will. They were created from tortured elves and mindraped into obedience. The fact that Sauron was willing to use them as canon fodder rather than trying to find a way to reverse what Melkor did to them speaks worlds about his moral virtue.

Finally, the rings. Using mind control to turn foreign leaders into your obedient thralls, consoling them with the promise that they will be able to crush others under their heel as you crushed them. Real nice of Sauron.

Middle Earth was a flawed world filled with the same evils and injustices as our own, but Sauron was almost definitely the worst thing in it. I'll give David Brin some credit, though, as Tolkien did a pretty bad job of explaining the situation in Lord of the Rings. You have to read the Silmarilion (or, as in my case, talk to another guy who has read the Silmarilion, as I lacked the patience to wade through another gazillion pages of archaic English) to understand what's going on, which is a major failing of LotR.

They were created from tortured elves and mindraped into obedience.

Did you learn this from an unbiased source?

Finally, the rings. Using mind control to turn foreign leaders into your obedient thralls, consoling them with the promise that they will be able to crush others under their heel as you crushed them. Real nice of Sauron.

Suppose you're the prime minister of a parliamentary republic, and the neighboring country is ruled by hereditary nobility that mostly hate each other, and wars between the barons ruin a lot of the land and kill a lot of the peasants. You, being a genius engineer, have figured out a way to control people, but it requires they wear the device for an extended period of time, the effects are obvious, and they can take it off before the process is complete if they feel like it.

This hereditary nobility situation is obviously not going to fix itself- and you figure that the easiest way to fix it is to corrupt all the nobility, playing on their hatred of each other to get them to wear the devices long enough for them to work, and then have them give you power in a bloodless coup. As a bonus, you now have fanatically loyal assassins / spec ops forces, and an eternity of servitude seems like a fitting punishment for their misconduct as rulers.

"Did you learn this from an unbiased source?"

I'm pretty sure it was in Tolkien's notes.

"Suppose you're the prime minister of a parliamentary republic, and the neighboring country is ruled by hereditary nobility that mostly hate each other, and wars between the barons ruin a lot of the land and kill a lot of the peasants. You, being a genius engineer, have figured out a way to control people, but it requires they wear the device for an extended period of time, the effects are obvious, and they can take it off before the process is complete if they feel like it."

Except that's exactly what Sauron DIDN'T do. Mordor was not a parliamentary republic; more like a military dictatorship with semi-mindless orc drones enforcing Sauron's commands over his human subjects. The monarchs who were given the rings - however just or unjust their rule might have been, and however flawed the notion of monarchy as a political system - were lied to about what the rings did, and the rings' effects were very subtle at first.

Its also worth noting that the human kings didn't become any kinder or more democratic in their sensibilities once they fell under Sauron's influence. The Witch King was still a king, and a much more murderous one than he was in life. Unleashing barrow-wrights on a partially civilian population, torturing Gollum for information, and stabbing an innocent (if possibly misguided) hobbit when he didn't have to are all things that the Witch King did in person.

"This hereditary nobility situation is obviously not going to fix itself- and you figure that the easiest way to fix it is to corrupt all the nobility, playing on their hatred of each other to get them to wear the devices long enough for them to work, and then have them give you power in a bloodless coup. As a bonus, you now have fanatically loyal assassins / spec ops forces, and an eternity of servitude seems like a fitting punishment for their misconduct as rulers."

In other words, the only way to improve the world is to become just as bad as the people currently running it? The best solution to dictatorships is to make slaves of your own, and for all eternity no less?

I think you're going out of your way to defend Brin's essay rather than actually using your own moral judgement. You can easily say that the "good guys" in Lord of the Rings weren't all that good, but Sauron was very obviously worse.

I'm pretty sure it was in Tolkien's notes.

Right, and Brin's premise is that Tolkien is a biased source.

In other words, the only way to improve the world is to become just as bad as the people currently running it? The best solution to dictatorships is to make slaves of your own, and for all eternity no less?

If those slaves were the dictators of the old era? Seems suitably karmic.

I think you're going out of your way to defend Brin's essay rather than actually using your own moral judgement.

"My own moral judgment" is a tricky thing in this situation, as it depends on which situation we're describing.

If I have first-hand experience of the events of LotR, and everything is as Tolkien describes it, then yeah, it's pretty obvious that Sauron is the bad guy.

If I have third-hand experience of the events of LotR, think that at most 90% of the description is accurate, and I think that the philosophies of the modern day are present in the LotR world, then it seems plausible that Sauron is the good guy, for the reasons Brin describes.

You might be interested in The Sword of Good, if you haven't read it. [edit] It looks like you commented there today, but I'll leave the recommendation here for any spectators to the conversation.

You might be interested in The Sword of Good, if you haven't read it.

Amusing because you linked to this very post.

That is amusing, and what I get for jumping into conversations from the Recent Comments link and not thinking to check where the conversation is happening. I'm tempted to edit it out, but might as well leave it for posterity.

Why not put a copy of those acknowledgements at the bottom of the story itself, as well?

I suspect lots of people are going to see the story and only think of it as a neat story. Explicitly bringing out the lesson would help, plus I thought the Passover example was really interesting. It wouldn't hurt to make more people see it.

My first impression of this story was very positive, but as it asks us to ask moral questions about the situation, I find myself doing so and having serious doubts about the moral choices offered.

First of all, it appears to be a choice between two evils, not evil and good. On one hand is a repressive king-based classist society that is undeniably based on socially evil underpinnings. On the other hand we have an absolute unquestionably tyranny that plans to do good. Does no one else have trouble deciding which is the lesser problem?

Secondly, we know for a fact that, in our world, kingdoms and repressive regimes sometimes give way to more enlightened states, and we don't know enough about the world to even know how many different kingdoms there are or what states of enlightenment exist elsewhere. For all we know things are on the verge of a (natural) revolution. We can't say much about rule by an infinite power, having no examples to hand, but there is the statement that "power corrupts". Now, I'm not going to say that this is inevitable, but I have at least to wonder if an integration over total sentient happiness going forward is higher in the old regime and its successors, or in the Infinite Doom regime.

Finally, the hero is big into democracy. Where in either of these choices does the will of the peasants fit in anywhere?

EDIT: One more point I wanted to add, since its clearly not a Choice Between Good and Evil as the prophesy states, why assume there is a choice, or that there are only two options. Would not a truly moral person look for a third alternative?

Does no one else have trouble deciding which is the lesser problem?

I gathered that the choice being a difficult one was the whole point. It's not a genuine choice if the right choice is obvious, that much was explicitly stated.

You say it "clearly" wasn't a Choice Between Good and Evil, but I don't think that's clear. One choice might still have a good outcome and the other an evil one. It's just that we don't know which one is which.

It would say that the likelihood is overwhelming that BOTH choices will lead to bad ends. The only question is: which is worse. That's why I was saying it was between two evils.

Besides, its hard to reconcile the concept of 'Good' with a single flawed individual deciding the fate of the world, possibly for an infinite duration. The entire situation is inherently evil.

Though it wasn't explicitly said, it was heavily implied that either choice would be for a potentially infinite duration. This is a world of fantasy and prophecy, after all: I got the impression that the current social order was stable, and given that there was magic (not psychic ability but magic) it's also fair to assume that the scientific method doesn't work (not that this makes any sense, but you have to suspend that disbelief for magic to work [gnomes are still allowed to build complex machines, they're just not allowed to build useful machines]).

The way I interpreted it was that he had a choice between the status quo for 1000 years, or and unknown change, guided by good intentions, for 1000 years.

Besides, the Big Bad was Marty Stu. How could I not side with him?

(Another great work, Yudkowski - you really should send one of these to Asimov's SciFi)

Interesting. Is hard to reconstruct my reasoning exactly, but I think that I assumed that things I didn't know were simply things I didn't know, and based my answer on the range of possibilities -- good and bad.

Huh; I thought my browser had failed, and this post hadn't appeared. Anyway...

There's an old army saying: "Being in the army ruins action movies for you." I feel the same way about 'scifi' - Aside from season 3, every episode of Torchwood (that I've recently started watching, now that I finished Sopranos) is driving me up the wall. I propose a corollary saying:

"Understanding philosophical materialism and the implications thereof ruins 99% of Science Fiction... and don't get me started on Fantasy!"

In my opinion, there are three essential rules to Fantasy:

  1. The protagonist is a priori important; by their very nature they have metaphysical relevance (even though they don't know it yet!). All other characters are living their rightful and deserved life, unless they are below their means with a Heart of Gold.

  2. The scientific method (hypothesis, experiment, conclusion, theory) only works in the immediate sense, not the broad sense; your immediate world will be logical, but the world as a whole is incomprehensible. You can only build machines if a) they already exist; or b) they serve no practical purpose. Magic, on the other hand, generally works as intended; the human will guides it, and can only be countervened by another magical authority (a navigation spell will not require knowledge of the local plant life, nor will it require accurate grid coordinates given a non-simultaneous Relativistic geometry).::If magic doesn't work as the protagonists intend, it will be working under a higher moral power.

  3. There is an abstract and absolute division between Right and Wrong; somebody is keeping score, and no actions are hidden. Your evil acts might escape the notice of the local authorities, but they will show through by your bearing, your beauty, or your image.

Heh, this might be worth a top level post except tvtropes has covered it all already.

The most rationalist-relevant TV Tropes would easily be worth a top post or three.

You'd lose your whole crop of rationalists. They would never come back out.

I agree, which is why I tend to shy away from performing a moral analysis of Fantasy stories in the first place. That way lies a bottomless morass.

Fantasy stories, and ninety percent of science fiction nowadays...

Extend this beyond fiction. What misdeeds are we shrugging off because they're normal?

Apartheid based on age that replaces the previous versions based on race and sex.

The morally indefensible and insanely self-destructive attempt to mitigate drug addiction by banning drugs.

What misdeeds are we shrugging off because they're normal?

Religion. Schools. Television. Not caring about people remote from you. Spending effort on trifles. Akrasia. Irrationality.

Some would say, having political beliefs different from mine.

Burial/cremation.

Loss of time to work. Loss of utility to unemployment.

The way children get so few civil rights they're used as excuses for removing rights from adults.

The way children get so few civil rights they're used as excuses for removing rights from adults.

I am aware of the poor state of affairs re: children's rights, but I'm not sure what you're getting at by citing consequences for adults. Can you elaborate?

Just think how much legislation that restricts adults has been sold on the premise that it "protects children", especially from non-harmful things like porn and homosexuality.

As far as schools, do you mean something about the specific way that we have schooling set up currently (and do you include universities in that?) or do you mean more generally?

As far as schools, do you mean something about the specific way that we have schooling set up currently (and do you include universities in that?) or do you mean more generally?

I had in mind the education of children in school, as done in, I think, all of the developed world and a lot of the rest, and critiques like this one.

Universities may also have their faults, but not on the scale of misdeeds being considered, and, anyway, the people in them chose to go there.

Aaaah, okay. Yeah, I agree that that's a nasty aspect of our system.

having to kowtow/kiss-up to bullies because they're part of a hierarchy (eg in business), rather than being treated with respect as a human being. Further - being expected to also treat your subordinates badly and thus perpetuate the hierarchy.

What misdeeds are we shrugging off because they're normal?

Eating mammals. More generally; non-vegetarianism.

I think being non-vegetarian is less evil than being a morally inconsistent non-vegetarian. If you would have moral trouble being introduced to your food (or raising it) then you shouldn't be eating it.

I've never understood this argument. I have a visceral reaction against surgery (even the sight of blood can set me off); I certainly couldn't stand to be in the same room in which surgery was being performed. Does this mean that for consistency I'm required to morally oppose surgery?

I don't see why. For clarity, since we probably agree it's wrong, imagine you're making the same argument for cannibalism instead. One person says, "I'm fine with eating and farming humans but if I get to know one first, doing it would make me feel bad." Another says, "Screw that, I'll eat anyone, even if I know them and their children!"

The second person is more morally consistent and also more callous. Even if there's no difference in the way they live their lives, trying to end the holocaust of humans for food would be easier in a world full of the first type than the second.

Just as I would prefer the opposite of rule of law when the law is uniformly terrible, I prefer the opposite of moral consistency when a morality is terrible.

I don't see why. For clarity, since we probably agree it's wrong, imagine you're making the same argument for cannibalism instead. One person says, "I'm fine with eating and farming humans but if I get to know one first, doing it would make me feel bad." Another says, "Screw that, I'll eat anyone, even if I know them and their children!"

The second person is more morally consistent and also more callous.

The consistency difference seems minimal. The most obvious moral rule in play is "Don't do harmful things to those people who are socially near" combined with a moral indifference to cannibalistic farming but acknowledgement that it is undesirable to be so farmed and eaten. This isn't a complex or unusual morality system (where arbitrary complexity seems to be what we mean when we say 'inconsistent').

Specifically, most people assert that animals are sentient; yet most people are not vegetarians, even though eating meat is no longer necessary for survival. There is an inconsistency between these positions.

You missed the step where you assert that most people assert it is wrong to eat sentient animals, which is what would create the inconsistency, were most people to assert that.

You also need the word sentient to mean the same thing in both premises, otherwise it's like “feathers are light, what's light is not dark, therefore feathers are not dark”.

Okay, but if offered the opportunity to kill and eat a human, or an elf, or a Wookie, most people would recoil in moral revulsion, and if you asked them "is that because you think it's wrong to kill and eat sentient beings" would probably say yes, so I think most people do assert that.

Actually, "Would you eat a Wookie?" is probably a helpful distinguishing question here. For me the answer is obviously "No!" and occurs with the same fleeting nausea as "Would you eat a human being?" But I grew up reading SF books like Little Fuzzy that teach personhood theory in a very visceral way. Other readers claimed they weren't bothered by the Babyeaters because the children eaten weren't human!

because the children eaten weren't human!

Indeed, one thing that surprises ethicists their first time teaching is that in ordinary English, 'person' and 'human' mean the same thing - so most intro students, when asked 'is Yoda a person' will answer 'no', even though they'd answer 'yes' to 'is Luke Skywalker a person'.

Maybe you need to ask, "Would you eat Yoda if his species were tasty?"

I was just the other day lamenting how many people, even largely intelligent and conscientious seeming individuals, answer "Yes" to the OkCupid match question "If you landed on an alien planet where the local intelligent life form tasted unbelievably good, would you eat them?"

I'm TAing discussion sections for the first time today, and based on some of the nonsense the students spouted in lecture yesterday, I'm going to need to cover what those words mean.

Update: I had one person say she would be fine with barbecuing Yoda because he wasn't human. I used this to segue into my explanation of what it means to bite the bullet.

I imagine that would be because most people don't understand that sentient beings includes chickens, lobsters[1], and unborn fetuses (not that many people would agree with eating fetuses). If you asked "is that because you think it's wrong to kill and eat beings that are capable of perceiving stimuli" most would probably disagree with you. Now, if you asked "is that because you think it's wrong to kill and eat beings that are capable of doing algebra," you'd probably get a different response.

The reason people wouldn't eat an elf isn't because it's a sentient being, it's because it's a human equivalent sentient being. So you need to reach beyond sentience to find your inconsistency.

And of course, the reason people wouldn't eat a Wookie is because it probably would taste like an old boot.

[1]Research in recent years suggests that crustaceans may be capable of feeling pain and stress.

Research in recent years suggests that crustaceans may be capable of feeling pain and stress.

Pain and stress in crustaceans? Source: Applied Animal Behaviour Science.

We consider evidence that crustaceans might experience pain and stress in ways that are analogous to those of vertebrates. Various criteria are applied that might indicate a potential for pain experience: (1) a suitable central nervous system and receptors, (2) avoidance learning, (3) protective motor reactions that might include reduced use of the affected area, limping, rubbing, holding or autotomy, (4) physiological changes, (5) trade-offs between stimulus avoidance and other motivational requirements, (6) opioid receptors and evidence of reduced pain experience if treated with local anaesthetics or analgesics, and (7) high cognitive ability and sentience. For stress, we examine hormonal responses that have similar function to glucocorticoids in vertebrates. We conclude that there is considerable similarity of function, although different systems are used, and thus there might be a similar experience in terms of suffering. The treatment of these animals in the food industry and elsewhere might thus pose welfare problems.

No more prawn cocktails or shrimp sandwiches for me.

Is there really a place where both 'prawn' and 'shrimp' are used? What's the difference?

You probably looked it up a long time ago, but for any future readers: They're different groups of species. Both are soft-shelled crustaceans, but that's where the similarity ends.

Any morphological similarities are probably down to converging evolution.

Ha... actually, I didn't look it up at all. According to Wikipedia, you're right, but 'shrimp' is the common name for a lot of things that get called 'prawns' outside of the US.

Part of what bothers me about the idea of eating an elf or Wookie is that they don't feel like prey -- they feel like peers. When I see a fox, e.g., it doesn't make me hungry -- the fox doesn't seem like it's below me on the food chain. When I see a rabbit or a pigeon, it does make me hungry -- I can imagine what it would be like to hunt, clean, roast, and gnaw on it.

On the other hand, I wouldn't hesitate to kill 5 foxes to save one elf or human or Wookie.

I would not, however, hunt a rabbit or a fox for sport; that seems unnecessarily cruel.

One way of accounting for all these moral intuitions is that rabbits, foxes, and Wookies are all sentient; one should not cause pain to sentient creatures for amusement. Foxes and Wookies are ecological peers; one should not eat ecological peers. Wookies are people; one should not trade off the lives of people against roughly comparable numbers of lives of non-people.

I think I'm among the few who after realizing this, as well as how icky the sources of most foods are when you think about them, and that most danger from eating stuff is from things that don't seem disgusting, decided that food revulsion is not a part of me and that I should be perfectly fine with eating human flesh or drink [self-censored]. I haven't actually tested any of this, so I'm not sure if my brain would go along with it.

I think Joshua Greene, among others, has investigated these sort of things (moral intuitions, and the justifications people typically give, which may be a sort of confabulation).

http://www.wjh.harvard.edu/~jgreene/

Specifically, most people assert that animals are sentient; yet most people are not vegetarians, even though eating meat is no longer necessary for survival. There is an inconsistency between these positions.

No there isn't. It implies that they violate another norm that you value but it is not inconsistent.

It's striking how different our cultural response seems to be to political assassination by knife or political assassination by airstrike.

I don't think this is the right distinction. Osama bin Laden for example was killed in person by American special forces, which probably isn't that unusual a type of targeted killing but rarely