LESSWRONG
LW

I preferred the superhappy ending, and in fact the story nudged me further in that direction. I guess I don't really get what the big deal about pain and suffering is, there's no physical pain on the internet and it seems to work just fine.

[-]nazgulnarsil317y00

three worlds collide would make a decent movie...just have to make the reasoning of the characters more explicit for people unfamiliar with concepts involved.

[-]Andy_McKenzie17y130

Eliezer, one qualm: You consistently bring up mirror neurons and consider it to be obvious prima facie that they are used for action understanding in humans. Unfortunately, most contemporary neuroscientists in the field agree that there is no consistent evidence of this:

http://www.cognitionandculture.net/index.php?option=com_content&view=article&id=223%3Ado-we-have-mirror-neurons-at-all&catid=32%3Aoliviers-blog&Itemid=34

http://talkingbrains.blogspot.com/2008/08/eight-problems-for-mirror-neuron-theory.html

That is not to say that humans don't understand other people's actions or that we do not have adequate theory of minds! But it does mean that there is no reason to suspect that those complicated cognitive events can be reduced to simply a group of "mirror" neurons. Ramachandran often mentions them as well, which irks me slightly as well.

[-]Eliezer Yudkowsky17y50

3WC would be a terrible movie. "There's too much dialogue and not enough sex and explosions", they would say, and they'd be right. And you shouldn't just tack them on, either; sex and explosions should flow out naturally as an indispensable part of the plot.

Andy, consider "mirror neurons" as shorthand for "empathic architecture" rather than implying that the whole thing gets done by a small group of actual neurons a la the "grandmother neuron".

[-]Strange716y230

"There's too much dialogue and not enough sex and explosions", they would say, and they'd be right. And you shouldn't just tack them on, either; sex and explosions should flow out naturally as an indispensable part of the plot.

Bullshit. There's that part about rape having been legalized, and the canonical ending involves a planet being destroyed by a supernova. In a visual medium, the hypothetical discussion about exterminating baby-eaters would be voiceover for a montage of helpless crystalline civilians being hunted down by human infantry with power armor and personal laser cannons, culminating in an orbital bombardment.

[-]Zubon17y120

Eliezer: if you show more of it from the perspective of the Superhappies, dialogue itself takes care of that problem.

[-]Tom_McCabe217y10

"3WC would be a terrible movie. "There's too much dialogue and not enough sex and explosions", they would say, and they'd be right."

Hmmm.. Maybe we should put together a play version of 3WC; plays can't have sex and explosions in any real sense, and dialogue is a much larger driver.

[-]Kaj_Sotala17y120

And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light.

Hmm. I felt that while the Superhappy argument was presented in the best possible light, the Superhappy ending wasn't. The non-Superhappy ending was in two parts and contained all kinds of cool things, like the use of emergency flags to manipulate the local prediction markets and more insights into the personalities of the different characters. The Superhappy ending, on the other hand, was just one part and was basically just a pretty dull overview of how everybody considered humanity's future to be horrible.

To me, it felt like a pretty blatant statement of "this is the future that we don't want".

[-]Psy-Kosh17y70

Frankly, what I've kinda on and off wanted to see was someone turn Nick Bostrom's "Fable of the Dragon Tyrant" into a movie. That could, perhaps, actually work. Maybe.

4MBlume13y

Fable of the Dragon Tyrant would make a good animated short, I think.

[-]Tyrrell_McAllister217y30

I think that you make good points about how fiction can be part of a valid moral argument, perhaps even an indispensable part for those who haven't had some morally-relevant experience first-hand.

But I'm having a hard time seeing how your last story helped you in this way. Although I enjoyed the story very much, I don't think that your didactic purposes are well-served by it.

My first concern is that your story will actually serve as a counter-argument for rationality to many readers. Since I'm one of those who disagreed with the characters' choice to des... (read more)

[-]Eliezer Yudkowsky17y00

Stories and movies are deeply different media. I'm surprised the transition works as often as it does.

[-]JamesAndrix17y10

I've recently become jaded on the use of fiction due to a nearly opposite line of thought: Fiction is really not important.

There are plenty of real problems and real drama.

Eliezer was not able to write about the experience of being a rationalist merely because he read about the experience of being a rationalist in some other work of fiction. Narrative might be required to convey experience, fiction is not.

It is the truths you are trying to convey, that is what matters. But to this we add invented societies, speculation about future technologies, or future ... (read more)

[-]billswift17y120

"Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves."

Nearly all people for thousands of years thought it was perfectly all right to keep slaves. Are you saying they didn't have the ability to sympathize? This is the sort of profoundly ahistorical "thinking" that irritates so many people. Someone who considers his own society's beliefs to be laws of reality when there is obvious historical evidence in the other direction that they never bothered to think about.

0blashimov13y

My take is that without the ability to sympathize we wouldn't have stopped. Not that we suddenly learned/developed sympathy.

3johnlawrenceaspden13y

I don't think it's perfectly all right to eat meat. And I consider battery farming to be an obscenity which I'd like to see outlawed. And yet I do eat meat, guiltlessly, and not all of it is free-range ethics-meat. It occurs to me that I might have trouble explaining this to some future ethicist, and I wonder if that might be the same sort of moral state that the Romans, say, were in.

5Richard_Kennaway13y

It's the same sort of moral state we're all in.

5MugaSofer13y

I mock you from the safety of my vegetarianism, while desperately downplaying who made my trainers.

-2MugaSofer13y

In most societies, "slaves" were closer to indentured workers than the modern conception of slavery, which descends from the racist North American slave trade, IIRC.

2johnlawrenceaspden13y

I think I'd take a Southern cotton plantation over a Roman mine any day. And I never heard that the Confederacy lined their roads with crucified rebels.

6MugaSofer13y

Bear in mind that most slaves didn't go to the mines - and they were often used as a punishment. I had trouble finding details on the punishment of rebel slaves in the Americas - they appear to have been rarer and more successful in escaping capture compared to Roman slaves - but here's something on "maroons" (self-governing pockets of escaped slaves.) [source]

1johnlawrenceaspden13y

These guys sound pretty heroic, but I don't think they're evidence that the racist transatlantic slave trade was worse than the non-racist Roman world. I'm not an expert on either, though. Part of what I'm trying to assert is that people are capable of treating other people terribly, even in the absence of theories of racial superiority. I'm pretty sure that the Romans looked up to the Greeks at the same time as enslaving them. And fairly sure that the Greeks enslaved other Greeks. But you'd need to know a lot more about the classical world than I do to work out what kinds of racial theories were current. And maybe they did have foreign groups that they mistreated particularly badly. If we think that xenophobia is a built-in feature of the brain then it would be damned weird if the Romans weren't superiority-complex racists. After all, consider the amount of evidence they had that their system was superior and that the gods loved them. I'd be surprised if it wasn't worse to be the slave of someone who despises you and your type than the slave of someone who accepts you as a brother. I just don't think any of this is particularly modern. And on ethical matters I tend to think that progress is upwards (or at least correlated with per-capita GDP). If we think that the recent past was particularly awful it's usually because we've got better records of it. So here's a prediction for you: There were things going on in the Dark Ages that were worse than either Roman or early Victorian slavery. The problem is, I can't think of anything worse. There's something particularly terrible about mass industrial slavery. Maybe some passing atrocitologist can help.

3DaFranker13y

I'm not quite an atrocitologist so I have no idea whether some of these things I can think of were actually ever put into practice, but I can think of lots of things worse. I can also guarantee you with 90% confidence that there's a lot of manga (especially doujinshi) out there that do picture things you'd consider much worse, especially when you delve into the darker circles. Some japanese artists have literally become world-renowned 'experts' on the topic of fictional mass atrocity. I'm not comfortable discussing specific examples without a wall of spoiler prevention features requiring the viewer to pass a mental fortitude test to view the content. I might have mentioned this before, but I've once had an acquaintance bend down and vomit on the spot upon recounting one of my more horrible nightmares. I try to avoid dishing out such mental damage on unprepared individuals nowadays.

0johnlawrenceaspden13y

Touché. I meant something that was likely to have actually happened on a fairly large scale.

2MugaSofer13y

Now I'm all curious. To hear things so bad they make unprepared listeners spontaneously vomit, not to hear things worse than slavery. There are plenty of those, they just tended not to catch on.

5Nornagest13y

Well, I'm not exactly an atrocitologist, but I have studied the early medieval period in some detail. There are some problems in comparing it to other periods, especially in subjective terms -- the Dark Ages were called "dark" precisely because they left a relative dearth of subjective material -- but here's what I can remember off the top of my head. There was a widespread slave trade, beginning during or before Roman times and ending in Britain around 1100 AD. It was not racially motivated or justified, as we'd understand race; slaves came from all the European ethnic groups, including those of their holders. Taking slaves seems to have been more common in conflicts between ethnic groups, however. Unransomed captures in wartime and freemen who fell into various kinds of legal trouble could both become slaves; the former seem to have been more common. They generally could be bought and sold and didn't have legal independence. The law codes of the time prescribed punishments for mistreating other people's slaves but not your own. Slave labor was not usually highly concentrated or regimented (there were, for example, no galley slaves in that period); slaveholders came from all free social classes, and slaves performed much the same work as freemen (though usually the harder and dirtier shares of it, where division of labor was possible). At the time of the Domesday Book, slaves made up about 9% of the population. From what I know of it, this seems more comparable to Roman slavery than to the trans-Atlantic slave trade. Early medieval Europe was a poorer place than either Rome or the early modern colonies, and its people probably led harsher lives, but in social terms I don't see much in the way of unique awfulness.

1fezziwig13y

I wish I'd thought to pick 'Atrocitologist' as a screen name. Oh well. I can't think of any medieval atrocities comparable in scope to those of either the Roman or Victorian eras. But I don't think that has anything to do with philosophy or tolerance, it's just that Rome and pre-Victorian England were a lot more powerful and effective than any of the intermediate governments, and so were able to achieve greater scope than e.g. Poland ever could. But to your more general point: modern racism is just a special case of the human tendency to define ingroup/outgroup divisions, right? It's ok to enslave Them, because they're not Us. That finding is extremely robust through history: Greeks enslaved other Greeks (but they called themselves Spartans and Helots), Italians enslaved other Italians (but the victims were never Roman citizens so it didn't count), the Jews wiped out the Amelikites (they worshipped the wrong gods, what can you do?) and French nobles ruled over French serfs (but you can't compare a noble to a serf).

0MugaSofer13y

Romans could be sold into slavery to pay off their debts. The Romans were reletively free of out-group hostility - they felt the barbarians outside the empire were savages, but they tended to absorb local power structures and religions, granting the local nobles (if they cooperated) Roman citizenship, (which was more exclusive than, say, American citizenship,) and while there was some generic snobbery there does not appear to be any belief that non-Romans were inherently inferior. Once they joined the empire, they gained all the rights and privileges of your average Roman (including protection from those barbarian savages over the hill.)

0MugaSofer13y

They aren't. However, grants context to your statement that "I never heard that the Confederacy lined their roads with crucified rebels." Xenophobia and racism are different things. Is that ... a Culture ship name?

4DaFranker13y

I'm not sure that holds. In my model and priors, the racist NA slave-owners are much less likely to do certain categories of negative things, and are actually on the higher end of the nice-slavery scale on account of them usually being too disgusted or other-negative-feelings to bother doing anything to them but the occasional light no-contact beating and lots of forced labor. For instance, considering that I'd guesstimate roughly 10% of Roman Empire slaves to be privately-owned females, and that of those a certainly non-negligible amount would be attractive to their owners or people-the-owner-wants-to-be-friends-with somehow, then it's not that far-fetched to guesstimate that around 5% of Roman Empire slaves were, in fact, used often or mostly for sexual purposes. This obviously includes rape and torture and whatever other fun things the owners/friends-of wanted to do. I'd also presume that it was much easier to start off a roman sex-and-food orgy with slaves than with free females that are otherwise just interested in having such an orgy considering what I know of cultural gender expectations/roles for that culture and of the numbers of such free vs slave women. I've also heard that the ancient Chinese had some pretty sick ideas (both horrible and genius at the same time) of what to do with slaves during their constant warfare, including but not limited to: Training items (e.g. "These slaves have wooden swords, but we've broken half their fingers and wounded their arms. Today, you will be training how to effectively cripple and immobilize an enemy with your ranged weapons before they get within sword range!"), battlefield entertainment, emergency food reserves, literal meat walls (no, not putting them on the front lines to fight; literally slaying and piling them at a particular spot just to hinder the enemy movements and cause psychological damage), and various forms of experimentation, testing of poisons/herbs, or whatever-the-owner-felt-like-doing-to-them.

4TimS13y

This is true, but chattel slavery (the type of slavery practiced in the American South) is not the only kind of slavery practiced in history. Some slave cultures (Greek and Roman in particular) allowed a real possibility of manumission into freedom, or even citizenship. Assuming chattel (or ethnicity-based) slavery was the only kind is a failure mode in the general public. Edit: I'm not sure where you got this idea: The brutality of US chattel slavery varied from place to place and time to time, but there's mixed evidence at best for genuine sympathy for slaves among the slave-owners or free folks. A lot of pro-slave court cases are successful attempts by slave-owners to punish slave-renters who mistreated the property or free bystanders engaging in destructive "vandalism".

0DaFranker13y

Re. that quote, I was referring mostly to the kinds of abuse I cite in example in the rest of the comment. From what I know, it wasn't particularly common for American South slave-owners to specifically select for attractive female slaves and routinely rape or otherwise abuse them for personal pleasure - having sex with "niggers" was, as I understand it, a deep wrongdoing. I also don't imagine the american slave-owners of that culture engaging in all that many orgies or the various practices of viking naval slaves or ancient chinese generals.

2TimS13y

I'm not sure where you are getting the violent sex aspect of slavery - I'm not an expert, but I'm not aware of that as a widespread practice. Treating slaves' lives as cheap (like your examples from Chinese history) doesn't necessarily imply violent orgies. In my mental model, sex slaves are mentally coerced, not physically coerced. Assaultive rape isn't the first image that appears in my head for sex-with-slaves. And US chattel slavery is weird about sex with slaves. For example, Sally Hennings is not an usual story, even if that kind of conduct was considered improper. Further, the idea of slaves as sexually charged beings continues to run through current American ideas about US blacks. Consider the trope that black men have above average sized penii.

0DaFranker13y

This is what I have in mind when I say "routinely rape or otherwise abuse them for personal pleasure". I'm not sure either where I implied a violent or assaultive aspect. Playing on the helplessness of the victim and various forms of mindbreak or psychological coercion is clearly the dominant tactic in most cultures of sexual slavery as far as I'm aware.

0TimS13y

Well, from the great grandparent: In general, strangers-to-the-bond were not permitted the same liberties with slaves as owners.

0DaFranker13y

Ah. I'm doubtful that the line between owners doing what they wanted with their slaves and their friends doing the same to same slaves was that clear-cut or actually upheld, but it makes sense now that I think about it - it certainly makes sense that strangers would avoid damaging the slaves of others or when in doubt, since that would legally amount to high vandalism and property damage. My conception of "torture" is very large. Sure, that includes strapping someone to a chair and flogging them, but I'm not so skewed as to believe such things were commonly practiced onto most categories of slaves unless there were special circumstances. Things like promising slaves a real meal if they work twice as hard for the day and then giving them authentic dog feed once they're done counts as torture within this enlarged label. Such things were, I'm given to understand, very widespread among various slavery cultures, sometimes as a form of entertainment. To my understanding, they're currently very widespread in current 2012 slavery (though apparently most people nowadays use the term "human trafficking", which obscures from discussion what actually happens once the person has been trafficked).

0TimS13y

There's a relatively recent post about overly expansive definitions and why the changes in connotation they create are bad - but I can't find. I will say that a wildly nonstandard definition of a term increases inferential distance. And there are labels for the concepts you want to gesture towards - expanding the reach of the label "torture" is not necessary.

0DaFranker13y

I wasn't aware of better labels that could be used in nontechnical discussion, and I can't think of any at the moment. In practice I've used this expanded "torture" label in discussions with various sorts of people both specialized and not (though this does not include professional sociologists, historians or moral philosophy experts), and found that it was usually understood from context without additional input, and otherwise a single extensional example (like the typical bullying case where a couple of kids take someone's bag and throw it around while the owner helplessly runs around trying to get their stuff back) was sufficient for them to make the remaining inferences. Granted, this may in part be due to body language and other unspoken information channels, and I have transmitted neither this nor any examples before the grandparent. I also didn't think much of it at the time of writing, so I'll agree that it was a mistake, since I could just as well have rationalized in this manner even if I did not observe it as a successful label in previous instances.

1TimS13y

That makes sense. I'm just trying to distinguish between behavior a "law-abiding" slave might never encounter in her lifetime, as opposed to what she probably deals with on a weekly basis (on average - and that's not intended to count the standard "get back to work" instruction). A substantial part of the confusion in this conversation was my reading your words as asserting the former behavior (archetypal torture) was more frequent than historically occurred. My understanding was that behavior towards escape attempts and such (most of which would qualify as torture) was very different than "ordinary" treatment of slaves in the American South. A phrase like emotional abuse or just abuse might be reasonable label, depending on what other connotations it brings to the conversation.

0DaFranker13y

Indeed, in retrospect that should have been an obvious choice. I've been conditioned by the French word "abus" (also the verb "abuser") which carries different connotations and which in common usage would generate a lot more misunderstanding, so I tend to underuse the words "abuse" and its subsets.

-4MugaSofer13y

No. Raping slaves was common, and was often prompted by racism - after all, if the kid's half-white he'll make a better slave, wont he? No-one thought they were unattractive, just "inferior".

8MugaSofer13y

It's hard to get estimates for beatings - there appears to have been a great deal of variation - but let's just say that racism is unlikely to decrease beatings. And American slaves were definitely raped. A lot. Sometimes because they were considered inferior - after all, if the kid is half white, they'll make a better slave, right? Roman slaves are the ones I know most about. And while they had essentially no rights, there was no concept of them being inherently inferior - they simply had lost their freedom, whether selling it to cover debts or having it taken as spoils of war. They could not be identified visually, and if freed were the equal of any Roman. Roman slaves gradually acquired more rights as time went on. Some links: http://www.thetalkingdrum.com/wil.html http://www.spartacus.schoolnet.co.uk/USASpunishments.htm http://en.wikipedia.org/wiki/Treatment_of_slaves_in_the_United_States#Punishment_and_abuse http://en.wikipedia.org/wiki/Fugitive_slave Roman slaves: http://en.wikipedia.org/wiki/Slavery_in_ancient_Rome http://www.roman-colosseum.info/roman-life/slave-punishment.htm http://www.mlahanas.de/Greeks/LX/SlavesRomanEmpire.html

1DaFranker13y

Much thanks for all the extra data! That was some interesting reading.

[-]Doug_S.17y30

http://en.wikipedia.org/wiki/The_Iron_Dream

[-]billswift17y10

"The complex and detailed universes inevitably lead to utterly pointless arguments."

This is actually an argument for using fiction. Real situations are more complex and much, much more likely to result in peripheral arguments than fictional situations.

[-]frelkins17y-10

@billswift

"Nearly all people for thousands of years thought it was perfectly all right to keep slaves"

And many still do today - for example, Shari'a endorses slavery. Our Western values are far from universal and cannot be taken for granted.

[-]Z._M._Davis17y10

Tom: "Hmmm.. Maybe we should put together a play version of 3WC [...]"

That reminds me! Did anyone ever get a copy of the script to Yudkowski Returns? We could put on a benefit performance for SIAI!

ducks

[-]Eliezer Yudkowsky17y30

And perhaps the current slavers would change their minds if they read the right book, and perhaps not - more probably not, I think, without other changes as well. As I noted in the post text, the mirror neurons do have an off switch. It might take some abstract argument to turn them back on. Or it might take a "slave" rescuing their daughter in real life, instead of fiction. Maybe even that wouldn't do it. Maybe their and my reflective equilibria are so far apart that they can't be called by the same word "right".

Nonetheless - Uncl... (read more)

[-]HughRistik17y110

Tyrrell said:

Nor did I see anything in your preceding "rigorous" posts to establish that being modified fell in this range It appeared to be a moral assertion for which no argument was given.

Yeah. Eliezer, in your story, being modified just didn't seem bad enough to be obviously preferable to killing 15 billion people. This creates moral ambiguity that is great in a story, but not if you wanted to communicate a clear moral.

The way the story was presented, I was think "humanity without suffering, and having to eat non-sentient babies?... Is that really so bad to justify killing 15 billion people?" Now, as a reader of Overcoming Bias, I know that Value is Fragile, and that scaling up the human brain is a highly risky proposition that that the brain is not designed for. So, the end result of the Superhappy proposal would not be "Humans minus suffering, eating non-sentient babies." It would not be human at all.

Superhappies can't just surgically remove negative emotions and pain from our brains and leave everything else untouched. It would be likely the Superhappies would make a first pass and remove suffering, but the neurochemical changes would drive ... (read more)

[-]JamesAndrix17y00

billswift: But those peripheral argument will still be about things that in some sense matter, as opposed to say, midichlorians.

[-]Sideways217y20

HughRistik:

Speaking as a new reader of Overcoming Bias myself--I think that the sort of people who read this blog are more likely to miss how dangerous the Superhappies are, because we've considered ways that human suffering could be reduced or eliminated while still letting humans develop properly. Then, when people who already have ideas about how to reduce suffering read that the Superhappies want to eliminate suffering, they assume that the Superhappies' plans are the same as their own. (I'm not sure if this is a previously discussed and named bias, bu... (read more)

0johnlawrenceaspden13y

Isn't any rational agent a paperclip maximiser for something? I thought that was what 'rational' was supposed to mean round here.

3MugaSofer13y

No.

[-]Yvain217y110

Uncle Tom's Cabin is not a valid argument that slavery is wrong. "My mirror neurons make me sympathize with a person whose suffering is caused by Policy X" to "Policy X is immoral and must be stopped" is not a valid pattern of inference.

Consider a book about the life of a young girl who works in a sweatshop. She's plucked out of a carefree childhood, tyrannized and abused by greedy bosses, and eventually dies of work-related injuries incurred because it wasn't cost-effective to prevent them. I'm sure this book exists, though I haven't personally come across it. And I'm sure this book would provide just as emotionally compelling an argument for banning sweatshops as Uncle Tom's Cabin did for banning slavery.

But the sweatshop issue is a whole lot more complex than that, right? And the arguments in favor of sweatshops are more difficult to put into novel form, or less popular among the people who write novels, or simply not mentioned in that particular book, or all three.

The problem with fiction as evidence is that it's like the guy who say "It was negative thirty degrees last night, worst snowstorm in fifty years, so how come them liberals are still talking a... (read more)

[-]Anonymous4817y20

Not enough sex and explosions in 3WC? Are you joking?

Oh, and it would be easier to find someone make it into a good visual novel rather than a good movie.

[-]Robin_Hanson217y00

Yvian, I warned against granting near-thought virtues to fictional detail here. I doubt Uncle Tom's cabin would have persuaded many slave holders against slavery; I expect well-written well-recommended anti-slavery fiction more served to signal to readers where fashionable opinion was moving.

[-]Vladimir_Nesov17y10

@Yvain: Mathematical proof is a valid argument, even if it doesn't contain any information, what was true remains so.

Fiction isn't supposed to act as evidence, it's supposed to place you in a specific focus of attention, where you resolve your own questions for yourself, from evidence you already hold. It doesn't explicitly state abstractions which you are supposed to learn in order to master new thoughts, reinterpret old data, or bind existing morals. It invites you to invent abstractions on a given topic for yourself.

Of course, all the usual biases will ... (read more)

[-]Cabalamat217y00

Just on the off-chance, are there any OB readers who could get a good movie made?

If machinima counts as "a good movie" you might want to talk to Hugh Hancock (I've no idea if he reads OB, but based on his other interests he may well do).

[-]Robin_Hanson217y00

Eliezer, as I indicate in my new post, the issue isn't so much whether you the author judge that some fiction would help inform readers about morals, but whether typical readers can reasonably trust your judgment in such things, relative to the average propaganda content of authors writing apparently similar moral-quandary stories.

[-]TreeFrog17y00

I know that Tucker Max, whose movie I Hope They Serve Beer in Hell will be out sometime this year, has read this site since July of 2007 at least. He's actually how I discovered Overcoming Bias.

He's said numerous times that Eliezer would be absolutely fantastic if his posts weren't so ridiculously long and wandering at times.

He'd be a good person to talk to about making a movie (since his own was designed specifically to avoid that "dumbing-down process") and is probably going to make several tens of millions over the next few years.

[-]TGGP417y20

Why should we believe there are "moral truths"? And why are the rules so different with regard to physics? What other topics have a standard more like morality than physics?

I agree with Yvain. The mirror neuron argument was just shoddy. After acknowledging that the science didn't necessarily support your point about them, you then said that doesn't matter. If the truth of an argument is irrelevant, why bring it up at all? Doesn't such an argument falling back on "deeper truth" have the same weaknesses as the religious/mystical in their ... (read more)

[-]billswift17y00

"But those peripheral argument will still be about things that in some sense matter, as opposed to say, midichlorians."

Of course they are. That's why they distract so strongly from the central point.

Why would anyone argue about midi-chlorians? The "explanation" of Jedi powers in that movie detracted from the Star Wars universe.

[-]JamesAndrix17y10

Of course they are. That's why they distract so strongly from the central point.

Even if they distract more from the central point, theya re still real. They still have some potential relevance to the reader.

I think it would be a disservice to train someone up in the arts of rationality only to have most of their thoughts revolve around the facts of some fantasy universe.

Also, just because our universe is more detailed than fictional ones, doesn't mean that all of that detail has to be available to the reader. We can offer simplified descriptions of situations.

[-]michael_vassar317y10

Eliezer: It may be worth noting that SIAI just hired a new president FROM a branch of the film industry who has some familiarity with the sort of tax laws that can make indie movies a good investment even when expected value appears negative, and that SIAI's largest donor is the producer of an excellent movie about the marketing of cigarettes.

Other than that.

I agree with Kaj I really like Hugh's point I don't think 3WC or Dragon Tyrant work as movies. I don't know what Eliezer's got however WRT stories.

[-]Michael_Vassar517y00

Tree Frog: do you know Tucker and are you suggesting that I speak with him? That's basically my job after all.

[-]Norman_Noman17y00

I still don't see the actual practical benefit of suffering. I've lived a very sheltered life, physically and emotionally. I've never needed stitches, never had my heart broken, I've always been pretty low-key emotionally, and I don't feel like I'm missing anything.

Besides, what are we going to do NEXT time we run into a more advanced race of aliens? I suppose we can just keep blowing up starlines, but what happens if they get the jump on us, like the superhappies got the jump on the babyeaters? It seems like we need powerful allies much more than we need our precious aches and pains.

[-]V.G.17y10

Eliezer,

I am very much inclined to analyze your articles because you are indeed very enthusiastic about your theories, which is a rarity these days. On the link there’s a Wordle tag cloud picture of your article.

As you can see, there’s a lot of “argument(s)”,”moral”, “abstract”, “fiction’, and a somewhat humble “experience”.

To the point – fiction in the literary domain is often a method for implying moral concepts. But fiction is, above all, an invitation to imagine. There is a catch. We can imagine a setting, a world, a relationship. But we sometimes cann... (read more)

[-]Yvain217y20

@Robin: Thank you. Somehow I missed that post, and it was exactly what I was looking for.

@Vladimir Nesov: I agree with everything you said except for your statement that fiction is a valid argument, and your supporting analogy to mathematical proof.

Maybe the problem is the two different meanings of "valid argument". First, the formal meaning where a valid argument is one in which premises are arranged correctly to prove a conclusion eg mathematical proofs and Aristotelian syllogisms. Well-crafted policy arguments, cost-benefit analyses, and stati... (read more)

[-]Vladimir_Nesov17y00

Well, my point is that fiction isn't argument at all, it's a theme for musing on your own questions. It's sometimes useful to read even something you know to be wrong, on a theme interesting for you, written by a thoughtful author. You don't expect to move towards agreement, you know the stuff is wrong, but you can light sparks of your own insight off its pages. When given a mathematical proof, its correctness is for you to appraise. Fiction gives you your own thoughts, take them or leave them.

[-]TreeFrog17y00

Michael Vassar: I have no idea who you are, but I'll proceed on the assumption that the "job" you mention is one of representing Eliezer and/or the other OvercomingBias authors in some sort of business capacity.

I don't know Max on a personal level. We've talked a few times on his board and he might be dimly aware of my existence, but I make no claims as to what he will do if contacted by Eliezer or an agent of Eliezer's.

Serious discussions of potential OvercomingBias projects/movies and whatnot should be sent to Max's assistant, Ian Claudius - ian.claudius(AT)gmail.com. The Rudius people are smart and good content creators (multiple book contracts, one soon to be hit movie and stuff I actually like).

[-]David_Ellis17y00

Whatever the potential abuses, I think fiction has a valid role to play in dealing with philosophical questions.

One example that comes to mind: I've observed in several discussions on the problem of evil that theists tend to want to discuss the matter in the most abstract possible terms. It seems to be easier to swallow a theodicy when you don't have the unpleasant facts of extreme suffering vividly in mind.

In the case of the POE, fiction can serve to cut through rationalizations in a way argumentation alone can never hope to do.

[-]Kyle317y20

Eliezer,

Took a bit after reading your babyeater pieces to get my thoughts in order, but my general picture is that you're mis-representing the human condition, and that the whole of the story relies upon that misrepresentation. This is humanity you're talking about. While the readership of OB may match your profile decently, the human species, even extended into a moderately improved state does not hold the value-set you represent.

Child-love/protection is (a) proximity-focused, (b) stronger than abstract value of reciprocity. Folks are much better at

... (read more)

[-]nolrai17y00

I think I have a co-operation instinct that is pushing me towards the supper happy future.

It feels better, but is probably not what I would do In real life. or I am more different then others then I give credit for.

[-]lessdazed14y10

Telling people about the logical fallacy of generalization from fictional evidence doesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen.

That danger can be minimized in an elegant way: by referring people to a story that shows one shouldn't generalize from fictional evidence.

The more likely they are to inappropriately take the messages of stories to heart, the more they will be to take to heart the message not to do that.

[-]orthonormal14y30

Can someone fix the formatting on this article?

3Alicorn14y

I can't see what's causing the problem in either the WYSIWYG editor or the HTML one, both of which show spaces around the words that are cramped in the published version. I'm confused.

3JGWeissman14y

It looks like the indented quoted sections are implemented with "div" elements, which I recall have somehow caused spacing problems around other tags in the same article. It would probably help to change those sections to use that LW standard quoting format.

0Swimmer963 (Miranda Dixon-Luinenburg)14y

That's happened to me before. I just put in extra spaces, so that when it was published it appeared normal. Still don't know what caused it.

2Vladimir_Nesov14y

Fixed. (It's more reliable to PM me, since I don't read all comments; it just so happens that you are among the people whose comments I'm subscribed to.)

0Swimmer963 (Miranda Dixon-Luinenburg)14y

You can subscribe to people's comments? That's...pretty nifty! How do you go about doing it?

2AdeleneDawner14y

Go to the person's user page, and click 'subscribe to RSS feed' in the sidebar. This works on most pages on LW, in fact.

[-]A1987dM12y10

I actually did try to send a long message consisting only of

DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK

Reading that was quite effective in pushing me towards Torture.

(I'm not being sarcastic.)

Moderation Log

More from Eliezer Yudkowsky

Curated and popular this week

82Comments

A comment by Anonymous on Three Worlds Collide:

After reading this story I feel myself agreeing with Eliezer more on his views and that seems to be a sign of manipulation and not of rationality.

Philosophy expressed in form of fiction seems to have a very strong effect on people - even if the fiction isn't very good (ref. Ayn Rand).

Robin has similar qualms:

Since people are inconsistent but reluctant to admit that fact, their moral beliefs can be influenced by which moral dilemmas they consider in what order, especially when written by a good writer. I expect Eliezer chose his dilemmas in order to move readers toward his preferred moral beliefs, but why should I expect those are better moral beliefs than those of all the other authors of fictional moral dilemmas?

If I'm going to read a literature that might influence my moral beliefs, I'd rather read professional philosophers and other academics making more explicit arguments.

I replied that I had taken considerable pains to set out the explicit arguments before daring to publish the story. And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light. (The opposing viewpoint is the counterpart of the villain; you want it to look as reasonable as possible for purposes of dramatic conflict, the same principle whereby Frodo confronts the Dark Lord Sauron rather than a cockroach.)

Robin didn't find this convincing:

I don't think readers should much let down their guard against communication modes where sneaky persuasion is more feasible simply because the author has made some more explicit arguments elsewhere... Academic philosophy offers exemplary formats and styles for low-sneak ways to argue about values.

I think that this understates the power and utility of fiction. I once read a book that was called something like "How to Read" (no, not "How to Read a Book") which said that nonfiction was about communicating knowledge, while fiction was about communicating experience.

If I want to communicate something about the experience of being a rationalist, I can best do it by writing a short story with a rationalist character. Not only would identical abstract statements about proper responses have less impact, they wouldn't even communicate the same thought.

From The Failures of Eld Science:

"...Work expands to fill the time allotted, as the saying goes. But people can think important thoughts in far less than thirty years, if they expect speed of themselves." Jeffreyssai suddenly slammed down a hand on the arm of Brennan's chair. "How long do you have to dodge a thrown knife?"

"Very little time, sensei!"

"Less than a second! Two opponents are attacking you! How long do you have to guess who's more dangerous?"

"Less than a second, sensei!"

"The two opponents have split up and are attacking two of your girlfriends! How long do you have to decide which one you truly love?"

"Less than a second, sensei!"

"A new argument shows your precious theory is flawed! How long does it take you to change your mind?"

"Less than a second, sensei!"

"WRONG! DON'T GIVE ME THE WRONG ANSWER JUST BECAUSE IT FITS A CONVENIENT PATTERN AND I SEEM TO EXPECT IT OF YOU! How long does it really take, Brennan?"

Sweat was forming on Brennan's back, but he stopped and actually thought about it -

"ANSWER, BRENNAN!"

"No sensei! I'm not finished thinking sensei! An answer would be premature! Sensei!"

"Very good! Continue! But don't take thirty years!"

This is an experience about how to avoid completing the pattern when the pattern happens to be blatantly wrong, and how to think quickly without thinking too quickly.

Forget the question of whether you can write the equivalent abstract argument that communicates the same thought in less space. Can you do it at all? Is there any series of abstract arguments that creates the same learning experience in the reader? Entering a series of believed propositions into your belief pool is not the same as feeling yourself in someone else's shoes, and reacting to the experience, and forming an experiential skill-memory of how to do it next time.

And it seems to me that to communicate experience is a valid form of moral argument as well.

Uncle Tom's Cabin was not just a historically powerful argument against slavery, it was a valid argument against slavery. If human beings were constructed without mirror neurons, if we didn't hurt when we see a nonenemy hurting, then we would exist in the reference frame of a different morality, and we would decide what to do by asking a different question, "What should* we do?" Without that ability to sympathize, we might think that it was perfectly all right* to keep slaves. (See Inseparably Right and No License To Be Human.)

Putting someone into the shoes of a slave and letting their mirror neurons feel the suffering of a husband separated from a wife, a mother separated from a child, a man whipped for refusing to whip a fellow slave - it's not just persuasive, it's valid. It fires the mirror neurons that physically implement that part of our moral frame.

I'm sure many have turned against slavery without reading Uncle Tom's Cabin - maybe even due to purely abstract arguments, without ever seeing the carving "Am I Not a Man and a Brother?" But for some people, or for a not-much-different intelligent species, reading Uncle Tom's Cabin might be the only argument that can turn you against slavery. Any amount of abstract argument that didn't fire the experiential mirror neurons, would not activate the part of your implicit should-function that disliked slavery. You would just seem to be making a good profit on something you owned.

Can fiction be abused? Of course. Suppose that blacks had no subjective experiences. Then Uncle Tom's Cabin would have been a lie in a deeper sense than being fictional, and anyone moved by it would have been deceived.

Or to give a more subtle case not involving a direct "lie" of this sort: On the SL4 mailing list, Stuart Armstrong posted an argument against TORTURE in the infamous Torture vs. Dust Specks debate, consisting of a short story describing the fate of the person to be tortured. My reply was that the appropriate counterargument would be 3^^^3 stories about someone getting a dust speck in their eye. I actually did try to send a long message consisting only of

DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK
DUST SPECK

for a thousand lines or so, but the mailing software stopped it. (Ideally, I should have created a webpage using Javascript and bignums, that, if run on a sufficiently large computer, would print out exactly 3^^^3 copies of a brief story about someone getting a dust speck in their eye. It probably would have been the world's longest finite webpage. Alas, I lack time for many of my good ideas.)

Then there's the sort of standard polemic used in e.g. Atlas Shrugged (as well as many less famous pieces of science fiction) in which Your Beliefs are put into the minds of strong empowered noble heroes, and the Opposing Beliefs are put into the mouths of evil and contemptible villains, and then the consequences of Their Way are depicted as uniformly disastrous while Your Way offers butterflies and apple pie. That's not even subtle, but it works on people predisposed to hear the message.

But to entirely turn your back on fiction is, I think, taking it too far. Abstract argument can be abused too. In fact, I would say that abstract argument is if anything easier to abuse because it has more degrees of freedom. Which is easier, to say "Slavery is good for the slave", or to write a believable story about slavery benefiting the slave? You can do both, but the second is at least more difficult; your brain is more likely to notice the non-sequiturs when they're played out as a written experience.

Stories may not get us completely into Near mode, but they get us closer into Near mode than abstract argument. If it's words on paper, you can end up believing that you ought to do just about anything. If you're in the shoes of a character encountering the experience, your reactions may be harder to twist.

Contrast a verbal argument against the verbal belief that "non-Catholics go to Hell"; versus reading a story about a good and decent person, who happens to be a Protestant, and dies trying to save a child's life, who is condemned to hell and has molten lead poured down her throat; versus the South Park episode where a crowd of newly dead souls is at the entrance to hell, and the Devil says, "Sorry, it was the Mormons" and everyone goes "Awwwww..."

Yes, abstraction done right can keep you going where concrete visualization breaks down - the torture vs. dust specks thing being an archetypal example; you can't actually visualize that many dust specks, but if you try to choose SPECKS you'll end up with circular preferences. But so far as I can organize my metaethics, the ground level of morality lies in our preferences over particular, concrete situations - and when these can be comprehended as concrete images at all, it's best to visualize them as concretely as possible. Unless we know specifically where the concrete image is going wrong, and have to apply an abstract correction. The moral abstraction is built on top of the ground level.

I am also, of course, worried about the idea that stories aren't "respectable" because they don't look sufficiently solemn and dull; or the idea that something isn't "respectable" if can be understood by a mere popular audience. Yes, there are technical fields that are genuinely impossible to explain to your grandmother in an hour; but ceteris paribus, people who can write at a more popular level without distorting technical reality are performing a huge service to that field. I've heard that Carl Sagan was held in some disrepute by his peers for the crime of speaking to the general public. If true, this is merely stupid.

Explaining things is hard. Explainers need every tool they can get their hands on - as a matter of public interest.

And in moral philosophy - well, I suppose it could be the case that moral philosophers have discovered moral truths that are deductive consequences of most humans' moral frames, but which are so difficult and technical that they simply can't be explained to a popular audience within a one-hour lecture. But it would be a tad more suspicious than the corresponding case in, say, physics.

I realize that I speak as someone who does a lot of popularizing, but even so - fiction ought to be a respectable form of moral argument. And a respectable way of communicating experiences, in particular the experience of applying certain types of thinking skills.

I've always been of two minds about publishing longer fiction pieces about the future and its consequences. Not so much because of the potential for abuse, but because even when not abused, fiction can still bypass critical faculties and end up poured directly into the brains of at least some readers. Telling people about the logical fallacy of generalization from fictional evidence doesn't make it go away; people may just go on generalizing from the story as though they had actually seen it happen. And you simply can't have a story that's a rational projection; it's not just a matter of plot, it's a matter of the story needing to be specific, rather than depicting a state of epistemic uncertainty.

But to make shorter philosophical points? Sure.

And... oh, what the hell. Just on the off-chance, are there any OB readers who could get a good movie made? Either outside Hollywood, or able to bypass the usual dumbing-down process that creates a money-losing flop? The probabilities are infinitesimal, I know, but I thought I'd check.