The most dangerous dark side meme I can think of is the idea of sinful thoughts: that questioning one's faith is itself a sin even if not acted upon. A close second is "don't try to argue with the devil -- he has more experience at it than you".
Especially when it's explicitly enforced, a la death penalty for leaving Islam in Islamic countries.
Not all who wear robes are either Jedi or fakes What do you mean by "wear robes"? Could we move away from references to fictional stories?
Are you trying to argue against the use of metaphor for argument? The fact that Star Wars is a fiction doesn't make analogies made with its concepts wrong.
To clarify the phrase that you take issue with, "robes" from what I can gather signifies memetic authority, like scientists or priests or marketers who have dominion over a region of thought patterns - as the Jedi wield the Force.
Eliezer,
I agree with you what regards people deceiving themselves. But I disagree regarding people that are deceiving others with purpose. Some of these people can be very smart and know very well what they are doing and on what biases they are playing. They have elevated the art of deception to a science, ohhh yes, read marketing books as an example. Otherwise a superintelligence would become stupid in the process of lying to the human operator with the intention to get out of the box.
-faith: i.e. unconditional belief is good. It's like loyalty. Questioning beliefs is like betrayal. -The saying "Stick to your guns.": Changing your mind is like diserting your post in a war. Sticking to a belief is like being a heroic soldier. -The faithfull: i.e. us, we are the best, god is on our side. -the infedels: i.e. them, sinners, barely human, or not even. -God: Infenetly powerful alpha male. Treat him as such with all the implications... -The devil and his agents: They are always trying to seduce you to sin. Any doubt is evedence the devil is seducing you to sin and suceeding. Anyone opposed to your beliefs is cooperating with/being influenced by the devil. -Assasination fatwas: Whacking people who are anti-Islam is the will of Allah. -a sexually satisfying lifestyle is bad: This makes people more angsty(especially young men). This angst is your fault and it's sin. To be less angsty you should be less sinful ergo fight your sexual urges. And so the cycle of desire, guilt, angst and confusion continues. -no masturbation: see above. -you are born in debt to Jesus because he died for your sins 2000 years ago. That's all I could think of right now.
The endorsement of information cascades: claiming that X is indisputably true in the name of philosophical majoritarianism, and thus biasing research and statements to foster belief in X is desirable as a way to foster true beliefs (where the majority only exists because of such biased efforts).
Just to be clear, I'm not looking for random false beliefs defended by Dark Side epistemology, I'm looking for Dark Side epistemology itself - the Generic Defenses of Fail.
Roland, these are the Sith masters.
In general, beliefs require evidence.
In general? Which beliefs don't?
Think of what it would take to deny evolution or heliocentrism
Or what it would take to prove that the Moon doesn't exist.
As for listing common memes that were spawned by the Dark Side - would you care to take a stab at it, dear readers?
Cultural relativity. Such-and-such is unconstitutional. The founding fathers never intended... (various appeals to stick to the founding fathers original vision) Be reasonable (moderate) Show respect for your elders It's my private property _ is human natur...
"'In general, beliefs require evidence.' In general? Which beliefs don't?"
This is a language problem. "In general" or "generally" to a scientist/mathematician/engineer means "always," whereas in everyday speech it means "sometimes."
For example I could tell you that a fence with 2 sections has 3 posts ( I=I=I ), or I could tell you that "in general" a fence with N sections has N+1 posts.
“How many posts does a fence have, if you call the tree a post?”
"We need to switch to alternative energies such as wind, solar, and tidal. The poor are lazy ... Animal rights"
I don't think these fit. Regardless of whether you agree with them, they are specific assertions, not general claims about reasoning with consistently anti-epistemological effects.
Everyone has a right to their own opinion. When you think about it, where was that proverb generated?
In the words of the great sage Emo Phillips, "I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this."
I thought of some more. -there is a destiny/Gods plan/reason for everything: i.e. some powerful force is making things the way they are and it all makes sense(in human terms, not cold heartless math). That means you are safe but don't fight the status quo. -everything is connected with "energy"(mystically): you or special/chosen people might be ably to tap into this "energy". You might glean information you normally shouldn't have or gain some kind of special powers. -Scientists/professionals/experts are "elitists". -Mystery is good: It makes life worth while. Appreciating it makes us human. As opposed to destroying it being good. That's it for now.
I'm looking for Dark Side epistemology itself - the Generic Defenses of Fail.
Relax. It will be over soon.
We're past that now.
X is supernatural.
X is natural.
You're correct, but it will make people uncomfortable.
You're smart. You should go to college.
I've had forms of this said to me; it basically means "I'm losing the debate because you personally are smart, not because I'm wrong. Whichever authority I listen to in order to reinforce my existing beliefs would surely crush all your arguments. So stop assailing me with logic..."
It's Dark Side because it surrenders personal understanding to authority, and treats it as a default epistemological position.
It's Dark Side because it surrenders personal understanding to authority, and treats it as a default epistemological position.
Dark side or not it is quite often valid. People who do not trust their ability to filter bullshit from knowledge should not defer to whatever powerful debater attempts to influence them.
It is no error to assign a low value to p(the conclusion expressed is valid | I find the argument convincing).
I'm pretty confident that ""Everyone has a right to their own opinion." was generated by people trying to protect themselves from people who were trying to protect themselves from the truth.
We really need some talk about what the consequences of an AI with access to its own source code and self-protecting beliefs would be.
I'm looking for Dark Side epistemology itself - the Generic Defenses of Fail.
In that case - association, essentialism, popularity, the scientific method, magic, and what I'll call Past-ism.
Wait a second - the scientific method? How? It may not be the most efficient way to get the truth, and it may not take into account Baye's theorem that could speed it up, but I don't see how the scientific method is epistemologically (is that a word?) wrong.
We are missing something. Humans are ultimatly driven by emotions. We should look for which emotions beliefs tap into in order to understand why people seek or avoid certain beliefs.
A particular flavor of "if it ain't broke, don't fix it" that points to established traditions as "having worked for ages". Playing off the fear of the unknown? The meme of traditions in general adds weight to many of these.
I second "cultural relativity" as being an extension of "everyone having a right to their opinion", but in both cases point to them as also being tools to find things in one's own life that are arbitrary and in need of evaluation on a more objective basis.
Isn't the scientific method a servant of the Light Side, even if it is occasionally a little misguided?
@Eliezer: Roland, these are the Sith masters.
Ok, got your point. One thing I worry though is how much those movie analogies end up inducing biases in you and others.
@Eliezer:
To drive home my earlier point. The whole idea of jedis vs. siths reflects a Manichaeistic worldview(good vs. bad). Isn't this a simplification?
Isn't the scientific method a servant of the Light Side, even if it is occasionally a little misguided?
Too restrictive. Science is not synonymous with the hypothetico-deductive method, and nor is there any sort of thing called the "scientific method" from which scientists draw their authority on a subject. Neither is it a historically accurate description of how science has done its work. Read up on Feyerabend.
Science is inherently structureless and chaotic. It's whatever works.
Eliezer writes, "In general, beliefs require evidence."
To which Peter replies, "In general? Which beliefs don't?"
Normative beliefs (beliefs about what should be) don't, IMHO. What would count as evidence for or against a normative belief?
How about "Comparing Apples and Oranges," or "How Dare you Compare," a misrepresentation of the scope of analogies. For a recent example, see the response to John Lewis's drawing an analogy between certain aspects of the McCain campaign and those of George Wallace -- the response is not a consideration of the scope and aptness of the analogy but a rejection that any analogy at all can be drawn between two subjects when one is so generally recognized to be Evil. The McCain campaign does not attempt to differentiate the aspects under analogy (rhetoric and its potential for the fomentation of violence) from those of Wallace, but rather condemns the idea that the analogy can be considered at all. Under the epistemology of Fail, any difference between two subjects of comparison is enough to reject its validity, regardless the relevance of the distinction to the actual comparison being drawn. See also: Godwin's Law.
Some self-entitled males like to use this one, particularly in defense of the notion that one has in inviolate right to make sexual advances toward other people regardless of circumstance or outward sign. Sooner or later, after demonstrating how each of the...
Normative beliefs (beliefs about what should be) don't [require evidence], IMHO. What would count as evidence for or against a normative belief?
That's correct if you don't consider pure reason to be evidence - but I consider it to be so. So morality and ethics and all these normative things are, in fact, based on evidence - although it is a mix of abstract evidence (reason) with concrete evidence (empirical data). If you base your morality, or any normative theory (how the world should be) on anything other than how things actually are (including mathematics), you necessarily have to invoke ascribe some supernatural property onto it
One giant category of dark side reasoning looks like "That idea is _" Where the idea is an "is" (not a "should") and _ is any negative affect word with a meaning other than "untrue".
Examples include {unpatriotic, communist, capitalist, liberal, conservative, provincial, any-demonym-goes-here, cultish, religious, atheistic, sinful, evil, dangerous, repugnant, elitist, condescending, out-of-touch, politically incorrect, offensive, argumentative, hateful, cowardly, fool-hardy, inappropriate, indecent, unsettling, lewd, silly, idiotic, new-fangled, old-fashioned, staid, dead, uncool, too simple, too complicated} and many more.
Important note: The exception to this rule is if the speaker could goes on to show how _ is evidence about the truth of the proposition. If you can say why something is idiotic, that's fine. A seasoned scientist has the right to say "that theory looks too complicated" if the they have many examples of surprisingly simple theories explaining things well, but a creationist doesn't earn the right to accuse the theory of evolution of being "too complicated," until they explain what whatever it is they mean...
Saying 'There is lots of evidence for it' When in fact there is little to none. I guess the epistemology is 'It is ok to believe something if you believe there is evidence to support it.'
Creationists are told the fossil record supports X and Y, and they run with it.
The concept of different epistemological magisteria. E gave an example of it in this post (and also in the post about scientists outside the laboratory), but his example is just the tip of the iceberg. This failure of rationality doesn't manifest itself explicitly most of the time, but is engaged in implicitly by almost everybody that I know that isn't into hardcore rationality.
It's definitely engaged in by people who are into, or at least cheer for, science and (traditional) rationality and/or philosophy. It's the double standard between what epistemologi...
A few general schemas:
"True for", as in, "That may be true for you, but not for me. We each choose our own truths."
"I feel that X." Every sentence of this form is false, because X is an assertion about the world, not a feeling. Someone saying "I feel that X" in fact believes X, but calling it a feeling instead of a belief protects it from refutation. Try replying "No you don't", and watch the explosion. "How dare you try to tell me what I'm feeling!"
Write obscurely.
Never explicitly state your beliefs. Hint at them in terms that the faithful will pick up and applaud, but which give nothing for the enemy to attack. Attack the enemy by stating their beliefs in terms that the faithful will boo, while giving the enemy nothing to dispute.
Ignore the entire machinery of rationality. Treat all human interaction as nothing more than social grooming or status games in a tribe of apes.
Daniel: A close second is "don't try to argue with the devil -- he has more experience at it than you".
Would you still disagree with that one if "the devil" was replaced by "a strong AI"?
How about the notion of an insult as a first-order offence? "Don't insult God/Our Nation/The People/etc.". It is an explicit emotional fortress that reason cannot by definition scale. When it goes near there, all the 'intelligence defeating itself' mechanisms come into play. We take the fortress as our starting argument and start to think backwards until our agitated emotions are satisfied by our half-reasonable but beautiful explanation of why the fortress is safe and why what caused us to doubt it is either not so or can be explained some other way. Ergo, one step deeper into dark epistemology.
Would you still disagree with that one if "the devil" was replaced by "a strong AI"?
Yes. Suffice it to say I don't think I'd be a very reliable gatekeeper :-).
(Conversely, I don't even think the AI's job in the box experiment is even hard, much less impossible. Last week, I posted a $15 offer to play the AI in a run of the experiment, but my post disappeared somehow.)
I'm in strong agreement with Peter's examples above. I would generalize by saying that the epistemic "dark side" tends to arise whenever there's an implicit discounting of the importance of increasing context. In other words, whenever, for the sake of expediency, "the truth", "the right", "the good". etc., is treated categorically rather than contextually (or equivalently, as if the context were fixed or fully specified.)
Too restrictive. Science is not synonymous with the hypothetico-deductive method, and nor is there any sort of thing called the "scientific method" from which scientists draw their authority on a subject. Neither is it a historically accurate description of how science has done its work. Read up on Feyerabend. Science is inherently structureless and chaotic. It's whatever works.
See, now there's a prime example of corrupted reasoning right there. Science is carefully structured chaos, ordered according to certain fundamental principles. Meeti...
Roland: The whole idea of jedis vs. siths reflects a Manichaeistic worldview(good vs. bad).
That was part of my point - that, in this one facet of human endeavor, and in modern times rather than ancient ones, it's remarkable the extent to which an actual Light Side Epistemology and Dark Side Epistemology have developed. Like the sort of contrast that naive people draw between Their Party and the Other Party, only in real life.
There's a huge conspiracy covering it up
Well, that's just what one of the Bad Guys would say, isn't it?
Why should I have to justify myself to you?
Oh, you with your book-learning, you think you're smarter than me?
They said that to Einstein and Galileo!
That's a very interesting question, let me show you the entire library that's been written about it (where if there were a satisfactory answer it would be shortish)
How can you be so sure?
Marcello, I think your list generalizes too much. I see three main types of words on the list. The first type indicates in-group out-group distinction and seems pretty poisonous to me. The second are ad hominem arguments which are dangerous, but do apply sometimes. And then there are a few like "too complicated." You call those "negative affect words"? Surely it is better to say "that is too complicated to be true" than to say simply "that is not true"?
-You can't prove I'm wrong!
-Well, I'm an optimist.
-Millions of people believe it, how can they all be wrong?
-You're relying too much on cold rationality.
-How can you possibly reduce all the beauty in the world to a bunch of equations?
Douglas says: """ And then there are a few like "too complicated." You call those "negative affect words"? Surely it is better to say "that is too complicated to be true" than to say simply "that is not true"? """
Well, yes, but that's only when whatever you mean by complicated has something to do with being true. Some people though, just use the phrase "too complicated" just so they can avoid thinking about an idea, and, in that context it really is an empty negative-affect ph...
That was part of my point - that, in this one facet of human endeavor, and in modern times rather than ancient ones, it's remarkable the extent to which an actual Light Side Epistemology and Dark Side Epistemology have developed. Like the sort of contrast that naive people draw between Their Party and the Other Party, only in real life.
That sounds a lot more like you're being subject to the same bias. "Some people have this view, even though reality is more complex, but what's amazing is that in a subject area I care a lot about, that's what's there.&...
If you once tell a lie, the truth is ever after your enemy.
That isn't true.
I've told lies when I was a kid. If I got caught I gave up rather than doing an epistomological attack.
Richard Kennaway: "I feel that X." Every sentence of this form is false, because X is an assertion about the world, not a feeling. Someone saying "I feel that X" in fact believes X, but calling it a feeling instead of a belief protects it from refutation. Try replying "No you don't", and watch the explosion. "How dare you try to tell me what I'm f...
Nancy Lebovitz: If I say I feel something, I'm talking about an emotion.
That prohibits you from saying "I feel that X". No emotion is spoken of in saying "I feel that the Riemann hypothesis is true", or "I feel that a sequel to The Hobbit should never be made", or "I feel that there is no God but Jaynes and Eliezer (may he live forever) is His prophet", or in any other sentence of that form. "I feel" and "that X" cannot be put together and make a sensible sentence.
If someone finds themselves about...
I believe that there are circumstances in which you can say "I feel that X". What that could rationally mean is that you yourself recognize that you do not have enough evidence or knowledge to justify a belief about X vs. not-X, but that without evidence you lean toward X because you like that alternative. You are admitting ignorance on the subject. Ideally, this would then also imply an openness with regard to forming a belief about X or not-X given some evidence -- that recognition that all you have is a feeling about it means a very weak attachment to the idea of X.
PhilB
Caledonian: What fundamental principles? As far as I can tell the only fundamental principle is that it has to work. But I'm open to counterexamples, if you are.
The recognition of what 'working' is, and the tools that have been found useful in reaching that state, is what constitutes the scientific method.
The scientific method is actually pretty specific - and it is not a set of tools. There is no systematic method of advancing science, no set of rules/tools which are exclusively the means to attaining scientific knowledge.
Scientists do not concern themsel...
If you once tell a lie, the truth is ever after your enemy.
I have discussed the notion that lies are contagious. If you pick up a pebble from the driveway, and tell a geologist that you found it on a beach—well, do you know what a geologist knows about rocks? I don’t. But I can suspect that a water-worn pebble wouldn’t look like a droplet of frozen lava from a volcanic eruption. Do you know where the pebble in your driveway really came from? Things bear the marks of their places in a lawful universe; in that web, a lie is out of place.1
What sounds like an arbitrary truth to one mind—one that could easily be replaced by a plausible lie—might be nailed down by a dozen linkages to the eyes of greater knowledge. To a creationist, the idea that life was shaped by “intelligent design” instead of “natural selection” might sound like a sports team to cheer for. To a biologist, plausibly arguing that an organism was intelligently designed would require lying about almost every facet of the organism. To plausibly argue that “humans” were intelligently designed, you’d have to lie about the design of the human retina, the architecture of the human brain, the proteins bound together by weak van der Waals forces instead of strong covalent bonds . . .
Or you could just lie about evolutionary theory, which is the path taken by most creationists. Instead of lying about the connected nodes in the network, they lie about the general laws governing the links.
And then to cover that up, they lie about the rules of science—like what it means to call something a “theory,” or what it means for a scientist to say that they are not absolutely certain.
So they pass from lying about specific facts, to lying about general laws, to lying about the rules of reasoning. To lie about whether humans evolved, you must lie about evolution; and then you have to lie about the rules of science that constrain our understanding of evolution.
But how else? Just as a human would be out of place in a community of actually intelligently designed life forms, and you have to lie about the rules of evolution to make it appear otherwise, so too beliefs about creationism are themselves out of place in science—you wouldn’t find them in a well-ordered mind any more than you’d find palm trees growing on a glacier. And so you have to disrupt the barriers that would forbid them.
Which brings us to the case of self-deception.
A single lie you tell yourself may seem plausible enough, when you don’t know any of the rules governing thoughts, or even that there are rules; and the choice seems as arbitrary as choosing a flavor of ice cream, as isolated as a pebble on the shore . . .
. . . but then someone calls you on your belief, using the rules of reasoning that they’ve learned. They say, “Where’s your evidence?”
And you say, “What? Why do I need evidence?”
So they say, “In general, beliefs require evidence.”
This argument, clearly, is a soldier fighting on the other side, which you must defeat. So you say: “I disagree! Not all beliefs require evidence. In particular, beliefs about dragons don’t require evidence. When it comes to dragons, you’re allowed to believe anything you like. So I don’t need evidence to believe there’s a dragon in my garage.”
And the one says, “Eh? You can’t just exclude dragons like that. There’s a reason for the rule that beliefs require evidence. To draw a correct map of the city, you have to walk through the streets and make lines on paper that correspond to what you see. That’s not an arbitrary legal requirement—if you sit in your living room and draw lines on the paper at random, the map’s going to be wrong. With extremely high probability. That’s as true of a map of a dragon as it is of anything.”
So now this, the explanation of why beliefs require evidence, is also an opposing soldier. So you say: “Wrong with extremely high probability? Then there’s still a chance, right? I don’t have to believe if it’s not absolutely certain.”
Or maybe you even begin to suspect, yourself, that “beliefs require evidence.” But this threatens a lie you hold precious; so you reject the dawn inside you, push the Sun back under the horizon.
Or you’ve previously heard the proverb “beliefs require evidence,” and it sounded wise enough, and you endorsed it in public. But it never quite occurred to you, until someone else brought it to your attention, that this proverb could apply to your belief that there’s a dragon in your garage. So you think fast and say, “The dragon is in a separate magisterium.”
Having false beliefs isn’t a good thing, but it doesn’t have to be permanently crippling—if, when you discover your mistake, you get over it. The dangerous thing is to have a false belief that you believe should be protected as a belief—a belief-in-belief, whether or not accompanied by actual belief.
A single Lie That Must Be Protected can block someone’s progress into advanced rationality. No, it’s not harmless fun.
Just as the world itself is more tangled by far than it appears on the surface, so too there are stricter rules of reasoning, constraining belief more strongly, than the untrained would suspect. The world is woven tightly, governed by general laws, and so are rational beliefs.
Think of what it would take to deny evolution or heliocentrism—all the connected truths and governing laws you wouldn’t be allowed to know. Then you can imagine how a single act of self-deception can block off the whole meta level of truth-seeking, once your mind begins to be threatened by seeing the connections. Forbidding all the intermediate and higher levels of the rationalist’s Art. Creating, in its stead, a vast complex of anti-law, rules of anti-thought, general justifications for believing the untrue.
Steven Kaas said, “Promoting less than maximally accurate beliefs is an act of sabotage. Don’t do it to anyone unless you’d also slash their tires.” Giving someone a false belief to protect—convincing them that the belief itself must be defended from any thought that seems to threaten it—well, you shouldn’t do that to someone unless you’d also give them a frontal lobotomy.
Once you tell a lie, the truth is your enemy; and every truth connected to that truth, and every ally of truth in general; all of these you must oppose, to protect the lie. Whether you’re lying to others, or to yourself.
You have to deny that beliefs require evidence, and then you have to deny that maps should reflect territories, and then you have to deny that truth is a good thing . . .
Thus comes into being the Dark Side.
I worry that people aren’t aware of it, or aren’t sufficiently wary—that as we wander through our human world, we can expect to encounter systematically bad epistemology.
The “how to think” memes floating around, the cached thoughts of Deep Wisdom—some of it will be good advice devised by rationalists. But other notions were invented to protect a lie or self-deception: spawned from the Dark Side.
“Everyone has a right to their own opinion.” When you think about it, where was that proverb generated? Is it something that someone would say in the course of protecting a truth, or in the course of protecting from the truth? But people don’t perk up and say, “Aha! I sense the presence of the Dark Side!” As far as I can tell, it’s not widely realized that the Dark Side is out there.
But how else? Whether you’re deceiving others, or just yourself, the Lie That Must Be Protected will propagate recursively through the network of empirical causality, and the network of general empirical rules, and the rules of reasoning themselves, and the understanding behind those rules. If there is good epistemology in the world, and also lies or self-deceptions that people are trying to protect, then there will come into existence bad epistemology to counter the good. We could hardly expect, in this world, to find the Light Side without the Dark Side; there is the Sun, and that which shrinks away and generates a cloaking Shadow.
Mind you, these are not necessarily evil people. The vast majority who go about repeating the Deep Wisdom are more duped than duplicitous, more self-deceived than deceiving. I think.
And it’s surely not my intent to offer you a Fully General Counterargument, so that whenever someone offers you some epistemology you don’t like, you say: “Oh, someone on the Dark Side made that up.” It’s one of the rules of the Light Side that you have to refute the proposition for itself, not by accusing its inventor of bad intentions.
But the Dark Side is out there. Fear is the path that leads to it, and one betrayal can turn you. Not all who wear robes are either Jedi or fakes; there are also the Sith Lords, masters and unwitting apprentices. Be warned; be wary.
As for listing common memes that were spawned by the Dark Side—not random false beliefs, mind you, but bad epistemology, the Generic Defenses of Fail—well, would you care to take a stab at it, dear readers?
1Actually, a geologist in the comments says that most pebbles in driveways are taken from beaches, so they couldn’t tell the difference between a driveway pebble and a beach pebble, but they could tell the difference between a mountain pebble and a driveway/beach pebble (http://lesswrong.com/lw/uy/dark_side_epistemology/4xbv). Case in point . . .