Uncritical Supercriticality

Followup toResist the Happy Death Spiral

Every now and then, you see people arguing over whether atheism is a "religion".  As I touched on in Purpose and Pragmatism, arguing over the meaning of a word nearly always means that you've lost track of the original question.  How might this argument arise to begin with?

An atheist is holding forth, blaming "religion" for the Inquisition, the Crusades, and various conflicts with or within Islam.  The religious one may reply, "But atheism is also a religion, because you also have beliefs about God; you believe God doesn't exist."  Then the atheist answers, "If atheism is a religion, then not collecting stamps is a hobby," and the argument begins.

Or the one may reply, "But horrors just as great were inflicted by Stalin, who was an atheist, and who suppressed churches in the name of atheism; therefore you are wrong to blame the violence on religion."  Now the atheist may be tempted to reply "No true Scotsman", saying, "Stalin's religion was Communism."  The religious one answers "If Communism is a religion, then Star Wars fandom is a government," and the argument begins.

Should a "religious" person be defined as someone who has a definite opinion about the existence of at least one God, e.g., assigning a probability lower than 10% or higher than 90% to the existence of Zeus?  Or should a "religious" person be defined as someone who has a positive opinion, say a probability higher than 90%, for the existence of at least one God?  In the former case, Stalin was "religious"; in the latter case, Stalin was "not religious".

But this is exactly the wrong way to look at the problem.  What you really want to know—what the argument was originally about—is why, at certain points in human history, large groups of people were slaughtered and tortured, ostensibly in the name of an idea.  Redefining a word won't change the facts of history one way or the other.

Communism was a complex catastrophe, and there may be no single why, no single critical link in the chain of causality.  But if I had to suggest an ur-mistake, it would be... well, I'll let God say it for me:

"If your brother, the son of your father or of your mother, or your son or daughter, or the spouse whom you embrace, or your most intimate friend, tries to secretly seduce you, saying, 'Let us go and serve other gods,' unknown to you or your ancestors before you, gods of the peoples surrounding you, whether near you or far away, anywhere throughout the world, you must not consent, you must not listen to him; you must show him no pity, you must not spare him or conceal his guilt. No, you must kill him, your hand must strike the first blow in putting him to death and the hands of the rest of the people following.  You must stone him to death, since he has tried to divert you from Yahweh your God."  (Deuteronomy 13:7-11, emphasis added)

This was likewise the rule which Stalin set for Communism, and Hitler for Nazism: if your brother tries to tell you why Marx is wrong, if your son tries to tell you the Jews are not planning world conquest, then do not debate him or set forth your own evidence; do not perform replicable experiments or examine history; but turn him in at once to the secret police.

Yesterday, I suggested that one key to resisting an affective death spiral is the principle of "burdensome details"—just remembering to question the specific details of each additional nice claim about the Great Idea.  (It's not trivial advice.  People often don't remember to do this when they're listening to a futurist sketching amazingly detailed projections about the wonders of tomorrow, let alone when they're thinking about their favorite idea ever.)  This wouldn't get rid of the halo effect, but  it would hopefully reduce the resonance to below criticality, so that one nice-sounding claim triggers less than 1.0 additional nice-sounding claims, on average.

The diametric opposite of this advice, which sends the halo effect supercritical, is when it feels wrong to argue against any positive claim about the Great Idea.  Politics is the mind-killer.  Arguments are soldiers.  Once you know which side you're on, you must support all favorable claims, and argue against all unfavorable claims.  Otherwise it's like giving aid and comfort to the enemy, or stabbing your friends in the back.


  • ...you feel that contradicting someone else who makes a flawed nice claim in favor of evolution, would be giving aid and comfort to the creationists;
  • ...you feel like you get spiritual credit for each nice thing you say about God, and arguing about it would interfere with your relationship with God;
  • ...you have the distinct sense that the other people in the room will dislike you for "not supporting our troops" if you argue against the latest war;
  • ...saying anything against Communism gets you stoned to death shot;

...then the affective death spiral has gone supercritical.  It is now a Super Happy Death Spiral.

It's not religion, as such, that is the key categorization, relative to our original question:  "What makes the slaughter?"  The best distinction I've heard between "supernatural" and "naturalistic" worldviews is that a supernatural worldview asserts the existence of ontologically basic mental substances, like spirits, while a naturalistic worldview reduces mental phenomena to nonmental parts.  (Can't find original source thanks, g!)  Focusing on this as the source of the problem buys into religious exceptionalism.  Supernaturalist claims are worth distinguishing, because they always turn out to be wrong for fairly fundamental reasons.  But it's still just one kind of mistake.

An affective death spiral can nucleate around supernatural beliefs; especially monotheisms whose pinnacle is a Super Happy Agent, defined primarily by agreeing with any nice statement about it; especially meme complexes grown sophisticated enough to assert supernatural punishments for disbelief.  But the death spiral can also start around a political innovation, a charismatic leader, belief in racial destiny, or an economic hypothesis.  The lesson of history is that affective death spirals are dangerous whether or not they happen to involve supernaturalism.  Religion isn't special enough, as a class of mistake, to be the key problem.

Sam Harris came closer when he put the accusing finger on faith. If you don't place an appropriate burden of proof on each and every additional nice claim, the affective resonance gets started very easily.  Look at the poor New Agers.  Christianity developed defenses against criticism, arguing for the wonders of faith; New Agers culturally inherit the cached thought that faith is positive, but lack Christianity's exclusionary scripture to keep out competing memes.  New Agers end up in happy death spirals around stars, trees, magnets, diets, spells, unicorns...

But the affective death spiral turns much deadlier after criticism becomes a sin, or a gaffe, or a crime.  There are things in this world that are worth praising greatly, and you can't flatly say that praise beyond a certain point is forbidden.  But there is never an Idea so true that it's wrong to criticize any argument that supports it.  Never.  Never ever never for ever.  That is flat.  The vast majority of possible beliefs in a nontrivial answer space are false, and likewise, the vast majority of possible supporting arguments for a true belief are also false, and not even the happiest idea can change that.

And it is triple ultra forbidden to respond to criticism with violence.  There are a very few injunctions in the human art of rationality that have no ifs, ands, buts, or escape clauses.  This is one of them.  Bad argument gets counterargument.  Does not get bullet.  Never.  Never ever never for ever.


Part of the Death Spirals and the Cult Attractor subsequence of How To Actually Change Your Mind

Next post: "Evaporative Cooling of Group Beliefs"

Previous post: "Resist the Happy Death Spiral"

Moderation Guidelines: Reign of Terror - I delete anything I judge to be annoying or counterproductiveexpand_more

Err... did that post end up dying in a free speech happy death spiral?

Especially odd from a person who believes in the probable possibility of humanly irresistible bad arguments as a reason for not AI boxing. If there are minds that we can't let exist because they would make bad arguments that we would find persuasive this seems terribly close, from an aggregative utilitarian standpoint, to killing them.

I'm not an expert in the Rwandan genocide, but it's my impression that to a substantial extent the people behind it basically just made arguments (bad ones, of a primarily ad-hominem form like "Tutsis are like cockroaches") for killing them and people who listened to those arguments on the radio went along with it. At least with the benefit of hindsight I am reluctant to say that the people promoting that genocide should have been stopped forcibly. Similarly, it's my impression that Charles Manson didn't personally kill anyone. He merely told his followers ridiculous stories of what the likely results of their killing certain people would be.

It would be nice if, as Socrates claimed, a bad argument cannot defeat a good one, but if that was true we wouldn't need to overcome bias. With respect to our own biases, hopefully careful thought and study of psychology is the only tool we will ever need to overcome them, but with respect to the biases of others it would be terribly biased to never consider the possibility that other tools are necessary. We can find good heuristics, like "don't violently suppress anyone who isn't actively promoting violence", but sadly violence isn't a basic ontological category, so we can't cleanly divide the world into violent and non-violent actions, no into statements that promote or don't promote some conclusion (in the context of what goal system?).

There are plenty of situations where violence is the correct answer. There are even situations where being the first to initiate violence is the correct answer, for example, to establish a property-ownership system and enforce against anyone being able to wander in and eat the crops you grew, even if they don't use violence before eating.

However, in real life, initiation of violence is never the correct answer to a verbal argument you don't like. Anyone can "imagine" exceptions to the rule, involving definite knowledge that an argument persuading other people is wrong, and (more difficult) absolute knowledge of the consequences, and (most difficult) themselves being the only people in the world who will ever pick up a gun. Except that it's easy to forget these as conditions, if you imagine in a naively realistic way - postulate a "wrong argument" instead of your own belief that an argument is wrong, postulate "I shoot them and that makes the problem go away" instead of your own belief that these are the consequences, and just not think about anyone else being inspired to follow the same rule. Real ethical rules, however, have to apply in the case of states of knowledge, rather than states of reality. So don't tell me about situations in which it is appropriate to respond to an argument with violence. Tell me about realistically obtainable states of belief in which it is appropriate to respond to an argument with violence.

What was the point of quoting Deuteronomy, then? The Deuteronomy quote is very specifically about introducing the worship of foreign or novel gods. Conflating this with a general decree to punish critics is a totally implausible reading to anyone who’s actually bothered to pay attention to the Bible; ancient Israelite prophets frequently claimed that Yahweh’s instructions had been wrongly construed, and that the dominant power structure (including both kings and the priesthood) was in error. They seem to have been a sufficiently protected class that kings and priests would sometimes yell at them, but rarely physically injure them.

The correct modern analogue to advocating the worship of a foreign god, is advocating cooperation with a foreign government. The contemporary analogue to stoning the person introducing the worship of foreign gods, would be imposing legal sanctions against Facebook for colluding with Russian intelligence services to manipulate American election results.

I've found this response to be incredibly useful in other discussions of morality. I hadn't found it formulated elsewhere, and had been looking for something like it for a long time.

Especially odd from a person who believes in the probable possibility of humanly irresistible bad arguments as a reason for not AI boxing. If there are minds that we can't let exist because they would make bad arguments that we would find persuasive this seems terribly close, from an aggregative utilitarian standpoint, to killing them.

Fine, let me rephrase: in the human art of rationality there's a flat law against meeting arguments with violence, anywhere in the human world. In the superintelligent domain, as you say, violence is not an ontological category and there is no firm line between persuading someone with a bad argument and reprogramming their brain with nanomachines. In our world there is a firm line, however.

Let me put it this way: If you can invent a bullet that, regardless of how it is fired, or who fires it, only hits people who emit untrue statements, then you can try to use bullets as part of a Bayesian analysis. Until then, you really ought to consider the possibility of the other guy shooting back, no matter how right you are or how wrong they are, and ask whether you want to start down that road.

If the other guy shoots first, of course, that's a whole different story that has nothing to do with free speech.

"Bad argument gets counterargument. Does not get bullet. Never. Never ever never for ever."

What about knowledge which is actually dangerous, eg., the Utterly Convincing and Irresistible Five-Minute Seminar on Why We Should Build a UFAI, with highly detailed technical instructions.

Not murdering people for criticizing your beliefs is, at the very least, a useful heuristic.