This is an unusually high quality post, even for you Eliezer; congrats!

It seems that it takes an Eliezer-level rationalist to make an explicit account of what any ten-year-old can do intuitively. For those not quite Eliezer-level or not willing to put in the effort, this is really frustrating in the context of an argument or debate.

I suspect that there are many people in this world who are, by their own standards, better off remaining deluded. I am not one if them; but I think you should qualify statements like "if a belief is false, you are better off knowing that it is false".

It is even possible that some overoptimistic transhumanists/singularitarians are better off, by their own standards, remaining deluded about the potential dangers of technology. You have the luxury of being intelligent enough to be able to utilize your correct belief about how precarious our continued existence is becoming. For many people, such a belief is of no practical benefit yet is psychologically detrimental.

This creates a "tradgedy of the commons" type problem in global catastrophic risks: each individual is better off living in a fool's paradise, but we'd all be much better off if everyone faced up to the dangers of future technology.

Some interesting, useful stuff in this post. Minus the status-cocaine of declaring that you're smarter than Robert Aumann about his performed religious beliefs and the mechanics of his internal mental state. In that area, I think Michael Vassar's model for how nerds interpret the behavior of others is your God. There's probably some 10 year olds that can see through it (look everybody, the emperor has no conception that people can believe one thing and perform another). Unless this is a performance on your part too, and there's shimshammery all the way down!

"How do I know if long-held belief X is false?"

Eliezer, I guess if you already are asking this question you are well on your way. The real problem arises when you didn't even manage to pinpoint the possibly false believe. And yes I was a religious person for many years before realizing that I was on the wrong way.

Why didn't I question my faith? Well, it was so obviously true to me. The thing is: did you ever question heliocentrism? No? Why not? When you ask the question "How do I know if Heliocentrism is false?" You are already on your way. The thing is, your brain needs a certain amount of evidence to pinpoint the question.

How did I overcome my religion? I noticed that something was wrong with my worldview like seeing a deja vu in the matrix every now and then. This on an intellectual level, not as a visible thing. But much more subtle and less obvious so you really have to be attentive no notice it, to notice that there is a problem in the pattern. Things aren't the way they should be.

But over time I became more and more aware that the pieces weren't fitting together. But from there to arrive at the conclusion that my basic assumptions where wrong was really ... (read more)

Good post but this whole crisis of faith business sounds unpleasant. One would need Something to Protect to be motivated to deliberately venture into this masochistic experience.

All these posts present techniques for applying a simple principle: check every step on the way to your belief. They adapt this principle to be more practically useful, allowing a person to start on the way lacking necessary technical knowledge, to know which errors to avoid, which errors come with being human, where not to be blind, which steps to double-check, what constitutes a step and what a map of a step, and so on. All the techniques should work in background mode, gradually improving the foundations, propagating the consequences of the changes to m... (read more)

Fact check: MDL is not Bayesian. Done properly, it doesn't even necessarily obey the likelihood principle. Key term: normalized maximum likelihood distribution.

My father is an atheist with Jewish parents, and my mother is a (non-practicing) Catholic. I was basically raised "rationalist", having grown up reading my father's issues of Skeptical Inquirer magazine. I find myself in the somewhat uncomfortable position of admitting that I acquired my belief in "Science and Reason" in pretty much the same way that most other people acquire their religious beliefs.

I'm pretty sure that, like everyone else, I've got some really stupid beliefs that I hold too strongly. I just don't know which ones they are!

Great post. I think that this sort of post on rationality is extremely valuable. While one can improve everyday judgment and decision making by learning about rationality from philosophy, econ and statistics, I think that these informal posts can also make a significant difference to people.

The recent posts on AI theorists and EY's biography were among my least favorite on OB. If you have a choice, please spend more time on either technical sequences (e.g. stuff on concepts/concept space, evolutionary bio, notion of bias in statistics) or stuff on rationality like this.

"Try to think the thought that hurts the most."

This is exactly why I like to entertain religious thoughts. My background, training, and inclination are to be a thoroughgoing atheist materialist, so I find that trying to make sense of religious ideas is good mental exercise. Feel the burn!

In that vein, here is an audio recording of Robert Aumann on speaking on "The Personality of God".

Also, the more seriously religious had roughly the same idea, or maybe it's the opposite idea. The counterfactuality of religious ideas is part of their strength, apparently.

I'd be interested in a list of questions you had decided to have a crisis of faith over. If I get round to it I might try and have one over whether a system can recursively self-improve in a powerful way or not.

A lot of truths in EY's post. Though I also agree with Hopefully Anon's observations -- as is so often the case, Eliezer reminds me of Descartes -- brilliant, mathematical, uncowed by dogma, has his finger on the most important problems, is aware of how terrifyingly daunting those problems are, thinks he has a universal method to solve those problems.

Thank you for this post, Eliezer. I must painfully question my belief that a positive Singularity is likely to occur in the foreseeable future.

Nazir Ahmad Bhat, you are missing the point. It's not a question of identity, like which ice cream flavor you prefer. It's about truth. I do not believe there is a teapot orbiting around Jupiter, for the various reasons explained on this site (see Absence of evidence is evidence of absence and the posts on Occam's Razor). You may call this a part of my identity. But I don't need people to believe in a teapot. Actually, I want everyone to know as much as possible. Promoting false beliefs is harming people, like slashing their tires. You don't believe in a flying teapot: do you need other people to?

Nazir, must there be atheists in order for you to believe in a god? The "identity" of those who believe that the world is round does not depend on others believing that the world is flat, or vice versa. Truth does not require disagreement.

Matthew C.,

You've been suggesting that for a while:

http://www.overcomingbias.com/2007/01/godless_profess.html#comment-27993437 http://www.overcomingbias.com/2008/09/psychic-powers.html#comment-130445874

Those who have read it (or the hundreds of pages available on Google Books, which I have examined) don't seem to be impressed.

Why do you think it's better than Broderick's book? If you want to promote it more effectively in the face of silence (http://www.overcomingbias.com/2007/02/what_evidence_i.html), why not pay for a respected reviewer's time and a writ... (read more)

Do these methods actually work? There were a few posts here on how more evidence and bias awareness don't actually change minds or reduce bias, at least not without further effort. Can a practical "Deduce the Truth in 30 Days" guide be derived from these methods, and change the world?

A fifty-fifty chance of choosing your previous belief does not constitute a reasonable test. If your belief is unreasonable, why would treating it as equally plausible as the alternative be valid?

The trick is to suspend belief and negate the biasing tendencies of belief when you re-evaluate, not to treat all potentials as equal.

MichaelG:

On the other hand, I don't think a human race with nerds can forever avoid inventing a self-destructive technology like AI.

The idea is that if we invent Friendly AI first, it will become powerful enough to keep later, Unfriendly ones in check (either alone, or with several other FAIs working together with humanity). You don't need to avoid inventing one forever: it's enough to avoid inventing one as the first thing that comes up.

If a belief is true you will be better off believing it, and if it is false you will be better off rejecting it.

• You have a set of beliefs optimized for co-occurence, and you are replacing one of these beliefs with a more-true belief. In other words, the new true belief will cause you harm because of other untrue (or less true) beliefs that you still hold.
• If an entire community can be persuaded to adopt a false belief, it may enable them to overcome a tragedy-of-the-common

I was raised in a christian family, fairly liberal Church of England, and my slide into agnosticism started when I about 5-7 when I asked if Santa Claus and God were real. I refused to get confirmed and stopped going to church when I was 13ish I think.

In other words, you are advocating a combative, Western approach; I am bringing up a more Eastern approach, which is not to be so attached to anything in the first place, but to bend if the wind blows hard enough.

The trouble is that you cannot break new ground this way. You can't do einstein like feats. Y... (read more)

If an entire community can be persuaded to adopt a false belief, it may enable them to overcome a tragedy-of-the-commons or prisoners'-dilemma situation.

In a PD, agents hurt each other, not themselves. Obviously false beliefs in my enemy can help me.

Study this deranged rant. Its ardent theism is expressed by its praise of the miracles God can do, if he choses.

And yet,... There is something not quite right here. Isn't it merely cloakatively theistic? Isn't the ringing denounciation of "Crimes against silence" militant atheism at its most strident?

So here is my idea: Don't try to doubt a whole core belief. That is too hard. Probe instead for the boundary. Write a little fiction, perhaps a science fiction of first contact, in which you encounter a curious character from a different culture. Wri... (read more)

If a belief is true you will be better off believing it, and if it is false you will be better off rejecting it. I think evolution facilitated self-delusion precisely because that is not the case.

I was a Fred Phelps style ultra-Calvinist and my transition involved scarcely any effort.

Also, anti-reductionist, that's the first comment you've made I felt was worth reading. You may take it as an insult but I felt compelled to give you kudos.

Eliezer, what do you mean here? Do you mean:

(A1) Individuals in the reference class really are always better off with the truth, with sufficient probability that the alternative does not bear investigating;

(A2) Humans are so unreliable as judges of what we would and would not benefit from being deceived about that the heuristic "we're always better off with the truth" is more accurate than the available alternatives;

(B) Individuals must adopt the Noble Might-be-truth "I'm always better off with the truth" to have a chance at the Crisis of Faith technique?

Eliezer: The position that people may be better off deluded in some situations is VERY compelling. If your audience is people who are literally NEVER better off deluded then I sincerely doubt that it includes you or anyone else. Obviously not every belief need receive all appropriate qualifications every single time, but when someone else points out a plausible qualification you should, as a rationalist, acknowledge it.

I'm very open to Anna's (A1), especially given the special difficulties of this sort of investigation, but only with respect to themselves. I would expect someone as smart as me who knew me well enough to some day come upon a situation where I should, by my values, be deceived, at least for some period.

@Anna:

I mean that you've given up trying to be clever.

@Vassar:

The position that people may be better off deluded in some situations is VERY compelling.

The position that people may be optimally deluded, without a third alternative, is much less compelling.

The position that realistic human students of rationality can be trying to do their best (let alone do the impossible), while trying to deliberately self-delude, strikes me as outright false. It would be like trying to win a hot-dog eating contest while keeping a golf ball in your mouth.

It is this outright falsity that I refer to when I say that by the time you attempt to employ techniques at this level, you should already have given up on trying to be clever.

As someone once said to Brennan:

She reared back in mock-dismay. "Why, Brennan, surely you don't expect me to just tell you!"

Brennan gritted his teeth. "Why not?"

"What you're feeling now, Brennan, is called curiosity. It's an important emotion. You need to learn to live with it and draw upon its power. If I just give you the information, why, you won't be curious any more." Her eyes turned serious. "Not that you should prefer ign

The problem with the idea that sometimes people are better of not knowing is that it has no practical impact on how an ideal rationalist should behave, even assuming it's true. By the time you've learned something you'd be better off not knowing, it's too late to unlearn it. Humans can't really do doublethink, and especially not at the precision that would be required to be extremely rational while using it.

it has no practical impact on how an ideal rationalist should behave

With respect to themselves, not necessarily to others. Withholding information or even lying can be rational.

In other words, you are advocating a combative, Western approach; I am bringing up a more Eastern approach, which is not to be so attached to anything in the first place, but to bend if the wind blows hard enough.

The trouble is that you cannot break new ground this way. You can't do einstein like feats. You should follow the direction of the wind, but engage nitrous to follow that direction. Occasionally stopping and sticking a finger out the window to make sure you are going the right direction.

In retrospect, I was certainly leaving christianity the day I decided that if god existed, he could not possibly object to me honestly trying to determine The Truth. Doubts stopped feeling like sins.

I think your something-to-protect must be the accuracy of your map, not anything on the map. (at least for a moment)

If someone says you need to fire a revolver at your something-to-protect, you will raise objection based on strongly held beliefs about the effects of revolvers. It's so hard to risk those beliefs because, with your current belief set, someone who... (read more)

In a PD, everyone having accurate information about the payoff matrix leads to a worse outcome for everyone, than some false payoff matrices you could misinform them with. That is the point.

Do you agree that in a PD, it is not the case for any individual that that individual is harmed by that individual's knowledge? Your point goes through if we somehow think of the collective consisting as a single "you" with beliefs and preferences, but raises all sorts of issues and anyway isn't what Eliezer was talking about.

I think Einstein is a good example of both bending with the wind (when he came up with relativity)

If the wind is following occams or something internal, then it can be blowing in the wrong direction...

If the wind is following occams or something internal, then it can be blowing in the wrong direction...

Isn't that the subject of The Ritual?

One more argument against deceiving epistemic peers when it seems to be in their interest is that if you are known to have the disposition to do so, this will cause others to trust your non-deceptive statements less; and here you could recommend that they shouldn't trust you less, but then we're back into doublethink territory.

Phil Goetz, who I was replying to, was saying that type of thought should be unnecessary, if you don't hang on to your ideas tightly.

Not hanging on to ideas tightly is great for engineers and experimental scientists. It doesn't matter to a chemist if MWI or bohm is right. He can use either, switching back or forth from the view points as he sees fit.

For a theoretical quantum physicist, he has to have some way of determining at which face of the knowledge mine to work, he has to pick one or the other. If it is not a strong reason then he might split his wor... (read more)

Well, I've just sat down and done one of those, and it was really difficult. Not so much because I was pushing against established beliefs (I had strong beliefs both ways, so it was more that any movement pushed somewhere) but because the largest worry I had, "Is this a fad?", is hard to answer specifically because I've recently changed to become so much more Bayesian. I used to do daft things like "giving ideas a chance". Consequently, I can't look to my long and undistinguished history in order to glean hints. I already don't do the o... (read more)

"I am better off deluding myself into believing my cherished late spouse was faithful to me."

1) Do you believe this is true for you, or only other people?

2) If you know that someone's cherished late spouse cheated on them, are you justified in keeping silent about the fact?

3) Are you justified in lying to prevent the other person from realizing?

4) If you suspect for yourself (but are not sure) that the cherished late spouse might have been unfaithful, do you think that you will be better off, both for the single deed, and as a matter of you... (read more)

I think Einstein is a good example of both bending with the wind (when he came up with relativity)

I am better off (in most circumstances) if deluding myself to believe that the weather in Maine on the 23rd of June 1865 was near what I think the seasonal average might be, for that decade, rather than memorising the exact temperature and rainfall if it was presented to me.

I believe this is true for most people, apart from climatologists.

I would be rather not be around people who kept telling me true minutiae about the world and he cosmos, if they have no bearing on the problems I am trying to solve.

Am I justified in giving people a guess of the average ... (read more)

From where I'm standing, the spouse thing looks like obvious nonsense (of the category: not looking for a third alternative). You'd be far better off learning to share - especially since, if your spouse died, you'd have someone to talk to.

Nazir, a secret hack to prevent Eliezer from deleting your posts is here. #11.6 is particularly effective.

1) Do you believe this is true for you, or only other people?

I don't fit the premise of the statement -- my cherished spouse is not yet late, so it's hard to say.

2) If you know that someone's cherished late spouse cheated on them, are you justified in keeping silent about the fact?

Mostly yes.

3) Are you justified in lying to prevent the other person from realizing?

Mostly no.

4) If you suspect for yourself (but are not sure) that the cherished late spouse might have been unfaithful, do you think that you will be better off, both for the single deed, and as a... (read more)

Carl Schuman,

I keep posting the link, for a very simple reason.

Eliezer continues to post about the certainty of reductionism, while he has completely failed to investigate the evidence that reductionism cannot account for all of the observations.

He also continues to post snide remarks about the reality of psi phenomena. Again, he has completely failed to investigate the best evidence that he is wrong about this.

The post he wrote here shows a great committment to intellectual integrity. And I honestly believe he means what he wrote here.

I suspect at some ... (read more)

Eliezer continues to post about the certainty of reductionism, while he has completely failed to investigate the evidence that reductionism cannot account for all of the observations.

His point is necessarily correct, as well as empirically so.

He also continues to post snide remarks about the reality of psi phenomena. Again, he has completely failed to investigate the best eviden

I guess I am questioning whether making a great effort to shake yourself free of a bias is a good or a bad thing, on average. Making a great effort doesn't necessarily get you out of biased thinking. It may just be like speeding up when you suspect you're going in the wrong direction.

If someone else chose a belief of yours for you to investigate, or if it were chosen for you at random, then this effort might be a good thing. However, I have observed many cases where someone chose a belief of theirs to investigate thoroughly, precisely because it was an ... (read more)

Religion is the classic example of a delusion that might be good for you. There is some evidence that being religious increases human happiness, or social cohesion. It's universality in human culture suggests that it has adaptive value.

See last week's Science, Oct. 3 2008, p. 58-62: "The origin and evolution of religious prosociality". One chart shows that, in any particular year, secular communes are four times as likely to dissolve as religious communes.

Caledonian,

Read chapter 3, then come back and explain why a reductionistic explanation best accounts for the phenomena described there. Because if you are inconversant with the evidence, you simply have no rational basis to make any comment whatsoever.

You also seem to be playing some kind of semantic games with the word "reductionism" which I'll just note and ignore.

It's important in these crisis things to remind yourself that 1) P does not imply "there are no important generally unappreciated arguments for not-P", and 2) P does not imply "the proponents of P are not all idiots, dishonest, and/or users of bad arguments". You can switch sides without deserting your favorite soldiers. IMO.

Matthew C -

You are advocating nonreductionism and psi at the same time.

Supposing that you are right requires us to suppose that there is both a powerful argument against reductionism, and a powerful argument in favor of psi.

Supposing that you are a crank requires only one argument, and one with a much higher prior.

In other words, if you were advocating one outrageous theory, someone might listen. The fact that you are advocating two simultaneously makes dismissing all of your claims, without reading the book you recommend, the logical response. We thus don't have to read it to have a rational basis to dismiss it.

I would be rather not be around people who kept telling me true minutiae about the world and he cosmos, if they have no bearing on the problems I am trying to solve.

Will, not wishing to be told pointless details is not the same as deluding yourself.

I was discussing the placebo effect with a friend last night though, and found myself arguing that this could well be an example of a time when more true knowledge could hurt. Paternalistic issues aside, people appear to get healthier when they believe falsehoods about the effectiveness of, say, homeopathy or s... (read more)

Phil: One of psi or non-reductionism being true would be a powerful argument in favor of the other.

Ben: great example.

Phil: One of psi or non-reductionism being true would be a powerful argument in favor of the other.

Caledonian, maybe you had arguments on this thread previously, but it seems more like the place for that sub-debate.

First of all, great post Eliezer. If somebody holding that kind of standard thinks that cryogenics is a good investment, I should someday take some time into investing the question deeper than I had.

Now, without lessening the previous praise, I would like to make the following remarks about friendly AI:
- The belief has long remained in your mind;
- It is surrounded by a cloud of known arguments and refutations;
- You have sunk costs in it (time, money, public declarations).
I do not know if it has emotional consequences for you or if it has gotten mixed u... (read more)

In fact most human beings at most times (including today) have accepted both propositions as real.

As for your text -- no one who seriously suggests that mental states affecting health and shamanistic death spells are evidence for either 'non-reductionism' or psi is worth taking the time to refute in detail.

(Hint: there are perfec... (read more)

I was raised a theist and came to no longer belive as an adult. One of the turning points was reading the Anglican confession of faith, and supposing what my own beliefs might look like to an Anglican, who was also a christian, saved by Jesus just like me - just a different variety of.

Eventually I began to wonder what my life experiences might look like to an atheist - religion is above all an interpretive filter that we use to make sense of our lives. Although I knew that my beliefs in God were right, what would my life look like to me if I did not belive... (read more)

I became a Christian because I was a Bayesian first. I know there are others like me. I saw and experienced evidence that caused me to positively update my belief.

Now if you don't like that argument, then please tell me how can anyone become an atheist via Bayesian updating? Can your posterior really go to a point mass at zero (belief in God)? If so, please tell me what prior you were using. If not, please tell me how you define atheism.

Atheism is believing that the state of evidence on the God question is similar to the state of evidence on the werewolf question.

s/werewolf/Easter bunny/ IMHO.

Would that apply to someone with a particularly high prior on the werewolf question? So, you would agree that anyone who believes that the state of evidence on "the God question" is more positive than the state of evidence on the "werewolf question" should consider labeling themselves an agnostic? theist?

And, I presume that you believe that one's current belief in the state of evidence would be controlled by 1) verifiable general evidence, 2) experience, and 3) priors on both questions?

Then we're in agreement: you should (apparently) call yourself an atheist, and I should call myself a Christian, as we differ on #2 and #3. (not that theism = Christian, but that goes back to #2).

[quote]I became a Christian because I was a Bayesian first. I know there are others like me. I saw and experienced evidence that caused me to positively update my belief.

Now if you don't like that argument, then please tell me how can anyone become an atheist via Bayesian updating? Can your posterior really go to a point mass at zero (belief in God)? If so, please tell me what prior you were using. If not, please tell me how you define atheism.
[/quote]
And how can your probability go to one? You erect a straw man, sir. My probablility that there is a god ... (read more)

@ Paul Murray

There is no straw man. You've presumed that I meant that Christian = "Pr(god)=1". That was never my claim. It had seemed that atheist was being used as Atheist="Pr(god)=0", but E. clarified his position. I think that agnostic (in the literal sense) is always a better term than atheist, but that's just semantics.

The real issue (to me) is what Christians (or other "people of faith") think of the atheistic position, and vice versa. Christians are often derided here as uneducated or un-Bayesian.

My point is not to conv... (read more)

Eliezer, that's a John McCarthy quote.