Followup to: Against Doublethink (sequence), Dark Arts of Rationality, Your Strength as a Rationalist


Doublethink

It is obvious that the same thing will not be willing to do or undergo opposites in the same part of itself, in relation to the same thing, at the same time. --Book IV of Plato's Republic

Can you simultaneously want sex and not want it? Can you believe in God and not believe in Him at the same time? Can you be fearless while frightened?

To be fair to Plato, this was meant not as an assertion that such contradictions are impossible, but as an argument that the soul has multiple parts. It seems we can, in fact, want something while also not wanting it. This is awfully strange, and it led Plato to conclude the soul must have multiple parts, for surely no one part could contain both sides of the contradiction.

Often, when we attempt to accept contradictory statements as correct, it causes cognitive dissonance--that nagging, itchy feeling in your brain that won't leave you alone until you admit that something is wrong. Like when you try to convince yourself that staying up just a little longer playing 2048 won't have adverse effects on the presentation you're giving tomorrow, when you know full well that's exactly what's going to happen.

But it may be that cognitive dissonance is the exception in the face of contradictions, rather than the rule. How would you know? If it doesn't cause any emotional friction, the two propositions will just sit quietly together in your brain, never mentioning that it's logically impossible for both of them to be true. When we accept a contradiction wholesale without cognitive dissonance, it's what Orwell called "doublethink".

When you're a mere mortal trying to get by in a complex universe, doublethink may be adaptive. If you want to be completely free of contradictory beliefs without spending your whole life alone in a cave, you'll likely waste a lot of your precious time working through conundrums, which will often produce even more conundrums.

Suppose I believe that my husband is faithful, and I also believe that the unfamiliar perfume on his collar indicates he's sleeping with other women without my permission. I could let that pesky little contradiction turn into an extended investigation that may ultimately ruin my marriage. Or I could get on with my day and leave my marriage intact.

It's better to just leave those kinds of thoughts alone, isn't it? It probably makes for a happier life.

Against Doublethink

Suppose you believe that driving is dangerous, and also that, while you are driving, you're completely safe. As established in Doublethink, there may be some benefits to letting that mental configuration be.

There are also some life-shattering downsides. One of the things you believe is false, you see, by the law of the excluded middle. In point of fact, it's the one that goes "I'm completely safe while driving". Believing false things has consequences.

Be irrationally optimistic about your driving skills, and you will be happily unconcerned where others sweat and fear. You won't have to put up with the inconvenience of a seatbelt. You will be happily unconcerned for a day, a week, a year. Then CRASH, and spend the rest of your life wishing you could scratch the itch in your phantom limb. Or paralyzed from the neck down. Or dead. It's not inevitable, but it's possible; how probable is it? You can't make that tradeoff rationally unless you know your real driving skills, so you can figure out how much danger you're placing yourself in. --Eliezer Yudkowsky, Doublethink (Choosing to be Biased)

What are beliefs for? Please pause for ten seconds and come up with your own answer.

Ultimately, I think beliefs are inputs for predictions. We're basically very complicated simulators that try to guess which actions will cause desired outcomes, like survival or reproduction or chocolate. We input beliefs about how the world behaves, make inferences from them to which experiences we should anticipate given various changes we might make to the world, and output behaviors that get us what we want, provided our simulations are good enough.

My car is making a mysterious ticking sound. I have many beliefs about cars, and one of them is that if my car makes noises it shouldn't, it will probably stop working eventually, and possibly explode. I can use this input to simulate the future. Since I've observed my car making a noise it shouldn't, I predict that my car will stop working. I also believe that there is something causing the ticking. So I predict that if I intervene and stop the ticking (in non-ridiculous ways), my car will keep working. My belief has thus led to the action of researching the ticking noise, planning some simple tests, and will probably lead to cleaning the sticky lifters.

If it's true that solving the ticking noise will keep my car running, then my beliefs will cash out in correctly anticipated experiences, and my actions will cause desired outcomes. If it's false, perhaps because the ticking can be solved without addressing a larger underlying problem, then the experiences I anticipate will not occur, and my actions may lead to my car exploding.

Doublethink guarantees that you believe falsehoods. Some of the time you'll call upon the true belief ("driving is dangerous"), anticipate future experiences accurately, and get the results you want from your chosen actions ("don't drive three times the speed limit at night while it's raining"). But some of the time, if you actually believe the false thing as well, you'll call upon the opposite belief, anticipate inaccurately, and choose the last action you'll ever take.

Without any principled algorithm determining which of the contradictory propositions to use as an input for the simulation at hand, you'll fail as often as you succeed. So it makes no sense to anticipate more positive outcomes from believing contradictions.

Contradictions may keep you happy as long as you never need to use them. Should you call upon them, though, to guide your actions, the debt on false beliefs will come due. You will drive too fast at night in the rain, you will crash, you will fly out of the car with no seat belt to restrain you, you will die, and it will be your fault.

Against Against Doublethink

What if Plato was pretty much right, and we sometimes believe contradictions because we're sort of not actually one single person?

It is not literally true that Systems 1 and 2 are separate individuals the way you and I are. But the idea of Systems 1 and 2 suggests to me something quite interesting with respect to the relationship between beliefs and their role in decision making, and modeling them as separate people with very different personalities seems to work pretty darn well when I test my suspicions.

I read Atlas Shrugged probably about a decade ago. I was impressed with its defense of capitalism, which really hammers home the reasons it’s good and important on a gut level. But I was equally turned off by its promotion of selfishness as a moral ideal. I thought that was *basically* just being a jerk. After all, if there’s one thing the world doesn’t need (I thought) it’s more selfishness.

Then I talked to a friend who told me Atlas Shrugged had changed his life. That he’d been raised in a really strict family that had told him that ever enjoying himself was selfish and made him a bad person, that he had to be working at every moment to make his family and other people happy or else let them shame him to pieces. And the revelation that it was sometimes okay to consider your own happiness gave him the strength to stand up to them and turn his life around, while still keeping the basic human instinct of helping others when he wanted to and he felt they deserved it (as, indeed, do Rand characters). --Scott of Slate Star Codex in All Debates Are Bravery Debates

If you're generous to a fault, "I should be more selfish" is probably a belief that will pay off in positive outcomes should you install it for future use. If you're selfish to a fault, the same belief will be harmful. So what if you were too generous half of the time and too selfish the other half? Well, then you would want to believe "I should be more selfish" with only the generous half, while disbelieving it with the selfish half.

Systems 1 and 2 need to hear different things. System 2 might be able to understand the reality of biases and make appropriate adjustments that would work if System 1 were on board, but System 1 isn't so great at being reasonable. And it's not System 2 that's in charge of most of your actions. If you want your beliefs to positively influence your actions (which is the point of beliefs, after all), you need to tailor your beliefs to System 1's needs.

For example: The planning fallacy is nearly ubiquitous. I know this because for the past three years or so, I've gotten everywhere five to fifteen minutes early. Almost every single person I meet with arrives five to fifteen minutes late. It is very rare for someone to be on time, and only twice in three years have I encountered the (rather awkward) circumstance of meeting with someone who also arrived early.

Before three years ago, I was also usually late, and I far underestimated how long my projects would take. I knew, abstractly and intellectually, about the planning fallacy, but that didn't stop System 1 from thinking things would go implausibly quickly. System 1's just optimistic like that. It responds to, "Dude, that is not going to work, and I have a twelve point argument supporting my position and suggesting alternative plans," with "Naaaaw, it'll be fine! We can totally make that deadline."

At some point (I don't remember when or exactly how), I gained the ability to look at the true due date, shift my System 1 beliefs to make up for the planning fallacy, and then hide my memory that I'd ever seen the original due date. I would see that my flight left at 2:30, and be surprised to discover on travel day that I was not late for my 2:00 flight, but a little early for my 2:30 one. I consistently finished projects on time, and only disasters caused me to be late for meetings. It took me about three months before I noticed the pattern and realized what must be going on.

I got a little worried I might make a mistake, such as leaving a meeting thinking the other person just wasn't going to show when the actual meeting time hadn't arrived. I did have a couple close calls along those lines. But it was easy enough to fix; in important cases, I started receiving Boomeranged notes from past-me around the time present-me expected things to start that said, "Surprise! You've still got ten minutes!"

This unquestionably improved my life. You don't realize just how inconvenient the planning fallacy is until you've left it behind. Clearly, considered in isolation, the action of believing falsely in this domain was instrumentally rational.

Doublethink, and the Dark Arts generally, applied to carefully chosen domains is a powerful tool. It's dumb to believe false things about really dangerous stuff like driving, obviously. But you don't have to doublethink indiscriminately. As long as you're careful, as long as you suspend epistemic rationality only when it's clearly beneficial to do so, employing doublethink at will is a great idea.

Instrumental rationality is what really matters. Epistemic rationality is useful, but what use is holding accurate beliefs in situations where that won't get you what you want?

Against Against Against Doublethink

There are indeed epistemically irrational actions that are instrumentally rational, and instrumental rationality is what really matters. It is pointless to believing true things if it doesn't get you what you want. This has always been very obvious to me, and it remains so.

There is a bigger picture.

Certain epistemic rationality techniques are not compatible with dark side epistemology. Most importantly, the Dark Arts do not play nicely with "notice your confusion", which is essentially your strength as a rationalist. If you use doublethink on purpose, confusion doesn't always indicate that you need to find out what false thing you believe so you can fix it. Sometimes you have to bury your confusion. There's an itsy bitsy pause where you try to predict whether it's useful to bury.

As soon as I finally decided to abandon the Dark Arts, I began to sweep out corners I'd allowed myself to neglect before. They were mainly corners I didn't know I'd neglected.

The first one I noticed was the way I responded to requests from my boyfriend. He'd mentioned before that I often seemed resentful when he made requests of me, and I'd insisted that he was wrong, that I was actually happy all the while. (Notice that in the short term, since I was probably going to do as he asked anyway, attending to the resentment would probably have made things more difficult for me.) This self-deception went on for months.

Shortly after I gave up doublethink, he made a request, and I felt a little stab of dissonance. Something I might have swept away before, because it seemed more immediately useful to bury the confusion than to notice it. But I thought (wordlessly and with my emotions), "No, look at it. This is exactly what I've decided to watch for. I have noticed confusion, and I will attend to it."

It was very upsetting at first to learn that he'd been right. I feared the implications for our relationship. But that fear didn't last, because we both knew the only problems you can solve are the ones you acknowledge, so it is a comfort to know the truth.

I was far more shaken by the realization that I really, truly was ignorant that this had been happening. Not because the consequences of this one bit of ignorance were so important, but because who knows what other epistemic curses have hidden themselves in the shadows? I realized that I had not been in control of my doublethink, that I couldn't have been.

Pinning down that one tiny little stab of dissonance took great preparation and effort, and there's no way I'd been working fast enough before. "How often," I wondered, "does this kind of thing happen?"

Very often, it turns out. I began noticing and acting on confusion several times a day, where before I'd been doing it a couple times a week. I wasn't just noticing things that I'd have ignored on purpose before; I was noticing things that would have slipped by because my reflexes slowed as I weighed the benefit of paying attention. "Ignore it" was not an available action in the face of confusion anymore, and that was a dramatic change. Because there are no disruptions, acting on confusion is becoming automatic.

I can't know for sure which bits of confusion I've noticed since the change would otherwise have slipped by unseen. But here's a plausible instance. Tonight I was having dinner with a friend I've met very recently. I was feeling s little bit tired and nervous, so I wasn't putting as much effort as usual into directing the conversation. At one point I realized we had stopped making making any progress toward my goals, since it was clear we were drifting toward small talk. In a tired and slightly nervous state, I imagine that I might have buried that bit of information and abdicated responsibility for the conversation--not by means of considering whether allowing small talk to happen was actually a good idea, but by not pouncing on the dissonance aggressively, and thereby letting it get away. Instead, I directed my attention at the feeling (without effort this time!), inquired of myself what precisely was causing it, identified the prediction that the current course of conversation was leading away from my goals, listed potential interventions, weighed their costs and benefits against my simulation of small talk, and said, "What are your terminal values?"

(I know that sounds like a lot of work, but it took at most three seconds. The hard part was building the pouncing reflex.)

When you know that some of your beliefs are false, and you know that leaving them be is instrumentally rational, you do not develop the automatic reflex of interrogating every suspicion of confusion. You might think you can do this selectively, but if you do, I strongly suspect you're wrong in exactly the way I was.

I have long been more viscerally motivated by things that are interesting or beautiful than by things that correspond to the territory. So it's not too surprising that toward the beginning of my rationality training, I went through a long period of being so enamored with a-veridical instrumental techniques--things like willful doublethink--that I double-thought myself into believing accuracy was not so great.

But I was wrong. And that mattered. Having accurate beliefs is a ridiculously convergent incentive. Every utility function that involves interaction with the territory--interaction of just about any kind!--benefits from a sound map. Even if "beauty" is a terminal value, "being viscerally motivated to increase your ability to make predictions that lead to greater beauty" increases your odds of success.

Dark side epistemology prevents total dedication to continuous improvement in epistemic rationality. Though individual dark side actions may be instrumentally rational, the patterns of thought required to allow them are not. Though instrumental rationality is ultimately the goal, your instrumental rationality will always be limited by your epistemic rationality.

That was important enough to say again: Your instrumental rationality will always be limited by your epistemic rationality.

It only takes a fraction of a second to sweep an observation into the corner. You don't have time to decide whether looking at it might prove problematic. If you take the time to protect your compartments, false beliefs you don't endorse will slide in from everywhere through those split-second cracks in your art. You must attend to your confusion the very moment you notice it. You must be relentless an unmerciful toward your own beliefs.

Excellent epistemology is not the natural state of a human brain. Rationality is hard. Without extreme dedication and advanced training, without reliable automatic reflexes of rational thought, your belief structure will be a mess. You can't have totally automatic anti-rationalization reflexes if you use doublethink as a technique of instrumental rationality.

This has been a difficult lesson for me. I have lost some benefits I'd gained from the Dark Arts. I'm late now, sometimes. And painful truths are painful, though now they are sharp and fast instead of dull and damaging.

And it is so worth it! I have much more work to do before I can move on to the next thing. But whatever the next thing is, I'll tackle it with far more predictive power than I otherwise would have--though I doubt I'd have noticed the difference.

So when I say that I'm against against against doublethink--that dark side epistemology is bad--I mean that there is more potential on the light side, not that the dark side has no redeeming features. Its fruits hang low, and they are delicious.

But the fruits of the light side are worth the climb. You'll never even know they're there if you gorge yourself in the dark forever.

Moderation Guidelinesexpand_more

Thanks for writing this!

I remain unconvinced. I agree with most of your points, and I think most of my disagreement stems from modeling my mind, the world, and/or 'dark techniques' in a different way than you do. I'd be happy to get together and try to converge sometime.

I do have one direct disagreement with the text, which is somewhat indicative of my more general disagreements.

Your instrumental rationality will always be limited by your epistemic rationality.

In my experience, many rationalists are motivation-limited, not accuracy-limited. I have met many people who are smarter than I am, who think faster than I do, who are better epistemic rationalists than I am---and who suffer greatly from akrasia or other stamina issues.

I seem to be quite good at achieving my goals. I am by no means convinced that this is due to some excess of willpower: my successes could alternatively be attributed to chance, genetics, self-delusion, or other factors. Even conditioned upon the assumption that my ability to avoid akrasia is a large part of my success, I am not convinced that my motivational techniques are the source of this ability.

However, I do see many "light-side" epistemic rationalists suffering from more akrasia than I do. In the real world, I am not convinced that epistemic rationality is enough. As such, I am cautious about removing motivational techniques in the name of the light.

(I also am under the impression that I can use my motivational techniques in such a way as to avoid many of the averse effects you mention, which gets back to us modeling things differently. This is, of course, exactly what my brain would tell me, and the objection should largely be disregarded until we have a chance to converge.)

There is, of course, some degree to which the above argument only indicates my ability to find self-protecting arguments that I myself find convincing. This topic is somewhat emotionally laden for me, so next time I find a few spare hours I will spend them strongly considering whether I am wrong. However, after cursory examination, I don't expect any particular update.

It seems somewhat absurd to say that your ability to achieve goals is limited by the thingspace cluster we refer to as epistemic rationality. After all, caring too much about epistemic rationality leads to needing things like this.

Epistemic rationality seems like it should be something that you care about when it matters to care about, and don't care about when it doesn't matter. Like any other investment, your capital invested should be proportional to your rate and belief of return. Similarly, you should always be willing to sacrifice some epistemic cleanliness if it means winning. You can clean up the dark nasty corners of your mind on top of your pile of utility.

I think the point Brienne made is that seemingly small tradeoffs of epistemic accuracy for instrumental power actually cost much more than you might expect. You can't pay a little epistemic accuracy for a lot of instrumental power, because epistemic rationality requires that you leave yourself no outs. If you sanction even one tiny exception, you lose the benefits of purity that you didn't even know were available.

If you sanction even one tiny exception, you lose the benefits of purity

What is that "purity" you're talking about? I didn't realize humans could achieve epistemic perfection.

Keep in mind here that I'm steelmanning someone else's argument, perhaps improperly. I don't want to put words in anyone else's mouth. That said, I used the term 'purity' in loose analogy to a 'pure' programming language, wherein one exception is sufficient to remove much of the possible gains.

Continuing the steelmanning, however, I'd say that while no human can achieve epistemic perfection, there's a large class of epistemic failures that you only recognize if you're striving for perfection. Striving for purity, not purity itself, is what gets you the gains.

So8ers, you're completely accurate in your interpretation of my argument. I'm going to read some more of your previous posts before responding much to your first comment here.

Yes, as Eliezer put it somewhat dramatically here:

If you once tell a lie, the truth is ever after your enemy.

To expand on this in context, as long as you are striving for the truth any evidence you come across helps you, but once you choose to believe a lie you must forever avoid dis-confirming evidence.

You've drawn an important distinction, between believing a lie and telling one. Right now we're talking about lying to ourselves so the difference isn't very great, but be very careful with that quote in general.

You've drawn an important distinction, between believing a lie and telling one. Your formulation is correct, but Eliezer's is wrong.

Yes, it's pretty much impossible to tell a lie without hurting other people, or at least interfering with them; that's the point of lying, after all. But right now we're talking about the harm one does to oneself by lying; I submit that there needn't be any.

One distinction I don't know if it matters, but many discussions fail to mention at all, is the distinction between telling a lie and maintaining it/keeping the secret. Many of the epistemic arguments seem to disappear if you've previously made it clear you might lie to someone, you intend to tell the truth a few weeks down the line, and if pressed or questioned you confess and tell the actual truth rather than try to cover it with further lies.

Edit: also, have some kind of oat and special circumstance where you will in fact never lie, but precommit to only use it for important things or give it a cost in some way so you won't be pressed to give it for everything.

Did you even read the comment I linked to? It's whole point was about the harm you do to yourself and your cause by lying.

I think you and fezziwig aren't disagreeing. You're saying as an empirical matter that lying can (and maybe often does) harm the liar. He's just saying that it doesn't necessarily harm the liar, and indeed it may well be that lies are often a net benefit. These are compatible claims.

in loose analogy to a 'pure' programming language, wherein one exception is sufficient to remove much of the possible gains.

Could you give some examples?

there's a large class of epistemic failures that you only recognize if you're striving for perfection.

I am not sure which class you're talking about... again, can you provide some examples?

You're definitely paying for epistemic rationality with instrumental power if you spend all of your time contemplating metaethics so that you have a pure epistemic notion of what your goals are.

Humans start with effectively zero rationality. At some point, it becomes less winning to spend time gaining epistemic rationality than to spend time effecting your goals.

So, it seems like you can spend potential epistemtic rationality for instrumental power by using time to effect change rather than becoming epistemically pure.

To respond to some of your later points:

Take a programming language like OCaml. OCaml supports mutable and immutable state, and you could write an analysis over OCaml that would indeed barf and die on the first instances of mutation. Mutable state does make it incredibly difficult, sometimes to the point of impossiblity, to use conventional analyses on modern computers to prove facts about programs.

But this doesn't mean that a single ref cell destroys the benefits of purity in an OCaml program. To a human reading the program (which is really the most important use case), a single ref cell can be the best way to solve a problem, and the language's modularity can easily abstract it away so that the interface on the whole is pure.

Similarily, I agree that it's important to strive for perfection, in all cases. But striving for perfection doesn't mean taking every available sacrifice for perfection. I can strive for epistemic perfection while still choosing locally to not improve my epistemic state. An AI might have a strict total ordering of terminal goals, but a human never will. So as a human, I can simultaneously strive for epistemic perfection and instrumental usefulness.

In any case, I still think there's a limit where the return on investment into epistemic rationality diminishes into nothingness, and I think that limit is much closer than most less wrongers think, primarily because what matters most isn't absolute rationality, but relative rationality in your particular social setting. You only need to be more able to win than everyone you compete with; becoming more able to win without actually winning is not only a waste of time, but actively harmful. It's better to win two battles than to waste time overpreparing. Overfocusing on epistemic rationality ignores the opportunity cost of neglecting to use your arts for something outside themselves.

You'd better remove Scott's real last name from your post before search engines index it, because he doesn't want it to be easy to find his blog given his full name.

Agreed, but unless a bunch of other things that have been there for ages get removed it's never going to take much effort to do. (E.g., Scott might want to look for ways to make typing in his real name not produce the top Google result that it currently does.)

Raikoth.net only links to his old blog. AFAICT none of the results in the first page allow someone who doesn't already know where his new blog is to find it in less than five minutes, and he isn't trying to make it impossible, only inconvenient.

(Edit: OTOH raikoth.net does show his new pen name, and someone might google it, but I don't think it'd occur to Aunt Tillie to do so.)

Would an apt summary be "Expertly used Dark Side techniques have a high local maximum of instrumental rationality, but there is a region of higher instrumental rationality that involves epistemic rationality techniques that are incompatible with Dark Side techniques"?

Being wrong about something may harm you in the long term. Being right when others are wrong can get you killed right now.

Not sure how exactly this relates to the article (maybe it doesn't), but I feel weird when this obvious part is missing from a debate about instrumental rationality. As if there is just me and the universe, and if I have the correct beliefs, the universe will reward me, and if I have incorrect beliefs, the universe will punish me, on average. Therefore, let's praise the universe and let's have correct beliefs! I agree that if I were a Robinson on an empty island, trying to have correct beliefs would probably be the best way. But most people are not in this situation.

It is a great privilege to live in the time and space when having the right beliefs doesn't get you killed immediately. It probably contributes to our epistemic rationality more than anything else. And I enjoy it, a lot! But it doesn't mean that the social punishments are gone completely. Even in the same country, different people live in different situations, so probably an important strategic move in becoming more rational is to navigate yourself in situations where the punishment for having correct beliefs is smaller. If you can't... then you play by the more complex rules; the outcomes of epistemic rationally may be smaller, and you might need some doze of Dark Arts just to survive. (And by the way, this is the situation we are optimized for by evolution.)

Uhm... not sure where I wanted to get by saying this. I guess I wanted to say that "epistemic rationality is the best way to win" depends on the environment. In theory, you could have epistemically correct beliefs and yet behave in public according to other people's wrong beliefs and expectations; but I think this is rather difficult for a human.

Lying constantly about what you believe is all well and good if you have Professor Quirrell-like lying skills and your conscience doesn't bother you if you lie to protect yourself from others' hostility to your views. I myself lie effortlessly, and felt not a shred of guilt when, say, I would hide my atheism to protect myself from the hostility of my very anti-anti-religious father (he's not a believer himself, he's just hostile to atheism for reasons which elude me).

Other people, however, are not so lucky. Some people are obliged to publicly profess belief of some sort or face serious reprisals, and also feel terrible when they lie. Defiance may not be feasible, so they must either use Dark Side Epistemology to convince themselves of what others demand they be convinced, or else be cursed with the retching pain of a guilty conscience.

If you've never found yourself in such a situation, lucky you. But realize that you have it easy.

Lying constantly about what you believe is all well and good if you have Professor Quirrell-like lying skills and your conscience doesn't bother you if you lie to protect yourself from others' hostility to your views.

Even then, it's more cognitively demanding to lie. It's like running a business with two sets of books -- the set you show to the IRS and the set you actually use to run the business. It may save you a lot in taxes but you still have to spend double the time keeping your books.

Agreeing with the people around you isn't demanding. And most people don't need to maintain any "true" beliefs about politics, religion and philosophy. They butter no parsnips in practice, and parsinip-buttering beliefs are not varied or unpredictable enough for ingroup signalling purposes.

Agreeing with the people around you isn't demanding.

I would say it depends on whether you really agree with them or not. If you believe X and you are surrounded by people who believe Y, and you need to conceal your belief in X, then you constantly have to be asking yourself "what would someone who believes in Y do or say?"

And most people don't need to maintain any "true" beliefs about politics, religion and philosophy.

I'm not sure what it means to "maintain 'true' beliefs." If you go through life, you will naturally develop a mental model (at least one, I suppose) of how the world works. If that model contains an Almighty Creator, then you are a theist. If it doesn't, then you are an atheist. Perhaps there is a third possibility, that your model is uncertain on this point, making you an agnostic.

If you are an atheist or an agnostic, and you are in a time or place where everyone is expected to be a theist, especially anyone who wants to get ahead in life, then that's a potential problem. Agreed?

I believe your first point us answered by my second.

You don't need mental models involving God or not god for any practical purpose .. other than solidarity with your community.

If you are one of the people, typical on LW but not in the population at large, who like to have beliefs on the "big" but practically unimportant questions, you will find dissimulation difficult. If not, not.

A man was telling one of his friends the secret of his contented married life: "My wife makes all the small decisions," he explained, "and I make all the big ones, so we never interfere in each other's business and never get annoyed with each other. We have no complaints and no arguments." "That sounds reasonable," answered his friend sympathetically. "And what sort of decisions does your wife make?" "Well," answered the man, "she decides what jobs I apply for, what sort of house we live in, what furniture we have, where we go for our holidays, and things like that." His friend was surprised. "Oh?" he said. "And what do you consider important decisions then?" "Well," answered the man, "I decide who should be Prime Minister, whether we should increase our help to poor countries, what we should do about the atom bomb, and things like that."

You don't need mental models involving God or not god for any practical purpose .. other than solidarity with your community.

I disagree with that. For example, suppose you are hunting in the woods and you find 10 gold coins. According to your village elders, Crom the grim gloomy unforgiving god commands that you donate any such windfall to the Village Shrine to Crom, and that to do so will guarantee you eternal paradise. And that to fail to do so will guarantee eternal damnation.

If your mental model of the universe includes Crom the grim gloomy unforgiving god, then of course you will make the donation. Otherwise you are likely to keep the windfall to yourself. Of course a decision must be made.

Of course you might object that those days are gone, that nobody is expected to follow religious precepts anymore, at least not in the United States. And I would disagree with that too. In today's United States, you must still decide where to live and whom to do business with. Does your mental model of the universe include the fact that certain groups are more prone to crime and disruptive behavior than others? If so, you would be wise to have a rationalization in mind for why you don't want to live anywhere near such groups. Or at least a few euphemisms.

Anyway, please answer my question from before:

If you are an atheist or an agnostic, and you are in a time or place where everyone is expected to be a theist, especially anyone who wants to get ahead in life, then that's a potential problem. Agreed?

Otherwise you are likely to keep the windfall to yourself.

Unless your model of the world includes people ostracizing you for doing so.

Unless your model of the world includes people ostracizing you for doing so.

I completely agree, but you are kinda fighting the hypothetical here.

You don't need mental models involving God or not god for any practical purpose .. other than solidarity with your community.

Oathes do work as a commitment device if you think that the God on which you swear is real and will punish you really exists. No automatic tracking like Beeminder, but still a decent alternative.

But such punishment is even further away in time than staying fat or failing the exam, so if the latter can't motivate you to diet or study...

Not talking about religion, politics and sex is position that's acceptable in many places.

Being an atheist is also an identity label. You don't need an identity label to have accurate beliefs. If you label yourself as an atheist than you will feel uncomfortable doing certain to participate in certain religious rituals because your family expects you to be at church.

If you just don't believe the ritual becomes a silly game that won't make you uncomfortable.

How skillful you need to be at lying depends on the culture you're in and the personalities of the people you're surrounded by.

Some cultures leave a lot of room for hypocrisy.

I myself lie effortlessly, and felt not a shred of guilt when, say, I would hide my atheism to protect myself from the hostility of my very anti-anti-religious father (he's not a believer himself, he's just hostile to atheism for reasons which elude me).

Hm, an atheist who hides his atheism, from his father who also seems to be an atheist (aka non-believer) but acts hostile towards atheists? Just out of curiosity, do you also act hostile towards atheists when you're around him?

I think you've identified a special case of a more general problem, which is that true beliefs do not have equal value, and that their values can vary wildly with your circumstances. To borrow blacktrance's example: if you're living in 6th-century Rome then it's useful to know that Jews aren't inherently evil...but it's more useful to know what happens to people who say so. And if you don't know how to profess that Jews are inherently evil without being corrupted by that lie, then it's more important to learn that than it is to believe true things about Jews.

This discipline, of predicting the value of information before you've learned it, is very difficult. For me, it's the most difficult thing. But it's also the center of the art; if it weren't, we could all level up endlessly by browsing Wikipedia.

As if there is just me and the universe, and if I have the correct beliefs, the universe will reward me, and if I have incorrect beliefs, the universe will punish me, on average. Therefore, let's praise the universe and let's have correct beliefs!

It is just you and the universe. "Other people" are a part of the universe.

(I actually kind of agree with you, though - the larger point is that your beliefs can impact outcomes directly rather than only via predictions. A non-sentient example of this would be Placebo effects. This seems not to have been included in the OP's discussion.)

The concept is called 'ketman' -- that term was popularized by Czeslaw Miłosz, who wrote about its practice under Communism.

I'm not sure if the pressure comes from lying per se -- it's not as if the practice is recent or uncommon -- or from having no place to go where you can escape the necessity to lie. Dalrymple was on to something when he said that the purpose of forcing public profession of the official idea under Communism was to humiliate; any place to tell the truth is a blow against the regime's goal of humiliation. Underground acts of non-public defiance aren't a new concept.

Secret societies aren't a new concept either; they don't seem to be as common anymore as they once were (but then again, how would I know?), but that's because they've been replaced by open but obscure/anonymous pseudosocieties online.

But there's a problem with the act of practicing ketman and going underground. Say you get n utility from having a secret society or similar, having an outlet to assert the truth outside the watch of the authority demanding that you lie -- but you'd get n^2 utility from getting the official lies dethroned. But you'd lose a great deal of utility if you got caught not believing in the lie.

That's a difficult coordination problem, since you clearly can't dethrone the official idea yourself. Perhaps it is deserving of study.

I'm not sure if the pressure comes from lying per se -- it's not as if the practice is recent or uncommon -- or from having no place to go where you can escape the necessity to lie.

I believe it's the latter. On emotional level, if I can't speak openly with a person, I have a feeling like they "don't belong to my tribe", they are a stranger. There is a difference between being sometimes with strangers, and being alone among strangers, all the time.

It is much easier to have clear rules about when to use my "public" face, and when to relax and be myself. Using my "public" face increases my internal pressure; I need a place to talk about it and relax. If I don't have that place, then I will lose attention in random moments, and expose my internal heresies. It is easier to keep control, if I have clear boundaries for when the hypocrisy begins and when it ends.

Having just one person to talk honestly with already helps a lot. (I am tired to google now, but there is probably some article on LW about how the first voice of dissent is most important.) It is much easier for me to think, if I can talk. Talking makes my thought processes much clearer. Not having a sane person to talk with is like not having a part of brain, or for a more realistic analogy, like being drunk or exhausted all the time.

Having correct beliefs does not mean expressing them. If I traveled back in time to medieval Rome, I would still believe that Jews aren't inherently evil and that Christ did not rise from the dead, but it would be unwise for me to be too public about those beliefs.

Nickpick: my understanding is that even in medieval Rome a lot of people didn't consider Jews inherently evil. At least to the extend that they were willing to engage in business dealings with them.

Ultimately, I think beliefs are inputs for predictions

As Robin Hanson has pointed out, beliefs are also a way of showing something about oneself. Tribal membership, moral superiority, etc. A good Cimmerian believes in Crom, the grim gloomy unforgiving god.

Often, when we attempt to accept contradictory statements as correct, it causes cognitive dissonance--that nagging, itchy feeling in your brain that won't leave you alone until you admit that something is wrong.

My impression is that most people never admit that their beliefs are contradictory, instead they either lash out at whoever is bringing the contradictions to the forefront of their mind or start ignoring him.

But I was wrong. And that mattered. Having accurate beliefs is a ridiculously convergent incentive. Every utility function that involves interaction with the territory--interaction of just about any kind!--benefits from a sound map.

Can you give three examples of improvements in your life since your epiphany?

Caring about contradictions signals geekishnes, which is generally undesirable.

Pointing out contradictions is generally seen as an attack, an attempt to lower status, rathernthan as something neutral or positive. Rationality and knowledge are high status as end states, for all that what you have to do to get ther is seen as low status nerdishness.

The ultimate in high status is effortless omniscience,as displayed by James Bond, who always knows everything about everything from nuclear reactors to the international diamond trade without ever reading a book.

signals geekishnes, which is generally undesirable.

Um. Undesirable to whom and for what?

I don't believe this to be true.

We might have a different concept of geekishness, though.

How many derogatory memes (in the internet sense, pictures with words on them) exist about Warren Buffet compared to Bill Gates?

You can't deny that one of the two is easier to laugh at. You might believe this to be morally wrong or undesirable for other reasons, but it seems to be obviously true.

Warren Buffet compared to Bill Gates? You can't deny that one of the two is easier to laugh at.

Let's throw in another non-geek/geek pair: Justin Bieber and Mark Zuckerberg.

You can't deny that one of the two is easier to laugh at.

I suppose I must just not frequent the right corners of the internet, because I can't remember the last time I saw a derogatory internet-meme about either of them.

And the things I can recall Gates getting laughed at for are mostly geeky inside-baseball. For instance: "640K should be enough for anybody" -- he got laughed at for saying that (even though, so far as anyone can tell, he didn't ever actually say it) but mostly by other geeks. I've seen him admired for being very smart, admired for being a shrewd businessman and buliding a hugely valuable company, excoriated for giving the world a lot of bad software, hated for shady business tactics, laughed at for things other geeks find funny -- but I really can't think of any occasion I've witnessed where he's been laughed at for being geeky. Perhaps I just have friends and colleagues who are too geeky, and all these years the Normal People have been pointing and laughing at Bill Gates for being a geek?

I'm sure there's been a lot more said and done (positive and negative) about Gates than about Buffett, because Gates was founder and CEO of Microsoft -- a company whose products just about everyone in the Western world uses daily and many people have strong feelings about, and that engaged in sufficiently, ah, colourful business practices to get it into hot water with more than one large national government -- and Buffett, well, wasn't. Can you, off the top of your head, think of three things Microsoft has done that you feel strongly about? OK, now what about Berkshire Hathaway?

So, I dunno, maybe Gates is easier to laugh at because he's geekier, but it seems to me there are other more obvious explanations for any difference in laughed-at-ness.

And the things I can recall Gates getting laughed at for are mostly geeky inside-baseball. For instance: "640K should be enough for anybody" -- he got laughed at for saying that (even though, so far as anyone can tell, he didn't ever actually say it) but mostly by other geeks. I've seen him admired for being very smart, admired for being a shrewd businessman and buliding a hugely valuable company, excoriated for giving the world a lot of bad software, hated for shady business tactics, laughed at for things other geeks find funny -- but I really can't think of any occasion I've witnessed where he's been laughed at for being geeky.

The Simpsons comes to mind as mocking Gates for being geeky, and I'd suggest that Gates gets mocked more than Buffett (I struggle to think of anyone mocking Buffett except Bitcoiners recently after he criticized it); that said, Gates gets mocked a lot less these days than he did in the '90s, and your inability to think of many examples is due to the disappearance of '90s popular media, magazines, Usenet posts, ./ comments, etc, from consciousness.

Yep. Note the mockery of Gates's monotone voice, arrogance, aversion to personal contact, overuse of computers...

a lot less these days than he did in the '90s [...] the disappearance of '90s popular media [...] etc., from consciousness.

Yes, very likely. (Recall that my original answer was: "Neither, these days." -- emphasis added.) I'm pretty sure I haven't seen the Simpsons episodes in which Gates is mocked, so of course I can't comment on them.

Again: if -- as is very likely the case -- Gates has been mocked much more than Buffett, it seems clear that there are plenty of explanations for this that have nothing to do with his being geekier. I don't see how Gates v Buffett can possibly be much use as an example of how geekiness is undesirable, given all the other factors in play there.

To be fare, I suspect a large number of the anti-Gates memes are by other geeks fighting the open/closed source holy war.

Geeks have most likely absorbed the "geeks are lesser, should be laughed at" meme to a certain extent as well.

Who's more prominent, Bill Gates or Warren Buffet? Yes, Bill Gates gets made fun of more, but he gets more attention in general.

If Bill Gates is mocked more than Warren Buffett, there are other, arguably more plausible reasons for this. Anecdotally, the most frequent cause I've encountered for criticizing or mocking Bill Gates is a dislike of his company, its products, practices, and prevalence.

Making fun of a high status person is a compensating action by low status people. Which person is made fun of depends more on the availability of trivia about that person than on their accomplishments (and geekiness surely is one such trivia). Also at the status high end the variance in any dimension is probably high.

Most trivia aren't funny. Simultaneous high and low status is funny. Dumb sports stars are another example.

Making fun of a high status person is a compensating action by low status people.

(And even if it isn't you will tend to be well served by claiming that is what the behaviors mean. Because that is the side with the power.)

I'm not sure I agree with Yvain's post.

One issue, with the abortion example:

Moldbug later uses the example of pro-lifers protesting abortion as an example of an unsympathetic and genuinely powerless cause. Yet as far as I can tell abortion protesters and Exxon Mobile protesters are treated more or less the same.

Well, there are laws limiting the ability of pro-life activists to protest outside abortion clinics. There are no analogous laws for Exxon Mobile.

His claim about how social power can't overcome structural power is dubious. Tell that to Mozilla co-founder Brendan Eich or GitHub co-founder Tom Preston-Werner. To be fair to Yvain both these incidents happened after the article was written and it appears he has at least moved in the direction of updating on them.

Also Yvain says:

Social power is much easier to notice than structural power, especially if you're not the one on the wrong end of the structural power.

This is pure BS. Structural power is very easy to notice, look at the org-chart. It is social power, as Yvain defines it, that is much harder to notice.

Which one do you mean, social power or structural power?

I mean power. The ability to significantly influence decision relevant outcomes without excessive cost to self. The statement doesn't care where the power is derived and it would sacrifice meaning to make either substitution.

The ultimate in high status is effortless omniscience,as displayed by James Bond, who always knows everything about everything from nuclear reactors to the international diamond trade without ever reading a book.

If James Bond wandered into a discussion of jewelry and started pontificating about the international diamond trade, I wonder if it would be seen as high status or low status.

Can you give three examples of improvements in your life since your epiphany?

Sure!

1) My conversations with friends are more efficient illuminating. 2) I learn more quickly from mistakes. 3) I prevent more mistakes before they get the chance to happen.

If I hadn't given those examples, could you have predicted positive changes resulting from having generally more accurate beliefs? It really doesn't seem that surprising to me that someone's life would improve in a zillion different ways if they weren't wrong so much.

1) My conversations with friends are more efficient illuminating. 2) I learn more quickly from mistakes. 3) I prevent more mistakes before they get the chance to happen.

Well can you give specific examples of mistakes you learned more quickly from and/or prevented? And can you give an example of some illumination you got more efficiently out of a conversation?

If I hadn't given those examples, could you have predicted positive changes resulting from having generally more accurate beliefs?

Not necessarily -- peoples' maps of reality tend to be pretty good when important personal interests are at stake. Perhaps a good Cimmerian believes, in theory, that if he dies then Crom will instantly take him to eternal paradise. But somehow that doesn't stop our Good Cimmerian from expending a lot of effort trying to stay alive, possibly including breaking some of Crom's rules.

Also, it costs mental energy to make your beliefs more accurate and there is no guarantee that it will be worth the trouble to do so.

Last, as mentioned above, beliefs serve other purposes besides being inputs for predictions.

Thoughts on this:

Obviously it's possible to want multiple things and believe multiple things. My mind, at least, is best approximately as a society of sub-agents than as a single unified self. I think "System 1 vs System 2" is already too much of an approximation–my System 1 definitely isn't unified, and even my System 2 doesn't agree on a single set of beliefs.

Can you simultaneously want sex and not want it?

Yes, and even large amounts of luminosity haven't made this divide go away. I used to not want sex because it was unpleasant, but want to want it because it was a way to profess love and, damn it, I wanted to do that. The not-wanting-sex happened on a more basic, less endorsed level, leading to weird mental resistance and frustration whenever I overrode it and had sex anyway because it was a thing I ought to do. I now do almost the opposite–I listen to my System 1 instincts and don't have sex, but I'm not totally happy with this state of affairs. There's good evidence that humans can't change their sexual orientations, so I've accepted it for now, but if that status quo changed, I would have some rethinking to do, and might press a button to make it different. These are different 'file formats' of belief–System 2 verbal beliefs don't automatically propagate into System 1 visceral urges–but they're nevertheless contradictory, and years of thinking about and paying a lot of attention to the issue hasn't allowed me to resolve that.

Another example: I want kids. By that, I mean that seeing a baby makes me feel all warm and fuzzy inside; that I daydream about it; that the first thought that comes when I see or learn many things is "I'm going to teach this to my kids!" I'm also fairly sure that having kids now is not the correct thing to do. It may not be the correct thing to do for a few years. In this case, System 2 rules win out, while System 1 whispers quietly in the background that why don't I have a baby already, and hey, you could put up with some unpleasantness and have a baby in nine months. I'm sure as hell not going to change my System 1, but there is or is not an instrumentally rational thing to do, and what my System 1 wants is only a small part of the calculation. So, if all the other variables push me in the other direction, I might end up not having kids for a long time–and having a mental contradiction for the same length of time.

Is this inevitable? Maybe, maybe not. But it certainly seems to be the default, even for people who spend a lot of time thinking about their beliefs.

Well, I for one am confused much of the time, and whenever I encounter someone who ostensibly isn't, I get nervous. Believing falsehoods isn't just the domain of dark artisans, it comes courtesy of having a brain. My belief that "most of my beliefs must have large error bounds" probably has among my lowest error bounds; I'm surest about being unsure.

I do wonder if convincing oneself of having given up deluding oneself isn't the greatest dark side achievement of all -- after all, how would you know it's not? Maybe you got tricked by your System 0.

But I'm being contrarian. Good post overall. I guess the metric I'd prefer in terms of belief improvement is "number of times I've noticed my confusion and bent myself to accept what I perceive, instead of bending my perceptions to myself". More of an engineering approach, still allowing for a few holy cows that don't get slaughtered (without overly compromising your overall strength as a rationalist).