This linkpost is in part a response to @Raemon's comment about why the procedure Raemon did doesn't work in practice to deal with the selection effects I talked about in my last post.

So in the last post, I was talking about a selection effect where believers of an argument can come to believe that their idea is true and that their critics are crazy wrong/trolling/dumb, no matter what argument is used.

And the believers are half-right, in that their random error swamps any truth signal/evidence for most critics due to bounded computation for most topics, but they incorrectly perceive that as evidence that their theory is correct, because they don't realize that most possible critics of an idea will have bad criticisms independently of whether your claims are correct.

This was part of what I was trying to get at when I said that selection effects are very, very hard to deal with using only arguments, because once you can't rely on the general public to know whether or not an argument is true, it becomes much, much easier to create bubbles/selection effects that distort your thinking, and because you can't have any other sources of grounding other than arguments, this can easily lead to false beliefs.

In practice, the way we generally ameliorate selection election effects is either by having ground truth feedback, or by having the subject matter be easy to verify like in mathematics, physics, and more domains, such that others not of your ideological bubble can correct you if you are wrong.

Absent these, we have fields that confidently produce a lot of nonsense that's clearly ideological like nutrition/health studies, sociology, psychology and more fields of science, though there are other problems in these fields, but selection bias is a big contributor.

Andy Masley has some tools at the end of the linkpost to help you be able to avoid selection bias more generally:

I’ve seen people around me and in the general public turn ideologically crazy at a higher rate than I expected to in my adult life. I lost one friend to a cult. Others became so brittle that they can barely interact with normal people without freaking out over tiny disagreements. Others have just become very ideologically single-minded and attached to very specific (and in my opinion not very convincing) accounts of the world. These people’s backgrounds, and the ideologies they went crazy over, were all wildly different from each other. But underlying all of them is a single really bad dynamic that I think is actually easy to identify and stop. The dynamic goes like this:

You develop an extremist idea.

You bump into a lot of people who disagree with the idea.

Almost every single person who disagrees with you comes off as wildly misinformed about basic facts about the world, and have pretty horrible background moral intuitions about simple ethical questions. Their disagreements are also very visible entirely rooted in their pre-existing social identity.

You conclude that the vast majority of critics of your extremist idea are really wildly misinformed, somewhat cruel or uncaring, and mostly hate your idea for pre-existing social reasons.

This updates you to think that your idea is probably more correct.

My claim here is that the only thing wrong here, the thing that really messes people up, is that last step where you update in favor of your idea.

I’m not against extremist ideas on principle. I have my own, the main one being that animal welfare is a much larger urgent catastrophe than most people believe. But I think extremist ideas should come with a few warning labels.

The main danger with believing an extremist idea is that it’s very very easy to be negatively polarized by a ton of bad conversations with normal people about it, until you feel like the whole rest of the world is crazy and asleep. The reason it’s so easy is that most people are in fact “asleep” about most things most of the time, not super thoughtful in how they develop beliefs, and really only have expertise in a few areas often related to their jobs.

Suppose you become politically radicalized and become a Marxist. You get convinced that the labor theory of value is correct. You learn all about the pantheon of Marxist theorists through the 20th Century. You really dig in and know all the arguments and concepts by heart. It coheres with what you see on the ground: poor people pointlessly treated badly by big systems, a lot of band-aid solutions to poverty that leave workers without authentic power and control over their lives. You update more and more in favor of Marxism. Then you start talking to other people about it.

Consider two categories of people:

The number of people in the general population who will object to Marxism.

The number of people in the general population who will know enough about the history and philosophy of economics to give you a clear account of why Marxism and the labor theory of value is incorrect, and can speak to your very specific concerns about worker power and class conflict in a way that gets across how modern social theorists and economists give a more compelling and ethically coherent account of the way the world works, and do so patiently and thoroughly.

How many people will you meet who are in both 1 and 2, vs just 1? I think it would probably look something like this:

https://substackcdn.com/image/fetch/$s_!OCbw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd7c69bc-07c3-417d-a7f1-04a27df6408f_960x560.png (The LW version of the linkpost will only get the link, not the photo, since I can't right click to directly paste an image in LW docs, and Markdown doesn't work for putting in images, and I can't move images from one tab to another).

With no offense meant to any group listed here, the average person in America didn’t complete college (only 37% have a degree). Fewer than half read a book in the last year. Two thirds don’t know that when inflation happens, money loses value. Many people adopt beliefs because it seems like the people they think of as good adopted them too. What percentage of these people can adequately give a convincing account of why liberal democratic capitalism is preferable to Marxism? Most Marxists I meet have a ton of stories of people they’ve met saying things like:

• Some people need to be rich, and others need to be poor. That’s just the way it is.
• Poor people deserve to be poor. If they worked harder they wouldn’t be having these problems.
• Marxism means everyone needs to be equal, but if everyone were equal no one would be…

This leaves them feeling like most people who criticize Marxism are uncaring, dumb, or crazy, and therefore that Marxism is more likely to be true, and people’s other ideas are more likely to be false. I remember one radical friend was pretty shocked to find out that I identified as a liberal capitalist, and brought up “I knew someone who identified as a liberal a while ago. He didn’t even know what mutually assured destruction is.” I paused the conversation to ask my friend if he thought I didn’t know what mutually assured destruction is. He had some trouble disconnecting that other guy from me, or from the concept of liberalism, or his own belief in Marxism.

My core claim here is that most people, most of the time, are going to be terrible critics of your extreme idea. They will say confused, false, or morally awful things to you, no matter what idea you have. This tells you absolutely nothing about the value of your idea, or the value of the ideas they believe in, and it should not cause you to become more confident in your beliefs at all. The average representative of the opposing side is almost always going to be pretty terrible, but that’s because the average person just doesn’t know much about your hyper specific thing, not that the other side is terrible. The average person (including you and me) is outrageously confused about most things. This means if you take on any idea that stands out as weird and draws a lot criticism, most of that criticism will be wildly confused too. Marxists are right that almost all critics of Marxism give terrible arguments, are confused, and seem to have sometimes reprehensible views about class and poverty and capitalism, but that’s only because a lot of people do. (Note here that I disagree with Andy Masley about whether moral views can be reprehensible as an absolute judgement, rather than being a moral relativist ala postmodernism, but the point stands independently of that issue).

This happened to me when I was 17 and beginning to identify as solidly left-wing. One day a 19 year old who I knew was very conservative was reading the news and said “Oh my God, Obama just signed a deal with Russia to get rid of a lot of our nuclear weapons. This is terrible.” I asked why that was bad, because Russia was also destroying its nuclear weapons as part of the deal. The guy got mad and said very sternly “How are we supposed to fight terrorists?” I decided the conversation wasn’t worth it, and came away thinking “Man conservatives are so stupid, this is more evidence that conservatives are stupid and I’m right to not be a conservative.” This was a terrible way to think. I know now that nuclear weapons policy is incredibly complicated, experts who spend their whole lives studying it disagree with each other, and it’s rare to find members of the general public who know enough to have a serious conversation about nuclear weapons. Nuclear weapons policy doesn’t really map onto a clear left-right axis anyway. If a thoughtful, knowledgeable conservative had challenged me on my beliefs about nuclear weapons, I would have probably said something equally or even more stupid. One guy I bumped into happening to know nothing about nuclear weapons should have counted as zero evidence for me about nuclear weapons, or left vs right-wing politics. A lot of people are ideological roulette-wheels adopting random beliefs that pop up in the random political tribes they identify with. I had bumped into one of those people. That was it. I had done the bad thing: updating positively on my beliefs when I heard a bad argument against them.

If you’re a vegan like me, you’ll bump into a lot of people who are pointlessly aggressive and have stupid counter-arguments to veganism (Plants have feelings too, killing is a normal part of nature and evolution, we know for a fact that only humans are conscious, etc.). This can make you feel like all anti-vegans must be equally stupid or blind, and the only people who feel even vaguely awake and intelligent are other vegans. The world outside of your community of vegans can look less interesting, and you’ll assume everyone who “really gets it” is a vegan. This is a very dangerous emotional place to end up! There are actually many compelling arguments against veganism, including many made by committed animal welfare activists. But many vegans I meet don’t seem aware that any exist at all.

In these situations, what the believer in the extremist ideology should do is assume “Most people are going to disagree with me about this. I won’t find many good arguments about this with most people. What I should do if I actually want to expose my belief to criticism is try to find the most compelling, thought-out criticism of the belief that I can and work with that. I should also assume that if I were having conversations with people who agree with me about the extreme belief, I’d still see them as ‘asleep’ about a lot of other stuff in life. There are maybe vistas waiting for me beyond my field of vision. I should be careful to know if they exist.

Thinking this way matters more than ever, now that the internet makes it so much easier to learn a lot about big ideologies and ideas very fast.

Be slightly elitist

What I’m saying is that you should ignore a lot of people’s thoughts on your extremist beliefs. How should you decide who to engage with? I think there are just two basic rules that will always clarify who’s worth listening to:

The person knows basic facts about what you’re disagreeing about, like

• Topic: Politics
• Fact to know: The 3 branches of government
• Topic: Marxism
• Fact to know: The basic idea of the labor theory of value
• Topic: Nuclear weapons
• Fact to know: Nuclear weapons exist in most countries purely to deter other countries from invading/using their own nukes
• The person is not super fired up and doesn’t just want to act as a soldier for their team

This is an elitist rule in the sense that I don’t think many everyday people you bump into will actually clear these two requirements. I’d guess that these rules would end a lot of conversations on their own and leave you free to only engage with people who will actually give you good, deep criticism. If you can’t find those people in the real world, turn to the internet and find them there.

Instead of associating who’s “awake” with your ideology, you should think of “awakeness” as a quality anyone with a careful interest in figuring out what’s actually true has. Assuming that anyone with reasonable beliefs can be “awake” in this way (but that it takes some effort and emotional patience) can make the world seem open and alive again, and make your ideology seem less like a solitary island home in a sea of insanity and more of an interesting puzzle you’re trying to solve with your critics. You should try to keep your sense of awakeness and your specific ideological beliefs as disconnected from each other as possible. This ability to separate specific beliefs from who someone considers to be awake is itself something that makes me experience that person as “awake.”

The sense that the person you’re speaking to is “awake” is intoxicating. It’s a reason people form communities around their belief systems. Feeling like you’re with someone who’s a live player in the world can give you a deep sense of company, a core human need. Straying from your belief system to get that sense of company elsewhere can be scary and intimidating, especially if you haven’t done it for a while. If you’ve recently only really felt that sense of company from people who agree with you about a lot of big core stuff, it’s time to stray.

But in straying, don’t update on the value of your or others’ beliefs based on your interactions with the average person.