For part one of the aspirant sequence - which may or may not be arranged into some totally different order when I'm done with it, because the connection here won't be obvious yet - see Would you work harder in the least convenient possible world?
Two groups of rationalists live in houses across the street from one another, as is tradition in San Francisco.
One Monday night, in the Purple House, everyone is having a debate about the prison abolition movement. Alice thinks that prison abolition is an irresponsible fantasy which would lead to murderers and arsonists running around doing whatever crimes they wanted. Bob thinks that prisons are a moral horror equivalent to slavery and that Alice doesn't care about the human rights of prisoners. They get quite heated for a few hours, then their housemate Charlie reminds them that they're rationalists and that they should identify a crux. It turns out that the crux is whether, in the absence of prisons, the crime rate would go up a lot. Alice and Bob can't test this - they can't get rid of prisons and see what happens - so they both go to bed frustrated.
In the Orange House, Deanna and Ethan are debating something terribly boring and unimportant. Deanna thinks that cutting behind the Walgreens is a shortcut to the metro station. Ethan thinks that, because that route sometimes results in waiting at an extra traffic light, it actually takes longer on average to walk that route than to simply go through the park - and the park has the nice bonus of getting to walk through some pleasant grass. Their housemate Francis suggests that they could just go and test this, so they take twenty minutes out of their day to go outside and resolve their argument. They both walk to the station on their preferred route, and Deanna arrives three minutes earlier. They then walk to the junction with the traffic signal, and time it. They discover that, if Deanna had been held up at the traffic signal, it would've added about 90 seconds extra to her time - so Deanna's route is faster if she's held up at the traffic signal on less than half of walks, and Ethan's is faster if it's more than half. On the walk home, they realise that Ethan is actually a slightly faster walker than Deanna, and they didn't control for this during their initial test; if they'd swapped and walked each other's preferred routes, Ethan would probably have beaten Deanna by more than three minutes. Also, the traffic signal is 90 seconds' delay for a full cycle - but if they had arrived midway through the stop, they would've had to wait for less than 90 seconds. Ethan therefore changes his mind and acknowledges that Deanna's route is a genuine shortcut - and Deanna decides the time saving is small enough that she'll take Ethan's route when she's not in a rush and doesn't want to smell the bins behind the Walgreens.
On Wednesday night, in the Purple House, everyone is having a debate about immigration[1]. Charlie supports open borders, and Bob thinks controls on immigration are necessary to protect workers' wages. Alice reminds them that they're rationalists, and should be using science and data in their debate - so Charlie shares several scientific studies that show that immigrants contribute more in taxes than they cost in welfare, are less likely to commit crimes than home-grown citizens, and that immigrants actually long-run increase wages for native-born citizens. Bob reads one study and has several criticisms of its statistical methods - and regardless, he argues, the fact that current strictly-controlled immigration is positive doesn't prove that hypothetical unlimited immigration would be equally positive. He cites that refugees in Germany are much less positive for Germany than H1B workers are positive for America, showing that it is necessary to apply some selection effect to get positive immigrants. Charlie argues that refugees are actually negatively selected - if your country only grants asylum to people who have been through sufficiently terrible things, you'll mostly get very poor and disabled and traumatised people who are less likely to be able to work and more likely to perpetuate cycles of abuse. Neither can find a study that the other agrees is applicable and well-designed. Eventually Alice tells them that if neither of them is likely to change their minds, they should just go to bed.
In the Orange House, Francis is focused on a rather mundane and unimportant-seeming project. They have gotten frustrated with the cupboard organization system; there's a giant cupboard that they use to store cans of food, but stacking the cans so high means that they rarely look at any of the cans on the bottom of stacks in the back of the cupboard. They have bought some plywood and are planning on putting some dividers into the cupboard so that it is easier to sort the cans categorically - beans in one section, tomatoes in another, soup in a third - which should remove the need to actually look at all the cans. Deanna has concerns that the plywood won't be strong enough to actually hold all the cans; it doesn't look very strong to her, and she's worried about glass jars smashing when it gives way. Ethan reminds them that they are rationalists, and they have phones in their pockets, so they go to Google to find out how strong plywood is. Deanna is surprised to learn that plywood is actually generally stronger than regular wood, because wood is strongest across its grain and plywood is made by gluing together sheets with grain going in alternating directions. Francis finds that the 3/4" thick plywood they bought should be able to hold at least 50-100 pounds and most soup cans weigh around 11 ounces, so they write a tiny "maximum 72 cans" label for their can shelf. Ethan suggests that they might even be able to rig up a wine-rack-style criss-cross system so that every type of can could be visible and accessible, but none of them know how to apply glue or screws in the correct way to make that sort of thing hang together; they can't put a nail straight through to attach two pieces without that nail awkwardly poking out of the other side. The three of them spend the rest of the night experimenting with plywood, and eventually invent a method of creating a joint by wrapping wire through some adjacent drilled holes, which they use to build a nice-looking spice rack.
On Friday night, at a house party in the Purple House, Alice and Charlie are arguing about veganism. Alice is a vegan because she is concerned about animal rights, and Charlie isn't a vegan because they don't think animals are sentient. Alice brings up all sorts of scientific evidence regarding observed animal behaviours like mourning and learning, studies on animals' brain activity, numbers of neurons in various animal brains, studies where animals chose to consume pain relief drugs, and the list of animals that can pass the mirror test. Charlie presents arguments that sentience is fundamentally connected to complex human behaviours like language use, and Alice asks if that means LLMs are sentient. Bob steps in to tell them that if they want to talk about AI at a house party then they need to go to the designated Arguing About AI room, where AI arguments get exiled so that the rest of the house can talk about literally anything else.
In the Orange House, Francis and Ethan are also debating veganism. Francis is a vegan. Ethan doesn't want to be a vegan because he thinks vegan diets are less healthy and more expensive. Francis believes it'd be more convenient for the whole household if everyone was at least vegetarian, since it'd save on storage space for different varieties of food and also let them eat more meals together, so they ask Ethan what his happy price would be to try veganism for a week. Ethan is uncomfortable naming a price, and realises that part of his crux isn't just the price - he's had bad experiences with accidentally consuming soy (which he's allergic to) while trying meat-free alternative products. He eventually names a very high price, but offers a much lower price if Francis is willing to handle grocery shopping for definitely-soy-free vegan products, and an even lower price if Francis cooks. Deanna suggests they delay the experiment by a week, so that first of all Ethan can eat his normal diet for a week while recording what everything costs. Then they can compare the cost of the vegan food that Francis cooks in the next week, to actually test whether it's more expensive. They spend some of the rest of their evening setting up a spreadsheet that tracks grocery costs - and in the process they realise several other places they could be making savings.
Who learned more, over the course of their week? Who practised more key rationality skills - designing fair tests, noticing confounders, changing their minds, making bets, identifying cruxes, noticing when they're confused, solving their problems, cooperating, and putting specific numbers on things? Who do you expect to be stronger rationalists after a few years of living together?
Part Two: The Nature Of The Mindkiller
Politics is the mindkiller; there are certain topics that it's just very difficult to discuss productively, and "politics" is one way of gesturing at a set of those topics. Why is politics so difficult to discuss? Well, the answer usually goes... people are often deeply emotionally invested in politics, or have certain political ideologies wrapped up in their identities, or find it difficult to respond to the ideas in front of them without also responding to all the worst versions of those ideas that they've seen pushed by the other side. This is also true of religion and football - hence why "religion and politics" are considered two topics you just shouldn't bring up in the workplace or over Thanksgiving dinner.
But there's another diagnosis of what makes certain topics mindkilling; the inability to resolve them.
I might be deeply emotionally invested in a piece of engineering that I'm proud of, but if somebody comes along and demonstrates a more-efficient method to solve my problem, I'm usually capable of getting over my feelings and thanking them. Being an introvert might be part of someone's identity, but if someone points out that they'd get more job offers if they went out to networking events, they're usually capable of reasoning about whether that's true and pushing themselves to test it. I might have been given awful boiled-soggy Brussels sprouts by my parents many times, but I can push myself to try my housemate's amazing Brussels sprouts fried with bacon, and acknowledge that they're actually pretty great.
In all of those cases, there's an actual reward for doing the right thing. If I can get over my embarassment about someone else's code being better than my code, I can make my program run faster and more efficiently. If I can untangle introversion from my identity, I can get more job offers. If I can get over my prejudices about sprouts, I get to enjoy my housemate's excellent cooking.
I don't know about you, but I'm a pretty simple learner. I find it easier to learn new skills if I can test it out, see what happens, get a cookie if it worked, and then rapidly iterate. I get better at cooking by making a meal, being rewarded with tasty food if I did a good job, and noticing when I've made non-tasty food and being motivated to fix that for next time. I can also work with negative motivation - I get better at swordfighting by trying to swordfight, repeatedly getting hit, and then feeling motivated to block those attacks next time. But if I had to learn, say, programming, by just writing code out with pencil-and-paper and getting other people to give feedback on whether it looked good - and I couldn't ever run the code to see if it worked - I don't think I'd ever learn to code.
Politics and religion are topics famous for the inability to resolve them. We can't die, check to see if there's an afterlife, and then come back. We can vote in another party to see if they do a better job... but only once every several years, and no individual person can do that without other voters cooperating. We can't, realistically in our lifetimes, vote in a third party in the USA to see if they'd do a better job than the Democrats or Republicans. We can't do open borders for a few months just to see what it's like and then hit 'undo' if we didn't like it.
I don't think "emotional investment" or "identity entanglement" or "lots of previous experiences with terrible arguments" is the thing that makes a topic really badly mindkilling. The thing that makes a topic a mindkiller is when you cannot resolve it.
It's not just that you can't run experiments and get feedback - and so you're effectively going into battle unarmed, lacking your best rationality tools. It's also that there is no incentive to be truth-seeking. If I have a terrible opinion like "a diet of nothing but beef, pasta and crackers is optimal" then someone might challenge me to actually stand by my beliefs, and actually eat that diet, and then I will get scurvy and it will suck. What's the penalty for having a terrible opinion like "we should try absolute hereditary monarchy again"? People will tell me my opinion is terrible, but I'm never going to actually have to stand by my belief; I will never have to say "okay, we tried putting King Charles III in charge of the USA and it went kind of terribly for me and I got scurvy, sorry about that everyone, I was wrong and it sucked". I might lose social points because people think I'm wrong - but alternatively I might gain social points because people think I'm really interesting and fun to debate with.
And it's also that, because you can't run experiments to resolve your questions and you don't ever get outcomes that give you feedback, these debates can go on literally forever - and that allows them to gain unlimited complexity, drift an unlimited distance from anyone's crux, and build up unlimited resentment over time. They are limited only by your frustration and/or your need to go to bed. I can fantasise all I like, but I'm never going to actually get to the point where we all say, "Great work everyone on resolving transgender inclusion in sports! It's solved! Now that we never need to debate this again, let's move on to solving Israel/Palestine."
It is nearly impossible to have a rationality community where nobody discusses politics, because sometimes topics with important real-life consequences are political. Unfortunately, if we swing too far the other way and spend lots of time blogging about politics, there are horrible consequences; several rationalists I know have had to completely block all social media from their work computers because otherwise they’ll be nerdsniped into endlessly yelling on Discord, which actively makes them more biased and less productive. Also, frankly, a lot of the political discourse in the rationality community actively drives away women.
If your model of the mindkiller is that emotional topics are mindkilling ones, you're left with a terrible choice between "ban discussion of anything that actually matters enough to make people emotional" and "allow everyone to yell endlessly about politics". If your model of the mindkiler is that unresolvable and untestable topics are the mindkilling ones, you can talk about lots of the politics that matters - while also heading off unproductive discussion with "hey, everyone, this seems like a mindkiller", and/or a blanket rule against mindkilling topics in your Discord / comments sections / house parties / etc.
This model explains why there are plenty of highly emotional topics (like whether to try chemotherapy if you have terminal cancer, or whether your sports team is very likely to win their big game) which it is very easy to find good discussion of online, despite the fact that they are deeply emotionally upsetting... and there are plenty of fundamentally unresolvable topics (like which fictional characters should date/marry/kiss which other fictional characters) which produce some of the internet's worst discourse despite their complete lack of rational importance to anyone.
This also predicts that DC rationalists - many of whom work in policy - will have far more productive, useful and rational conversations about politics than San Francisco rationalists - who are mostly software people - and so far I've found that to be extremely true. I expect DC rationalists could also have some spectacularly unproductive and useless conversations about programming, if they were so inclined.
You could argue, of course, that it’s good rationality practice to hone our de-biasing truth-seeking skills on tricky topics like guns and gender and gossip so that we’ll find it really easy to be not-biased when we come to easier topics. Unfortunately, I’m pretty sure that Discourse makes your rationality worse rather than better. I think engaging in online debate about this sort of topic reliably makes me angrier, more biased, more committed to abstract theoretical ideas that I’ve never had to bet on, less collaborative and less productive. I think online Discourse can be addictive in a way I really don’t endorse. And fundamentally I think you’ll get better practice by building IKEA furniture without the instructions, or reading about historical stories without spoilers and trying to predict how it turns out, or going out into your garden and trying to identify all the plants, or taking stats classes.
So, how do we decide when it’s healthy to engage in a piece of Discourse, and when it’s not worth the mindkill penalty?
Part Three: The Poke-It-With-A-Stick Test
I ask myself, "Can I poke it with a stick?"
This is, somewhat deliberately, a phrase that is more evocative than it is specific. The stick is metaphorical - but you never want to let it get too metaphorical. Asking the question while imagining a literal, physical, actual stick in your hands is honestly a great way to get the basic idea. The stick represents some combination of: being able to test the idea, being able to resolve your argument, being able to get feedback with actions, being close enough to the topic to be able to observe it well, being able to affect the topic at all, the topic being relevant to your life, leaving open the possibility of taking future actions on the topic, etc.
Let's consider several examples. Here are some kinds of Discourse that my friends and online spaces have engaged in:
Billionaires; should we try to befriend them and convert them to Effective Altruism so that we can divert millions of dollars to effective causes and also recruit super-awesome genius innovators, or are they evil exploitative monsters who should be taxed out of existence and who all came into their fortunes through luck and privilege and evilness?
Justice; should prison be abolished, does restorative justice work, how important is rehabilitating prisoners, and is the death penalty immoral?
Veganism; is it a healthy lifestyle available to anyone which is also morally mandatory, or just virtue-signalling for rich hipsters?
Sexual assault; does it mostly happen to women, and how often do people lie about it happening, and what should you do if two of your friends are both claiming that the other friend abused them?
Guns; should they be regulated so they’re really difficult to acquire, and does that decrease the murder rate, and how safe is it to carry one?
Marijuana; is it safer than caffeine or alcohol or nicotine, and should it be legal, and is it the best thing since sliced bread?
Abortion; should you get one, should you be obligated to pay for your partner's abortion, should you be obligated to get one if you promised your partner not to have a baby, and should they be legal?
I think some of these can be engaged in productively by rationalists who are trying to improve their minds and their lives. I think others are wastes of time and endless causes of strife, and I prefer to nudge my social circles away from discussing them, when and where I can.
(I won't pretend I'm very good at successfully avoiding terrible mindkilling discourse - I am not! I am, in part, writing this in the hope that I'd be better at applying my stick-poke test if others around me could help to collectively enforce it.)
I think before you read on, you should take thirty seconds to figure out if you have any clear simple rules that you’d use to make this determination, which you might want to think about without being biased by the post and/or you might want to write in the comment section later.
Billionaires: I cannot poke a billionaire with a stick. I don't think I know any billionaires well enough that I could get close enough to poke them with a stick. I don't work for any EA orgs which are responsible for deciding whether to befriend some billionaires, so I doubt I could affect the decision of whether to do so, so I can't poke that with a stick.
I'm sure someone could dig up a study on whether billionaires donate more or less to charity than average people, or whether billionaires commit more or fewer crimes than average people, or whether billionaire-founded charities are more or less effective than average charities. This is important: possession of a study is not the same thing as possession of a stick. Someone else can easily dismiss your study. They can say that there's a bias where the billionaires who don't donate to charity are likely to be less-public figures and therefore less likely to show up in your data, or dispute whether we should be looking at the total amount donated by billionaires or the percentage of income, or point out that billionaires' crimes are mostly things like wage-theft that go under-reported and under-prosecuted, or say that your study has massive methodological issues. Then you're right back where you started.
Justice: I am nowhere near a prison and cannot poke one with a stick. I cannot poke any prisoners with a stick since I don’t think I know any prisoners. I cannot poke the death-penalty laws with a stick, since I am not a lawmaker or policy analyst and nothing I say or do is likely to change any of the death-penalty laws in any jurisdiction.
It's not that experiments can't be run in this domain; some country or state could change sentencing guidelines and we could see whether the crime rate goes up or down. However, this doesn't make the topic pass the poke-it-with-a-stick test; it's very easy for someone to say "well, sure, that worked in Norway but it'd never work in America because America's different" and then you're right back where you started.
These topics fail the poke-with-a stick test (for me). If we try to debate them, a few bad things happen:
We’ll come up with hypotheses, but we won’t be able to test them. My friend might assert that, if you did convert more billionaires to EA, then they’d say selfish exploitative things and give us a bad reputation, and we might end up with SBF round 2. My other friend might assert that actually, we’d get a good reputation because we’d divert millions of dollars away from luxury yacht purchases and towards bednet purchases, and everyone agrees that that’s a good thing. If we were able to convert a few billionaires to EA, then hand out a survey asking how people felt about that, then wait a few years and see if any of them became SBF round 2, then we’d be able to test what the truth is. Unfortunately since my friends have no plans to test this, they will just yell at each other all night until one of them gets very angry and storms out.
There won’t be a point to the discussion - a key action point that we’re trying to figure out - so there’s no clearly defined end-point, no deadline and no clearly defined goal. This means we’re unusually likely to go off on tangents that end up with someone on Wikipedia looking up citations for every stupid thing Elon Musk has ever done, someone getting into a furious argument about methodological issues in That One Study About Scandinavian Prisons, someone furiously defending a guy that got executed in the 1800s who they swear was totally innocent, and someone in the corner banging the Communist Manifesto on the table angrily. (This is totally okay if these people are having fun, but if they’re not then they should probably stop.)
If we’re being really biased or building totally wrong models of the universe, nothing will stop us from doing that. We could all debate for five hours, come up with a model of rehabilitative justice that we all happily agree would definitely be optimal, and feel a great sense of satisfaction that we finally came to an agreement. That sense of satisfaction, that insight-porn-y glow, is not a good thing; it’s not rewarding any ability for your beliefs to track reality, or any achievement in the territory, since we all know we’ll never implement our proposed system. We don’t fear, even for a second, that we might even be asked to implement the system. Your maps might all agree now, but there is absolutely nothing preventing your maps from all being even wronger than before, and absolutely nobody who will stop you from declaring a satisfying-feeling victory even when you’ve all just made your models worse.
People will get status points for making insightful-seeming points about these things, and lose status points for not really having thought much about their opinions on these things. Safe in the knowledge that they'll never be called upon to actually stand behind their beliefs, it's easy for people just to write some high-status-sounding clever bullshit. I don’t want to assign status points that way; I want my community to give status to people who do a lot of saving-the-world-related activities and remove status from people who waste their time debating politics on Discord.
This is not to say that nobody can debate these things because nobody has a stick; some people have sticks!
However, the people who do have sticks are often dealing with all the messiness and chaos and detail of reality. Maybe your dream ideal rehabilitative justice system runs into issues with a particular committee that needs to approve a particular kind of expenditure and zoning laws about prisons built in semi-urban areas, and the thing the actual people-with-sticks are working on is something to do with pressuring a particular committee member to withdraw their veto so they can change how judges see sentencing requirements in paperwork. Maybe the people who affect EA outreach decisions are busy worrying about the tax implications for a particular billionaire who wants to sell stock to donate to AMF but keeps being told he’ll void a lockout agreement if he does that. I don’t know! The people with sticks probably know those things, which is why I will first acquire a stick (or at least the possibility of having a stick, or a friend with a stick I can ask) before considering it a productive activity to debate this.
Veganism: I can't really poke an animal's brain with a stick to see whether it's a moral patient. No amount of experiments with mirror tests will tell me whether the mirror test is even a relevant test to questions of moral worth. However, I can absolutely poke some tofu with a stick. Specifically I can put the tofu in my mouth, and that is an easy way of empirically testing whether eating the tofu is good. If I decide veganism is good, I can go to the store and poke some vegan groceries with a credit-card-shaped stick. If I poke my stomach with a tofu-shaped stick and I get violent diarrhoea, that is a solid signal that maybe veganism isn’t for me.
Guns: I can’t poke gun laws with a stick. It’s not like I can ban guns completely in half of the world’s countries, completely deregulate them in the other half, and see who’s got better murder rates next year. Since I don’t work in politics, it’s not actually like I can affect gun laws at all, anywhere, in any way. So I’m not going to debate gun regulations. However, I could maybe debate the question of whether I should carry a gun; that’s something I could poke with a stick! For instance, if we’re debating whether a gun would actually keep me safer, and I contend that it probably wouldn’t keep me safe because I’m untrained and I’d just miss every shot if I needed to use it, I can actually go to a firing range and check. If we’re debating whether martial arts is a practical self-defence option, then my friend can come to my dojo and spar with me.
These topics partially pass the test; I can poke some aspects, but not others, with sticks. There are several benefits to talking about the aspects of these topics that you can poke with sticks:
There’s probably something to be said for a holistic, in-context practice of rationality. It's sort of like free weights. In a gym, you can lift weights using machines that constrain the weight so it’s all attached to a lever that only goes up or down, or you can lift ‘free weights’ (just picking up a heavy object). Free weights are supposed to have a lot of advantages; you have to use multiple muscles to stabilise them and move them around, so you’re less likely to overtrain one muscle, and you’ll train a bunch of small stabilising muscles that you otherwise might forget. So free weights are supposed to be better at improving your performance at stuff outside the gym, like sports or moving furniture. When I debate with my friend about veganism, and I say it’s really expensive and impractical and he says it’s totally doable, and we decide to go test out our hypotheses at the grocery store, we’ll be training multiple rationality muscles. I might have to overcome some procrastination about actually going to the grocery store, I might have to work on some luminosity to analyse why I’m unhappy about eating the tofu, we might work on good interpersonal norms when I grudgingly admit that Impossible Beef isn’t bad and my friend remembers to praise me for my awesome changing-my-mind abilities instead of mockingly saying “I told you so”, etc, etc. I think that’s better rationality practice!
Since we’ll be able to actually test our models of the world, we have a defined goal that we can stay focused on. I might hope that once we go to the grocery store and check ingredients lists and my friend actually sees how many vegan options are ruled out by my medical dietary restrictions, they’ll finally leave me alone about becoming a vegan and start being willing to buy cow milk for me when I’m too sick to shop for myself. My friend might hope that after I actually try Impossible Beef, I’ll admit it’s okay and stop leaving actual-meat beef mince in the shared fridge. We’re more likely to find things we want to collaborate on, like arranging a time when it’s mutually convenient to get in the kitchen and try tempeh, and that means we’re less likely to fray our friendship. We’re also less likely to be irrational, and more likely to be able to conclusively call it out if we do realise we’ve made a mistake. We can’t get into an endless stalemate; we’re goal-oriented, and the topic will eventually be resolved. This also means we get to practise the nameless virtue.
In the process of poking a topic with a stick, one often ends up dealing with all the messiness and detail and chaos of the real world (as opposed to the beauty of abstract political ideals). It’s easy to sink endless miserable hours into yelling on the internet about gender while feeling weirdly overconfident in our abstract all-encompassing social theories; it’s much harder to do that about the question of whether $4 for 4 ounces is cheaper or dearer than the buy-one-get-one-free $8 per 100g package, or whether a particular vegan butter emulsifies correctly if you try to use it in a sauce rather than using it as a spread, or whether going to the further-away grocery store is actually faster once you take into account the checkout queues at the closer store during peak hours. Staying in touch with that messiness and detail is important; it’s what keeps you calibrated about the correct amount of nuance, what prevents you from proposing “just do this seemingly-easy obvious thing” about actually-very-complex problems, what gives you enough practice modelling (and testing) everyday life stuff that you can optimise it semi-effortlessly and focus on your world-saving priorities…
Even if I have absolutely no intention of actually poking the topic with a stick, the fact that I could still does a lot to keep me honest. For example, I contend that I’d actually be less safe if I visibly carried a gun, because a robber who thinks I’m not a threat will probably just rob me, whereas a robber who considers me an armed threat might pre-emptively kill me just to make sure I don’t kill them first. In theory, I could actually carry a gun around and record whether I ever get preemptively murdered; in practice that’s not a very practical test and I have no plans on ever conducting it. But if I say, "Well, if I ever have to deal with an armed robber then I'll just defeat them easily with my awesome kung fu skills," it will be very embarrassing for me if I ever run away screaming from a mugger in a dark alley.
I don’t get bogged down in stupid debates on the internet because I remember that the point of rationality is for everyone to self-improve so we’ll be more likely to be able to save the world, and so I’m here in the rationality community trying to poke myself with a self-improving stick so that I acquire a bigger world-improvement stick to poke the world with, and…. yeah, okay, my metaphor is breaking down. Point is, I remember that I’m here trying to improve the world, and if I notice that nothing I poke with the stick is going to really change in any world-improving sort of way, I’ll be more likely to drop the topic. And alternatively, I might take the opportunity to check in with myself about my goals and priorities: do I want to dedicate two hours a week to attending martial arts classes? If I discover that I actually would endorse going to martial arts classes, can I enlist the social support of my friends and suggest that we go together? Do I feel like I need to do more to stay safe, or am I happy with my current level of safety? I don’t need to schedule separate time for this kind of evaluation if I'm constantly checking whether I could practically apply an idea to improving my life and winning more.
(Also, sometimes you need to be honest with yourself about whether you’ll poke the thing with a stick. Debating YIMBY politics might be a productive and important activity for you if you’re planning on actively showing up at local community meetings and advocating for local developments. If you spend lots of time talking on the internet about how people should definitely do that, but you notice that you stay in bed every time it’s time to actually walk the walk, then consider that there’s actually some barrier preventing you from poking the topic with a stick - and it would be more productive to address that and then revisit the topic.)
Marijuana: I can't poke marijuana laws with a stick; I'm not interested in an extended debate about whether alcohol should be illegal because it's more dangerous than marijuana, because I couldn't enact that law if I tried. I could poke my brain with a marijuana-shaped stick and see what happens - but I’m not going to try taking marijuana, because I'm unusually concerned about following all the laws, and I’m worried I’m in the group that has pre-existing risk factors and might get psychosis.
Does this make marijuana something I can’t poke with a stick? I think no. It’s worth me having correct beliefs about things like, “how can I help my friends who do take marijuana stay safe?” - and that might require answering questions like, “does marijuana pose significantly more risk than caffeine?”
I could make disapproving noises every time my friends take marijuana, which would be one kind of poking my friends with a noise-stick, and I might do that if I concluded it was especially unsafe or antisocial - or I might avoid doing that if I thought it was impolite and judgemental. Alternatively, if I was in Amsterdam and I was sufficiently convinced that it was totally safe, I could take marijuana - so I at least hypothetically have a potential stick.
Sexual Assault: Okay, going to be serious and not trite about this one because it’s a serious topic. If I was running a heuristic like “only engage in debate about topics that are civil and nice and not hurtful, and don’t engage with Discourse that gets heated and mean and triggering” then this would absolutely be in the “don’t engage” pile; it’s triggering and people write a lot of bullshit about it. That’s why I wanted to specifically bring it up as an example of a topic that, according to the stick rule, you should engage with. You probably know sexual assault victims, and you may have to deal with practical questions about how to support them. You might know couples who mutually accuse one another of abuse after breaking up, and you’ll have to decide what you’re going to do about that. You might absolutely want to avoid debating sexual assault topics with people who are misogynist or cruel or careless, but you do actually want to have good sensible opinions about this topic for practical life reasons, so it’s worth engaging with enough sources of good information that you have some informed opinions. That means actually evidence-based resources about what practical things to do; watching antifeminist YouTuber rants is still among the fastest ways to become more irrational and you should never do that.
I think people talking about this topic reliably talk more compassionately and more truthfully if they’re coming less from a place of, “How can I be theoretically correct about this topic and win internet points for being super correct and super smart?” and more from a place of, “How do I support my friends if they’re assaulted? How do I know what to do and what to say if someone accuses my friend, partner or family member of assault? How can I help protect myself and my friends from being abused? What do I do about that one person who keeps pushing my boundaries? How do I make sure I’m not pressuring or hurting my partner when I ask them for sex?” - because there’s a pressure to try and be truthful when you think the topic matters, in some deeper sense of mattering where it’s about more than internet points. If you can’t come at the topic from that mindset, I don’t think you should be in the conversation.
Abortion: I can't change abortion laws single-handedly. I could try campaigning to get them changed, but that particular activity is among the least neglected activities and probably isn't very productive. However, I totally could get an abortion, or only agree to date partners who share my views on abortion, or drive a friend to the abortion clinic, or refuse to pay for someone's abortion. It's probably worth figuring out my views on abortion well in advance of a decision like that ever coming up - so it's worth talking about abortion, even if I currently don't intend on poking anything with a stick.
These topics require very great care when poking them with sticks. You absolutely have actions that you could take; you could ostracise somebody you suspect of sexual misconduct, you could take some drugs, you could tell someone that you do or don't believe their claims to have been raped, you could stock plan B in your household's first aid kit, you could flush your housemates' drugs down the toilet and refuse to share a house with said drugs, you could pressure your partner into getting an abortion they don't want. However, you can't take the attitude of "let's just run a quick test, get some results and then iterate fast" because most actions you could take are actions with a very large potential downside if you're wrong.
I still find it helpful to - if I may truly torture my metaphor - debate with my stick in hand. Remember the stakes - and remember that you care more about the stakes than you do about saying impressive-sounding clever arguments, or avoiding the shame of admitting you're wrong, or getting everyone to admit to the validity of your technically-correct-but-practically-irrelevant pet peeve. If you suspect that somebody else in the debate isn't holding a stick, then remember that they may just say inflammatory things because they are approaching the debate as a fun debating game and not as figuring out what to do in a difficult circumstance. Remember that you may be called upon to actually change your behaviour as a result of what you're arguing, so you need to stay in touch with the territory and not just consider the most beautiful map; really ask yourself about whether you'd be comfortable making a different decision if you changed your belief.
When you are able to focus the topic in on a particular opportunity to poke something with a stick - from "should we use condoms when we have sex?" to "should we fire this employee?" to "should we call the cops on the homeless drug user in this alley?" - it's almost always worth doing so, but it's also almost always worth restricting those conversations to people you trust deeply. Strangers on the internet will have absolutely terrible input on these questions. It's easy, over the internet, not to feel the weight of the stick in your hand, and not to emotionally reckon with the burden of "if the Reddit comment I just left is totally off-base and the OP actually listens to it, then I have just done something morally wrong."
To put it another way: If the topic is, “I found a weird object in the forest. Is it an unexploded bomb?” then you should definitely not, under any circumstances, poke the bomb with a stick. But there is a certain urgency and practicality lent to the discussion by the fact that you could poke it with a stick; you need to know the answer to the question because it affects important real-life considerations like whether or not you will die. This means you are highly motivated to seek out correct beliefs, and less likely to make mistakes like, “Well, it would be inconvenient and unpleasant to think that someone left an unexploded bomb in the woods, so it’s morally good to believe that this object is a harmless toy which I should totally pick up.” You may want to seek others' opinions on whether the object is an unexploded bomb - but you should not seek the opinions of anyone who seems to be focused on the goal of "get people to think it's really sexy that I know so much about explosions" over the goal of "don't get blown up".
Part Four: That Which I Am Not Saying
Am I recommending that you only ever talk about topics that can be poked with a stick? No. Sometimes you’re having fun talking about whether contemporary English longbowmen could’ve taken Genghis Khan, and you should just have fun sometimes. Sometimes you think you have a great non-stick-related way of staying in touch with the territory. Sometimes a topic is really important, like AI risk - so even if you can’t currently think of any ways to poke it with a stick, you should just keep thinking about it until you come up with something, or you should use a really long metaphorical stick or something.
Am I saying that it is bad, wrong and evil to engage in talking about politics? No. If you have lots of fun talking about politics, then that's a perfectly fine hobby for you to have. I am saying that (if you don't have a stick) you should approach it as a fun hobby, and stop if it stops being fun; don't relate to it as 'rationality practice' or 'doing important work of convincing people about important true things' or 'improving my models of the world'. Also, if you have a relevant stick, go ahead and talk about politics.
Am I recommending that certain topics are off-limits forever? No. There's almost always a way to turn a topic into something that you can poke with a stick. "Can I pass an Ideological Turing Test regarding this topic?" is something you can test and get a positive or negative answer to - and, relatedly, I expect discussions of abortion focused on "can we all pass each other's Ideological Turing Tests, and if not, why not?" to be more productive and useful than discussions focused on "should the US Constitution protect a right to abortion?"
Should everyone else avoid the same topics that I'm avoiding? No. Different people will also have different sticks. My partner is a policy analyst, so when he debates about political topics, there’s an actual chance that he could write a paper about the topic which convinces a lawmaker to do something differently and contributes a small amount to potentially changing the law. I think it’s rational for him to spend much more time debating politics than me. I’ve coached esports teams, so I've spent lots of time poking 'optimal video game strategy' with a stick and am familiar with messy real-world considerations like "in theory this attack does more damage but it's harder to successfully pull it off against semipro-level dodging, so this other attack is more reliable" or "that theoretical rock-paper-scissors model falls apart in this game because if you just rocket-propel your rocks then they smash straight through scissors". My partner has never had this kind of stick - he can test out his strategy, but his conclusions are limited to "this seems to work in my Bronze-tier games" and he can't check much regarding whether it'd work well if it were a high-level meta - so he's aware that, when he's debating optimal video game strategy, he's doing that just for fun and isn't doing useful rationalist self-improvement.
Am I saying that rationality is useless for politics? No. In the original “mindkiller” post, Eliezer suggests that politics can contain important topics to apply our rationality to, but it is a terrible domain to learn rationality; I basically agree, and I think the stick test is a great way to check whether you’re actually doing any applying.
Am I saying that it is bad that rationalists tend to pull out studies and historical examples when we're debating? Absolutely not! Political debates involving studies are usually a lot better than ones without. I think there’s value to the sort of debates where one friend proposes A, and another friend proposes not-A and instead B, so you decide to look it up to see who’s right, and Wikipedia and the first 3 studies on Google Scholar all support friend A, so you give friend A some Bayes points. My problem is with the sort of debates where friend B will come back with a few studies of their own, and a methodological critique of the studies you found supporting A, and claims that Wikipedia is biased, and then you talk about the topic for the next eleventy billion hours and never resolve anything. That is the point where it becomes a high priority to ask whether there’s a way to poke the topic with a stick; either to test the hypothesis for yourself in some way that both friends will admit is a fair and true test, or to think of some actionable decision you’ll take as a result of this debate (which provides framing and goal-orientation and context and collaboration incentives and a deadline to the debate), or to justify to yourselves that you really want to have this debate despite lacking sticks.
Am I saying that rationalists can't talk about topics that can't be poked with sticks? No. I am suggesting that debating about non-stick-pokeable topics is not rationalist. My disagreement is most fundamentally with people who believe things like "rationalists should debate politics because it's good practice at having debates" or "rationalists should debate politics because the rationality community is a safe space for taking ideas seriously" or "talking endlessly about YIMBYism is grey tribe therefore it's a very rationalist thing to do". Not every action I take throughout the day is rationalist; sometimes I just watch Netflix and play solitaire on my phone. I just try to refrain from pretending that my solitaire habit is somehow praiseworthy or interesting to other rationalists, and I try not to pretend to myself that it’s virtuous or productive, and I try to notice when I’m not enjoying an addictive thing and stop. Have fun however you want - but if you’re not enjoying yourself, try to stop rather than pretending to yourself that it’s terribly important to prevent anyone ever being Wrong On The Internet. (Much has been written by feminists about this: I’m a feminist, and sometimes I jaywalk, but that doesn’t mean jaywalking is a feminist act, and I will tie myself into all sorts of knots if I think that all my actions need to be feminist and therefore I need to somehow come up with a way for jaywalking to be inherently feminist and furthermore I need to blog about it.)
Am I saying politics isn't important? No. It's often very important to be correct about politics. I am saying that it's better to spend most of your time elsewhere, because politics isn't a topic that makes you stronger. Your garden may be unimportant compared to AI policy, but spending your hours developing correct beliefsabout how to grow tomatoes is better than spending your hours developing incorrect beliefs about AI policy. (Sometimes people say "if you wish to cut down a tree, first spend several hours sharpening your axe" but I think this understates the case significantly. Cutting down a tree with a blunt axe will still work eventually and mostly won't hurt anyone. I would say: If you wish to build a skyscraper, first spend several years in engineering school, then engage in fights about whether red bricks are a good material to use for your loadbearing support pillars.)
Am I saying that personal experiences are empirically/epistemically superior to RCTs? No. If you want to know whether smoking causes cancer, Googling a large-scale meta-analysis is a much better strategy than listening to someone saying, "Well, my non-smoking grandma died at fifty from lung cancer and my pack-a-day-smoker grandpa lived until ninety-two." The stick-poke test is a guide to whether a debate is likely to be mindkilling, not a guide to whether you're right. You're certainly more likely to be motivated to be correct about questions like "will quitting smoking reduce my risk of lung cancer?" when they're relevant to your own survival, but people can be perfectly motivated to engage in all the right truthseeking behaviours and still end up wrong.
Part Five: Conclusion
Much has been written before about the virtue of getting curious. This post is really just reframing the virtue of curiosity in a way that, for me, works slightly better for my brain. It’s easy to lie to myself that I’m totally definitely being curious while I engage in pointless ten-hour-long internet yelling matches; I’m super curious about whether this obnoxious totally-wrong commenter will eventually change his mind, and I’m also curious in an abstract sense as to whether I’m right about politics - but I have no actual plans on doing anything that would help me find out whether I’m right!
The image of poking something with a stick, for me, summons up just the right mindset: “Hmm, I wonder what happens if I do this?”
It is not a perfect metaphor. Someone asking, “I wonder what result I’ll get if I search for scientific studies addressing this question?” is doing way better on the stick-poke thing than someone standing by a hornets’ nest with a literal stick insisting, “Once I poke this with a stick and survive, I’ll prove all you doubters wrong!” - but the guy with a stick is still doing better than a commenter who insists online that poking hornets’ nests with sticks is totally safe (though of course they have absolutely no intention of ever putting themselves in a situation where they may be asked to demonstrate).
I try to mostly live with the poke-it-with-a-stick-test; if I can't poke something with a stick, I'm not interested in spending hours talking about it. Of course, I very frequently fail, and then usually I regret spending my hours on unproductive discourse.
One of the hardest parts of learning this skill is learning to find things fun. Debating transgender issues isn't exactly fun (in factit's often wildly upsetting) but there's something deeply engaging about it; it's easy to find myself deeply motivated to write thousands of words explaining exactly why I'm annoyed by a certain belief. Compared to yelling at internet people about creationism and evolution, it's easy to think that 'grocery prices' and 'routes to the metro station' and 'building shelves' are boring, mundane, uninteresting topics. One way to deal with this is to accept that they're boring and mundane, but we should debate them anyway, like going to the gym because it's healthy even though we don't really want to do the treadmill. Another way is to try very hard to be genuinely interested in the world, to look for fascination in ordinary everyday things and treat each outing as an exciting expeditionary adventure out into the wild and wonderful Territory. I think it would be a great rationalist virtue to cultivate admiration and appreciation when someone excitedly shares their knowledge about how international plugs are wired, or how different types of paint work chemically, or how to fix shoes. Getting into this topic would, unfortunately, be beyond the scope of this post.
Some questions I try to ask myself:
Hmm, I wonder if there’s a way to poke this topic with a stick?
OK, I can’t poke this with a stick. Do I think it’s productive for me to spend time debating about this, given that?
Can I get my interlocutor to reframe this debate so we’re collaboratively looking for ways to poke the world with a stick, and seeing what happens together? (Do they seem like they’d try and poke things with a stick if they had a stick, or do they just want to write angry things online about this topic?)
In the absence of sticks, is there some other way I can keep myself honest? (Can I bet on my beliefs in some other way?)
Does this person have different sticks to me? Even if I can’t poke this topic with a stick, is it useful for me to help them to poke it with their stick? How should I approach discussions differently when I’m facilitating someone else’s stick-poking and unable to do my own?
Am I having this debate for rationalist reasons, or for fun reasons? (If it’s for fun, we should make sure everyone involved is actually having fun!)
Remember: topics you can’t poke with a stick are the mindkiller!
Note: If you want to respond to this post with an object-level comment about immigration, you have to preface it with, "I didn't read or understand this post, but:"
For part one of the aspirant sequence - which may or may not be arranged into some totally different order when I'm done with it, because the connection here won't be obvious yet - see Would you work harder in the least convenient possible world?
Partly in response to: Politics is the Mind-Killer and Politics is Hard Mode
Part One: A Tale Of Two Houses
Two groups of rationalists live in houses across the street from one another, as is tradition in San Francisco.
One Monday night, in the Purple House, everyone is having a debate about the prison abolition movement. Alice thinks that prison abolition is an irresponsible fantasy which would lead to murderers and arsonists running around doing whatever crimes they wanted. Bob thinks that prisons are a moral horror equivalent to slavery and that Alice doesn't care about the human rights of prisoners. They get quite heated for a few hours, then their housemate Charlie reminds them that they're rationalists and that they should identify a crux. It turns out that the crux is whether, in the absence of prisons, the crime rate would go up a lot. Alice and Bob can't test this - they can't get rid of prisons and see what happens - so they both go to bed frustrated.
In the Orange House, Deanna and Ethan are debating something terribly boring and unimportant. Deanna thinks that cutting behind the Walgreens is a shortcut to the metro station. Ethan thinks that, because that route sometimes results in waiting at an extra traffic light, it actually takes longer on average to walk that route than to simply go through the park - and the park has the nice bonus of getting to walk through some pleasant grass. Their housemate Francis suggests that they could just go and test this, so they take twenty minutes out of their day to go outside and resolve their argument. They both walk to the station on their preferred route, and Deanna arrives three minutes earlier. They then walk to the junction with the traffic signal, and time it. They discover that, if Deanna had been held up at the traffic signal, it would've added about 90 seconds extra to her time - so Deanna's route is faster if she's held up at the traffic signal on less than half of walks, and Ethan's is faster if it's more than half. On the walk home, they realise that Ethan is actually a slightly faster walker than Deanna, and they didn't control for this during their initial test; if they'd swapped and walked each other's preferred routes, Ethan would probably have beaten Deanna by more than three minutes. Also, the traffic signal is 90 seconds' delay for a full cycle - but if they had arrived midway through the stop, they would've had to wait for less than 90 seconds. Ethan therefore changes his mind and acknowledges that Deanna's route is a genuine shortcut - and Deanna decides the time saving is small enough that she'll take Ethan's route when she's not in a rush and doesn't want to smell the bins behind the Walgreens.
On Wednesday night, in the Purple House, everyone is having a debate about immigration[1]. Charlie supports open borders, and Bob thinks controls on immigration are necessary to protect workers' wages. Alice reminds them that they're rationalists, and should be using science and data in their debate - so Charlie shares several scientific studies that show that immigrants contribute more in taxes than they cost in welfare, are less likely to commit crimes than home-grown citizens, and that immigrants actually long-run increase wages for native-born citizens. Bob reads one study and has several criticisms of its statistical methods - and regardless, he argues, the fact that current strictly-controlled immigration is positive doesn't prove that hypothetical unlimited immigration would be equally positive. He cites that refugees in Germany are much less positive for Germany than H1B workers are positive for America, showing that it is necessary to apply some selection effect to get positive immigrants. Charlie argues that refugees are actually negatively selected - if your country only grants asylum to people who have been through sufficiently terrible things, you'll mostly get very poor and disabled and traumatised people who are less likely to be able to work and more likely to perpetuate cycles of abuse. Neither can find a study that the other agrees is applicable and well-designed. Eventually Alice tells them that if neither of them is likely to change their minds, they should just go to bed.
In the Orange House, Francis is focused on a rather mundane and unimportant-seeming project. They have gotten frustrated with the cupboard organization system; there's a giant cupboard that they use to store cans of food, but stacking the cans so high means that they rarely look at any of the cans on the bottom of stacks in the back of the cupboard. They have bought some plywood and are planning on putting some dividers into the cupboard so that it is easier to sort the cans categorically - beans in one section, tomatoes in another, soup in a third - which should remove the need to actually look at all the cans. Deanna has concerns that the plywood won't be strong enough to actually hold all the cans; it doesn't look very strong to her, and she's worried about glass jars smashing when it gives way. Ethan reminds them that they are rationalists, and they have phones in their pockets, so they go to Google to find out how strong plywood is. Deanna is surprised to learn that plywood is actually generally stronger than regular wood, because wood is strongest across its grain and plywood is made by gluing together sheets with grain going in alternating directions. Francis finds that the 3/4" thick plywood they bought should be able to hold at least 50-100 pounds and most soup cans weigh around 11 ounces, so they write a tiny "maximum 72 cans" label for their can shelf. Ethan suggests that they might even be able to rig up a wine-rack-style criss-cross system so that every type of can could be visible and accessible, but none of them know how to apply glue or screws in the correct way to make that sort of thing hang together; they can't put a nail straight through to attach two pieces without that nail awkwardly poking out of the other side. The three of them spend the rest of the night experimenting with plywood, and eventually invent a method of creating a joint by wrapping wire through some adjacent drilled holes, which they use to build a nice-looking spice rack.
On Friday night, at a house party in the Purple House, Alice and Charlie are arguing about veganism. Alice is a vegan because she is concerned about animal rights, and Charlie isn't a vegan because they don't think animals are sentient. Alice brings up all sorts of scientific evidence regarding observed animal behaviours like mourning and learning, studies on animals' brain activity, numbers of neurons in various animal brains, studies where animals chose to consume pain relief drugs, and the list of animals that can pass the mirror test. Charlie presents arguments that sentience is fundamentally connected to complex human behaviours like language use, and Alice asks if that means LLMs are sentient. Bob steps in to tell them that if they want to talk about AI at a house party then they need to go to the designated Arguing About AI room, where AI arguments get exiled so that the rest of the house can talk about literally anything else.
In the Orange House, Francis and Ethan are also debating veganism. Francis is a vegan. Ethan doesn't want to be a vegan because he thinks vegan diets are less healthy and more expensive. Francis believes it'd be more convenient for the whole household if everyone was at least vegetarian, since it'd save on storage space for different varieties of food and also let them eat more meals together, so they ask Ethan what his happy price would be to try veganism for a week. Ethan is uncomfortable naming a price, and realises that part of his crux isn't just the price - he's had bad experiences with accidentally consuming soy (which he's allergic to) while trying meat-free alternative products. He eventually names a very high price, but offers a much lower price if Francis is willing to handle grocery shopping for definitely-soy-free vegan products, and an even lower price if Francis cooks. Deanna suggests they delay the experiment by a week, so that first of all Ethan can eat his normal diet for a week while recording what everything costs. Then they can compare the cost of the vegan food that Francis cooks in the next week, to actually test whether it's more expensive. They spend some of the rest of their evening setting up a spreadsheet that tracks grocery costs - and in the process they realise several other places they could be making savings.
Who learned more, over the course of their week? Who practised more key rationality skills - designing fair tests, noticing confounders, changing their minds, making bets, identifying cruxes, noticing when they're confused, solving their problems, cooperating, and putting specific numbers on things? Who do you expect to be stronger rationalists after a few years of living together?
Part Two: The Nature Of The Mindkiller
Politics is the mindkiller; there are certain topics that it's just very difficult to discuss productively, and "politics" is one way of gesturing at a set of those topics. Why is politics so difficult to discuss? Well, the answer usually goes... people are often deeply emotionally invested in politics, or have certain political ideologies wrapped up in their identities, or find it difficult to respond to the ideas in front of them without also responding to all the worst versions of those ideas that they've seen pushed by the other side. This is also true of religion and football - hence why "religion and politics" are considered two topics you just shouldn't bring up in the workplace or over Thanksgiving dinner.
But there's another diagnosis of what makes certain topics mindkilling; the inability to resolve them.
I might be deeply emotionally invested in a piece of engineering that I'm proud of, but if somebody comes along and demonstrates a more-efficient method to solve my problem, I'm usually capable of getting over my feelings and thanking them. Being an introvert might be part of someone's identity, but if someone points out that they'd get more job offers if they went out to networking events, they're usually capable of reasoning about whether that's true and pushing themselves to test it. I might have been given awful boiled-soggy Brussels sprouts by my parents many times, but I can push myself to try my housemate's amazing Brussels sprouts fried with bacon, and acknowledge that they're actually pretty great.
In all of those cases, there's an actual reward for doing the right thing. If I can get over my embarassment about someone else's code being better than my code, I can make my program run faster and more efficiently. If I can untangle introversion from my identity, I can get more job offers. If I can get over my prejudices about sprouts, I get to enjoy my housemate's excellent cooking.
I don't know about you, but I'm a pretty simple learner. I find it easier to learn new skills if I can test it out, see what happens, get a cookie if it worked, and then rapidly iterate. I get better at cooking by making a meal, being rewarded with tasty food if I did a good job, and noticing when I've made non-tasty food and being motivated to fix that for next time. I can also work with negative motivation - I get better at swordfighting by trying to swordfight, repeatedly getting hit, and then feeling motivated to block those attacks next time. But if I had to learn, say, programming, by just writing code out with pencil-and-paper and getting other people to give feedback on whether it looked good - and I couldn't ever run the code to see if it worked - I don't think I'd ever learn to code.
Politics and religion are topics famous for the inability to resolve them. We can't die, check to see if there's an afterlife, and then come back. We can vote in another party to see if they do a better job... but only once every several years, and no individual person can do that without other voters cooperating. We can't, realistically in our lifetimes, vote in a third party in the USA to see if they'd do a better job than the Democrats or Republicans. We can't do open borders for a few months just to see what it's like and then hit 'undo' if we didn't like it.
I don't think "emotional investment" or "identity entanglement" or "lots of previous experiences with terrible arguments" is the thing that makes a topic really badly mindkilling. The thing that makes a topic a mindkiller is when you cannot resolve it.
It's not just that you can't run experiments and get feedback - and so you're effectively going into battle unarmed, lacking your best rationality tools. It's also that there is no incentive to be truth-seeking. If I have a terrible opinion like "a diet of nothing but beef, pasta and crackers is optimal" then someone might challenge me to actually stand by my beliefs, and actually eat that diet, and then I will get scurvy and it will suck. What's the penalty for having a terrible opinion like "we should try absolute hereditary monarchy again"? People will tell me my opinion is terrible, but I'm never going to actually have to stand by my belief; I will never have to say "okay, we tried putting King Charles III in charge of the USA and it went kind of terribly for me and I got scurvy, sorry about that everyone, I was wrong and it sucked". I might lose social points because people think I'm wrong - but alternatively I might gain social points because people think I'm really interesting and fun to debate with.
And it's also that, because you can't run experiments to resolve your questions and you don't ever get outcomes that give you feedback, these debates can go on literally forever - and that allows them to gain unlimited complexity, drift an unlimited distance from anyone's crux, and build up unlimited resentment over time. They are limited only by your frustration and/or your need to go to bed. I can fantasise all I like, but I'm never going to actually get to the point where we all say, "Great work everyone on resolving transgender inclusion in sports! It's solved! Now that we never need to debate this again, let's move on to solving Israel/Palestine."
It is nearly impossible to have a rationality community where nobody discusses politics, because sometimes topics with important real-life consequences are political. Unfortunately, if we swing too far the other way and spend lots of time blogging about politics, there are horrible consequences; several rationalists I know have had to completely block all social media from their work computers because otherwise they’ll be nerdsniped into endlessly yelling on Discord, which actively makes them more biased and less productive. Also, frankly, a lot of the political discourse in the rationality community actively drives away women.
If your model of the mindkiller is that emotional topics are mindkilling ones, you're left with a terrible choice between "ban discussion of anything that actually matters enough to make people emotional" and "allow everyone to yell endlessly about politics". If your model of the mindkiler is that unresolvable and untestable topics are the mindkilling ones, you can talk about lots of the politics that matters - while also heading off unproductive discussion with "hey, everyone, this seems like a mindkiller", and/or a blanket rule against mindkilling topics in your Discord / comments sections / house parties / etc.
This model explains why there are plenty of highly emotional topics (like whether to try chemotherapy if you have terminal cancer, or whether your sports team is very likely to win their big game) which it is very easy to find good discussion of online, despite the fact that they are deeply emotionally upsetting... and there are plenty of fundamentally unresolvable topics (like which fictional characters should date/marry/kiss which other fictional characters) which produce some of the internet's worst discourse despite their complete lack of rational importance to anyone.
This also predicts that DC rationalists - many of whom work in policy - will have far more productive, useful and rational conversations about politics than San Francisco rationalists - who are mostly software people - and so far I've found that to be extremely true. I expect DC rationalists could also have some spectacularly unproductive and useless conversations about programming, if they were so inclined.
You could argue, of course, that it’s good rationality practice to hone our de-biasing truth-seeking skills on tricky topics like guns and gender and gossip so that we’ll find it really easy to be not-biased when we come to easier topics. Unfortunately, I’m pretty sure that Discourse makes your rationality worse rather than better. I think engaging in online debate about this sort of topic reliably makes me angrier, more biased, more committed to abstract theoretical ideas that I’ve never had to bet on, less collaborative and less productive. I think online Discourse can be addictive in a way I really don’t endorse. And fundamentally I think you’ll get better practice by building IKEA furniture without the instructions, or reading about historical stories without spoilers and trying to predict how it turns out, or going out into your garden and trying to identify all the plants, or taking stats classes.
So, how do we decide when it’s healthy to engage in a piece of Discourse, and when it’s not worth the mindkill penalty?
Part Three: The Poke-It-With-A-Stick Test
I ask myself, "Can I poke it with a stick?"
This is, somewhat deliberately, a phrase that is more evocative than it is specific. The stick is metaphorical - but you never want to let it get too metaphorical. Asking the question while imagining a literal, physical, actual stick in your hands is honestly a great way to get the basic idea. The stick represents some combination of: being able to test the idea, being able to resolve your argument, being able to get feedback with actions, being close enough to the topic to be able to observe it well, being able to affect the topic at all, the topic being relevant to your life, leaving open the possibility of taking future actions on the topic, etc.
Let's consider several examples. Here are some kinds of Discourse that my friends and online spaces have engaged in:
I think some of these can be engaged in productively by rationalists who are trying to improve their minds and their lives. I think others are wastes of time and endless causes of strife, and I prefer to nudge my social circles away from discussing them, when and where I can.
(I won't pretend I'm very good at successfully avoiding terrible mindkilling discourse - I am not! I am, in part, writing this in the hope that I'd be better at applying my stick-poke test if others around me could help to collectively enforce it.)
I think before you read on, you should take thirty seconds to figure out if you have any clear simple rules that you’d use to make this determination, which you might want to think about without being biased by the post and/or you might want to write in the comment section later.
Billionaires: I cannot poke a billionaire with a stick. I don't think I know any billionaires well enough that I could get close enough to poke them with a stick. I don't work for any EA orgs which are responsible for deciding whether to befriend some billionaires, so I doubt I could affect the decision of whether to do so, so I can't poke that with a stick.
I'm sure someone could dig up a study on whether billionaires donate more or less to charity than average people, or whether billionaires commit more or fewer crimes than average people, or whether billionaire-founded charities are more or less effective than average charities. This is important: possession of a study is not the same thing as possession of a stick. Someone else can easily dismiss your study. They can say that there's a bias where the billionaires who don't donate to charity are likely to be less-public figures and therefore less likely to show up in your data, or dispute whether we should be looking at the total amount donated by billionaires or the percentage of income, or point out that billionaires' crimes are mostly things like wage-theft that go under-reported and under-prosecuted, or say that your study has massive methodological issues. Then you're right back where you started.
Justice: I am nowhere near a prison and cannot poke one with a stick. I cannot poke any prisoners with a stick since I don’t think I know any prisoners. I cannot poke the death-penalty laws with a stick, since I am not a lawmaker or policy analyst and nothing I say or do is likely to change any of the death-penalty laws in any jurisdiction.
It's not that experiments can't be run in this domain; some country or state could change sentencing guidelines and we could see whether the crime rate goes up or down. However, this doesn't make the topic pass the poke-it-with-a-stick test; it's very easy for someone to say "well, sure, that worked in Norway but it'd never work in America because America's different" and then you're right back where you started.
These topics fail the poke-with-a stick test (for me). If we try to debate them, a few bad things happen:
This is not to say that nobody can debate these things because nobody has a stick; some people have sticks!
However, the people who do have sticks are often dealing with all the messiness and chaos and detail of reality. Maybe your dream ideal rehabilitative justice system runs into issues with a particular committee that needs to approve a particular kind of expenditure and zoning laws about prisons built in semi-urban areas, and the thing the actual people-with-sticks are working on is something to do with pressuring a particular committee member to withdraw their veto so they can change how judges see sentencing requirements in paperwork. Maybe the people who affect EA outreach decisions are busy worrying about the tax implications for a particular billionaire who wants to sell stock to donate to AMF but keeps being told he’ll void a lockout agreement if he does that. I don’t know! The people with sticks probably know those things, which is why I will first acquire a stick (or at least the possibility of having a stick, or a friend with a stick I can ask) before considering it a productive activity to debate this.
Veganism: I can't really poke an animal's brain with a stick to see whether it's a moral patient. No amount of experiments with mirror tests will tell me whether the mirror test is even a relevant test to questions of moral worth. However, I can absolutely poke some tofu with a stick. Specifically I can put the tofu in my mouth, and that is an easy way of empirically testing whether eating the tofu is good. If I decide veganism is good, I can go to the store and poke some vegan groceries with a credit-card-shaped stick. If I poke my stomach with a tofu-shaped stick and I get violent diarrhoea, that is a solid signal that maybe veganism isn’t for me.
Guns: I can’t poke gun laws with a stick. It’s not like I can ban guns completely in half of the world’s countries, completely deregulate them in the other half, and see who’s got better murder rates next year. Since I don’t work in politics, it’s not actually like I can affect gun laws at all, anywhere, in any way. So I’m not going to debate gun regulations. However, I could maybe debate the question of whether I should carry a gun; that’s something I could poke with a stick! For instance, if we’re debating whether a gun would actually keep me safer, and I contend that it probably wouldn’t keep me safe because I’m untrained and I’d just miss every shot if I needed to use it, I can actually go to a firing range and check. If we’re debating whether martial arts is a practical self-defence option, then my friend can come to my dojo and spar with me.
These topics partially pass the test; I can poke some aspects, but not others, with sticks. There are several benefits to talking about the aspects of these topics that you can poke with sticks:
(Also, sometimes you need to be honest with yourself about whether you’ll poke the thing with a stick. Debating YIMBY politics might be a productive and important activity for you if you’re planning on actively showing up at local community meetings and advocating for local developments. If you spend lots of time talking on the internet about how people should definitely do that, but you notice that you stay in bed every time it’s time to actually walk the walk, then consider that there’s actually some barrier preventing you from poking the topic with a stick - and it would be more productive to address that and then revisit the topic.)
Marijuana: I can't poke marijuana laws with a stick; I'm not interested in an extended debate about whether alcohol should be illegal because it's more dangerous than marijuana, because I couldn't enact that law if I tried. I could poke my brain with a marijuana-shaped stick and see what happens - but I’m not going to try taking marijuana, because I'm unusually concerned about following all the laws, and I’m worried I’m in the group that has pre-existing risk factors and might get psychosis.
Does this make marijuana something I can’t poke with a stick? I think no. It’s worth me having correct beliefs about things like, “how can I help my friends who do take marijuana stay safe?” - and that might require answering questions like, “does marijuana pose significantly more risk than caffeine?”
I could make disapproving noises every time my friends take marijuana, which would be one kind of poking my friends with a noise-stick, and I might do that if I concluded it was especially unsafe or antisocial - or I might avoid doing that if I thought it was impolite and judgemental. Alternatively, if I was in Amsterdam and I was sufficiently convinced that it was totally safe, I could take marijuana - so I at least hypothetically have a potential stick.
Sexual Assault: Okay, going to be serious and not trite about this one because it’s a serious topic. If I was running a heuristic like “only engage in debate about topics that are civil and nice and not hurtful, and don’t engage with Discourse that gets heated and mean and triggering” then this would absolutely be in the “don’t engage” pile; it’s triggering and people write a lot of bullshit about it. That’s why I wanted to specifically bring it up as an example of a topic that, according to the stick rule, you should engage with. You probably know sexual assault victims, and you may have to deal with practical questions about how to support them. You might know couples who mutually accuse one another of abuse after breaking up, and you’ll have to decide what you’re going to do about that. You might absolutely want to avoid debating sexual assault topics with people who are misogynist or cruel or careless, but you do actually want to have good sensible opinions about this topic for practical life reasons, so it’s worth engaging with enough sources of good information that you have some informed opinions. That means actually evidence-based resources about what practical things to do; watching antifeminist YouTuber rants is still among the fastest ways to become more irrational and you should never do that.
I think people talking about this topic reliably talk more compassionately and more truthfully if they’re coming less from a place of, “How can I be theoretically correct about this topic and win internet points for being super correct and super smart?” and more from a place of, “How do I support my friends if they’re assaulted? How do I know what to do and what to say if someone accuses my friend, partner or family member of assault? How can I help protect myself and my friends from being abused? What do I do about that one person who keeps pushing my boundaries? How do I make sure I’m not pressuring or hurting my partner when I ask them for sex?” - because there’s a pressure to try and be truthful when you think the topic matters, in some deeper sense of mattering where it’s about more than internet points. If you can’t come at the topic from that mindset, I don’t think you should be in the conversation.
Abortion: I can't change abortion laws single-handedly. I could try campaigning to get them changed, but that particular activity is among the least neglected activities and probably isn't very productive. However, I totally could get an abortion, or only agree to date partners who share my views on abortion, or drive a friend to the abortion clinic, or refuse to pay for someone's abortion. It's probably worth figuring out my views on abortion well in advance of a decision like that ever coming up - so it's worth talking about abortion, even if I currently don't intend on poking anything with a stick.
These topics require very great care when poking them with sticks. You absolutely have actions that you could take; you could ostracise somebody you suspect of sexual misconduct, you could take some drugs, you could tell someone that you do or don't believe their claims to have been raped, you could stock plan B in your household's first aid kit, you could flush your housemates' drugs down the toilet and refuse to share a house with said drugs, you could pressure your partner into getting an abortion they don't want. However, you can't take the attitude of "let's just run a quick test, get some results and then iterate fast" because most actions you could take are actions with a very large potential downside if you're wrong.
I still find it helpful to - if I may truly torture my metaphor - debate with my stick in hand. Remember the stakes - and remember that you care more about the stakes than you do about saying impressive-sounding clever arguments, or avoiding the shame of admitting you're wrong, or getting everyone to admit to the validity of your technically-correct-but-practically-irrelevant pet peeve. If you suspect that somebody else in the debate isn't holding a stick, then remember that they may just say inflammatory things because they are approaching the debate as a fun debating game and not as figuring out what to do in a difficult circumstance. Remember that you may be called upon to actually change your behaviour as a result of what you're arguing, so you need to stay in touch with the territory and not just consider the most beautiful map; really ask yourself about whether you'd be comfortable making a different decision if you changed your belief.
When you are able to focus the topic in on a particular opportunity to poke something with a stick - from "should we use condoms when we have sex?" to "should we fire this employee?" to "should we call the cops on the homeless drug user in this alley?" - it's almost always worth doing so, but it's also almost always worth restricting those conversations to people you trust deeply. Strangers on the internet will have absolutely terrible input on these questions. It's easy, over the internet, not to feel the weight of the stick in your hand, and not to emotionally reckon with the burden of "if the Reddit comment I just left is totally off-base and the OP actually listens to it, then I have just done something morally wrong."
To put it another way: If the topic is, “I found a weird object in the forest. Is it an unexploded bomb?” then you should definitely not, under any circumstances, poke the bomb with a stick. But there is a certain urgency and practicality lent to the discussion by the fact that you could poke it with a stick; you need to know the answer to the question because it affects important real-life considerations like whether or not you will die. This means you are highly motivated to seek out correct beliefs, and less likely to make mistakes like, “Well, it would be inconvenient and unpleasant to think that someone left an unexploded bomb in the woods, so it’s morally good to believe that this object is a harmless toy which I should totally pick up.” You may want to seek others' opinions on whether the object is an unexploded bomb - but you should not seek the opinions of anyone who seems to be focused on the goal of "get people to think it's really sexy that I know so much about explosions" over the goal of "don't get blown up".
Part Four: That Which I Am Not Saying
Am I recommending that you only ever talk about topics that can be poked with a stick? No. Sometimes you’re having fun talking about whether contemporary English longbowmen could’ve taken Genghis Khan, and you should just have fun sometimes. Sometimes you think you have a great non-stick-related way of staying in touch with the territory. Sometimes a topic is really important, like AI risk - so even if you can’t currently think of any ways to poke it with a stick, you should just keep thinking about it until you come up with something, or you should use a really long metaphorical stick or something.
Am I saying that it is bad, wrong and evil to engage in talking about politics? No. If you have lots of fun talking about politics, then that's a perfectly fine hobby for you to have. I am saying that (if you don't have a stick) you should approach it as a fun hobby, and stop if it stops being fun; don't relate to it as 'rationality practice' or 'doing important work of convincing people about important true things' or 'improving my models of the world'. Also, if you have a relevant stick, go ahead and talk about politics.
Am I recommending that certain topics are off-limits forever? No. There's almost always a way to turn a topic into something that you can poke with a stick. "Can I pass an Ideological Turing Test regarding this topic?" is something you can test and get a positive or negative answer to - and, relatedly, I expect discussions of abortion focused on "can we all pass each other's Ideological Turing Tests, and if not, why not?" to be more productive and useful than discussions focused on "should the US Constitution protect a right to abortion?"
Should everyone else avoid the same topics that I'm avoiding? No. Different people will also have different sticks. My partner is a policy analyst, so when he debates about political topics, there’s an actual chance that he could write a paper about the topic which convinces a lawmaker to do something differently and contributes a small amount to potentially changing the law. I think it’s rational for him to spend much more time debating politics than me. I’ve coached esports teams, so I've spent lots of time poking 'optimal video game strategy' with a stick and am familiar with messy real-world considerations like "in theory this attack does more damage but it's harder to successfully pull it off against semipro-level dodging, so this other attack is more reliable" or "that theoretical rock-paper-scissors model falls apart in this game because if you just rocket-propel your rocks then they smash straight through scissors". My partner has never had this kind of stick - he can test out his strategy, but his conclusions are limited to "this seems to work in my Bronze-tier games" and he can't check much regarding whether it'd work well if it were a high-level meta - so he's aware that, when he's debating optimal video game strategy, he's doing that just for fun and isn't doing useful rationalist self-improvement.
Am I saying that rationality is useless for politics? No. In the original “mindkiller” post, Eliezer suggests that politics can contain important topics to apply our rationality to, but it is a terrible domain to learn rationality; I basically agree, and I think the stick test is a great way to check whether you’re actually doing any applying.
Am I saying that it is bad that rationalists tend to pull out studies and historical examples when we're debating? Absolutely not! Political debates involving studies are usually a lot better than ones without. I think there’s value to the sort of debates where one friend proposes A, and another friend proposes not-A and instead B, so you decide to look it up to see who’s right, and Wikipedia and the first 3 studies on Google Scholar all support friend A, so you give friend A some Bayes points. My problem is with the sort of debates where friend B will come back with a few studies of their own, and a methodological critique of the studies you found supporting A, and claims that Wikipedia is biased, and then you talk about the topic for the next eleventy billion hours and never resolve anything. That is the point where it becomes a high priority to ask whether there’s a way to poke the topic with a stick; either to test the hypothesis for yourself in some way that both friends will admit is a fair and true test, or to think of some actionable decision you’ll take as a result of this debate (which provides framing and goal-orientation and context and collaboration incentives and a deadline to the debate), or to justify to yourselves that you really want to have this debate despite lacking sticks.
Am I saying that rationalists can't talk about topics that can't be poked with sticks? No. I am suggesting that debating about non-stick-pokeable topics is not rationalist. My disagreement is most fundamentally with people who believe things like "rationalists should debate politics because it's good practice at having debates" or "rationalists should debate politics because the rationality community is a safe space for taking ideas seriously" or "talking endlessly about YIMBYism is grey tribe therefore it's a very rationalist thing to do". Not every action I take throughout the day is rationalist; sometimes I just watch Netflix and play solitaire on my phone. I just try to refrain from pretending that my solitaire habit is somehow praiseworthy or interesting to other rationalists, and I try not to pretend to myself that it’s virtuous or productive, and I try to notice when I’m not enjoying an addictive thing and stop. Have fun however you want - but if you’re not enjoying yourself, try to stop rather than pretending to yourself that it’s terribly important to prevent anyone ever being Wrong On The Internet. (Much has been written by feminists about this: I’m a feminist, and sometimes I jaywalk, but that doesn’t mean jaywalking is a feminist act, and I will tie myself into all sorts of knots if I think that all my actions need to be feminist and therefore I need to somehow come up with a way for jaywalking to be inherently feminist and furthermore I need to blog about it.)
Am I saying politics isn't important? No. It's often very important to be correct about politics. I am saying that it's better to spend most of your time elsewhere, because politics isn't a topic that makes you stronger. Your garden may be unimportant compared to AI policy, but spending your hours developing correct beliefs about how to grow tomatoes is better than spending your hours developing incorrect beliefs about AI policy. (Sometimes people say "if you wish to cut down a tree, first spend several hours sharpening your axe" but I think this understates the case significantly. Cutting down a tree with a blunt axe will still work eventually and mostly won't hurt anyone. I would say: If you wish to build a skyscraper, first spend several years in engineering school, then engage in fights about whether red bricks are a good material to use for your loadbearing support pillars.)
Am I saying that personal experiences are empirically/epistemically superior to RCTs? No. If you want to know whether smoking causes cancer, Googling a large-scale meta-analysis is a much better strategy than listening to someone saying, "Well, my non-smoking grandma died at fifty from lung cancer and my pack-a-day-smoker grandpa lived until ninety-two." The stick-poke test is a guide to whether a debate is likely to be mindkilling, not a guide to whether you're right. You're certainly more likely to be motivated to be correct about questions like "will quitting smoking reduce my risk of lung cancer?" when they're relevant to your own survival, but people can be perfectly motivated to engage in all the right truthseeking behaviours and still end up wrong.
Part Five: Conclusion
Much has been written before about the virtue of getting curious. This post is really just reframing the virtue of curiosity in a way that, for me, works slightly better for my brain. It’s easy to lie to myself that I’m totally definitely being curious while I engage in pointless ten-hour-long internet yelling matches; I’m super curious about whether this obnoxious totally-wrong commenter will eventually change his mind, and I’m also curious in an abstract sense as to whether I’m right about politics - but I have no actual plans on doing anything that would help me find out whether I’m right!
The image of poking something with a stick, for me, summons up just the right mindset: “Hmm, I wonder what happens if I do this?”
It is not a perfect metaphor. Someone asking, “I wonder what result I’ll get if I search for scientific studies addressing this question?” is doing way better on the stick-poke thing than someone standing by a hornets’ nest with a literal stick insisting, “Once I poke this with a stick and survive, I’ll prove all you doubters wrong!” - but the guy with a stick is still doing better than a commenter who insists online that poking hornets’ nests with sticks is totally safe (though of course they have absolutely no intention of ever putting themselves in a situation where they may be asked to demonstrate).
I try to mostly live with the poke-it-with-a-stick-test; if I can't poke something with a stick, I'm not interested in spending hours talking about it. Of course, I very frequently fail, and then usually I regret spending my hours on unproductive discourse.
One of the hardest parts of learning this skill is learning to find things fun. Debating transgender issues isn't exactly fun (in fact it's often wildly upsetting) but there's something deeply engaging about it; it's easy to find myself deeply motivated to write thousands of words explaining exactly why I'm annoyed by a certain belief. Compared to yelling at internet people about creationism and evolution, it's easy to think that 'grocery prices' and 'routes to the metro station' and 'building shelves' are boring, mundane, uninteresting topics. One way to deal with this is to accept that they're boring and mundane, but we should debate them anyway, like going to the gym because it's healthy even though we don't really want to do the treadmill. Another way is to try very hard to be genuinely interested in the world, to look for fascination in ordinary everyday things and treat each outing as an exciting expeditionary adventure out into the wild and wonderful Territory. I think it would be a great rationalist virtue to cultivate admiration and appreciation when someone excitedly shares their knowledge about how international plugs are wired, or how different types of paint work chemically, or how to fix shoes. Getting into this topic would, unfortunately, be beyond the scope of this post.
Some questions I try to ask myself:
Remember: topics you can’t poke with a stick are the mindkiller!
Note: If you want to respond to this post with an object-level comment about immigration, you have to preface it with, "I didn't read or understand this post, but:"