I have not currently read most of this (just the tl;dr and some skimming), but wanted to quickly note: I think "rationality" in the LW sense is mostly useful for two reasons
1) having relatively ambitious, openended, confusing projects
2) navigating environmental disruption (i.e. covid)
3) being a good citizen, who is able to vote and participate in The Discourse in a way that shapes your country/world for the better.
I don't think it's especially great for "being a happy, well adjusted guy" (compared to other schools of thought with different vibes). I think it helps, but, not so overwhelmingly that I recommend it to a person who doesn't naturally vibe with it.
The tl;dr has spoilers, so I've put it at the end.
Also feel free to skip any of the chapters because the post turned out to be very long. I think you can read almost any of them on their own, and you can skip to On practicality if you want the big picture.
The title is stolen from Scott Alexander's post Extreme Rationality: It's Not That Great. Basically, it will be my opinion on applied rationality and its successes and failures. The title speaks for itself: I think there are few successes. A few weeks before Scott's post, Eliezer wrote A Sense That More Is Possible where he urged developing training for rationality; and though he thought rationalists were just ordinary people back then, he believed they could become much more. Center for Applied Rationality (CFAR) was created in 2012 with the goal of developing such an Art. So I will discuss rationality's basic values, where I disagree with them, and where they led.
But first, let me start with my personal story.
I was one of the main organizers of LessWrong events in Saint Petersburg, Russia, for two years. I started by reading HPMOR and the Sequences and running discussion groups about them. But then I took a CFAR-like workshop at Kocherga (Moscow) and became obsessed with applied rationality: I tried to practice Hammertime and CFAR handbook. After some time I decided I had enough knowledge to teach rationality, and together with a friend I developed an educational program, which failed for multiple reasons. I also studied bioinformatics and worked for two years at Gero, an anti-aging research company, precisely because I had read HPMOR and been inspired by transhumanism (I still am).
Time passed; I was in psychotherapy and realized that a lot of the things I had been trying to achieve were very unhealthy. I started to criticize applied rationality for that—partly because it is quite comforting to blame a group's ideology instead of your own personal problems. Since I was aware of that bias, I was curious to actually figure out how damaging CFAR practice had been to me and to other people. But I didn't focus on it too much: I just wrote a couple of small posts a few years ago and that was it (they were in Russian, and basically mirrored the posts of another co-organizer of rationality courses in Moscow). Recently I recorded a podcast (also in Russian) with the ex-leader of the Moscow LessWrong community Slava Matyukhin, who supported a lot of my claims, and I became really curious about what is going on—and what had been happening—in the English-speaking part of the community. So I started to assemble the complete picture.
The post will be structured as follows:
There are a few posts that I think intersect with what I am saying here, but I am not going to cite them: Rationalist Epistemics and the Sequences (Effective Altruism Definitions Sequence), Rationalist Epistemics and Social Epistemology (Effective Altruist Definitions Sequence), The Rationality Wars.
On biases
Loss Aversion
The first and foremost bias, the cornerstone of Kahneman and Tversky's theory, is Loss Aversion. When I only started to develop my understanding of the topic I stumbled upon a post by Jared Peterson: Biases Don't Exist, and Humans Are Not Irrational. It was debated there whether biases are actually bad things, because a bias is just a deviation from utility theory, which is not always right.
One example for a failure of utility theory he proposed is ergodicity. You probably know the game where you flip a coin: heads gives you a 50% increase to your bankroll, and tails decreases it by 40%. This process is non-ergodic, and the expected value of each flip is positive (+5%), yet almost every individual trajectory decays to zero, meaning you definitely lose everything. Any trader understands that you can't just maximize the utility of the next bet; some risks are not worth the cost, because after a couple of tries your account is depleted and you never recover.
A simple explanation of the coin example is that, with large N, you get the following (a 50% probability of adding 50% means utility is multiplied by 1.5, and a 50% probability of losing 40% means utility is multiplied by 0.6):
So utility decays with N at a rate of the square root of 0.9.
Another related caveat is that humans actually have diminishing returns on everything, so it is fine to fear losing a given amount of money more than you value not gaining the same amount.
But as it turns out, all of this is not new: Bernoulli invented diminishing returns while trying to solve the St. Petersburg paradox in 1738 (interestingly, the city of St. Petersburg was only 35 years old back then). The St. Petersburg paradox is a game of coin flips with a casino: it pays you
Ergodicity was introduced by Ludwig Boltzmann in 1871. Though I was too lazy to research when it was introduced into economics, I found a fairly recent paper that summarizes its effects well: Peters, O. (2019), "The ergodicity problem in economics." Nature Physics. The conclusion of interest for us here is that ergodicity forces us to use a log utility function, which is equivalent to accounting for diminishing returns — an adjustment already widely used in economics.
So did Kahneman and Tversky account for that as well? If you look at their original paper — Tversky & Kahneman 1992, "Advances in Prospect Theory" — they model utility in a fairly complex form:
Where x ≥ 0 is the utility of gains and x < 0 is the utility of losses. I am not sure if it is very important that the logarithm function required by ergodicity doesn't fit well here, but it definitely accounts for diminishing returns, and ergodicity gives a quite similar effect anyway.
And even though we kind of accounted for these two things, we still see two weird effects here.
First is the reflection effect: the fact that the function is convex on the negative side instead of concave, and the tricky part is that this change is relative to your current state, or reference point. On the standard theory you would not see anything special at the reference point at all: your utility function starts at zero wealth, and from there you have just log utility. It would not depend on whether you lose or gain in a particular scenario, but on the level of wealth it leaves you at. But humans fear additional losses less and less, regardless of how much money they start with. Also, the function should not have become convex. More plainly, you can describe it as risk-averse behaviour for gains and risk-seeking behaviour for losses. People want to risk more even when expected utility is negative, in cases like the choice between (−$4,000, .80) and (−$3,000 sure). Because the jump from −3000 to −4000 is less scary than the jump from −2000 to −3000, they may choose −4000 even though 4000 × 0.8 = 3200. There is actually another effect at play here, probability weighting, but I don't want to go into it.
Second is the kink. It is the fact that we have a lambda coefficient in front of losses, which tells us how much more losses loom than gains. It is not as straightforward as saying that losing a given amount of money has lambda times the weight of gaining the same amount, because we weight not the raw money but money already transformed by diminishing returns plus the reflection effect.
So, even though we should keep in mind that it is not as simple as losses just hurting more than gains, doesn't all this mean that ergodicity and diminishing returns don't save us from the famous loss aversion, and that people do act weirdly? Well, yes — but there is more.
Though The Brown et al. 2024 meta-analysis found that the loss aversion coefficient is 1.955, with a 95 percent probability that the true value falls in the interval [1.820, 2.102], other meta-analyses such as Walasek, Mullett & Stewart found a small λ of 1.31, 95% CI [1.10, 1.53]. I decided not to go down this rabbit hole, but apparently there are at least some debates over how big the effect is (which reminds me of the time I was trying to make sense of psychotherapy efficacy by reading meta-analyses, and all the effect sizes looked basically like random numbers to me, ugh).
And then there is another very important paper: "The Loss of Loss Aversion: Will It Loom Larger than Its Gain?" by David Gal & Derek D. Rucker, 2018. It raises a lot of points that I don't fully cover here, but what I want to do is point out some things Scott Alexander said about it in his post [Crosspost] On Hreha On Behavioral Economics, which is a reply to The Death Of Behavioral Economics by Jason Hreha.
I think this slightly understates the analysis they made. We already discussed that loss aversion is actually two distinct phenomena — the reflection effect and the kink — both united by the reference point, which is itself a contradiction of standard risk-aversion theory. The kink is what makes us think that losses loom larger than gains; it differs behaviourally from risk aversion — it produces a discontinuous change at the reference point and doesn't depend on scale (it is just a constant linear parameter), unlike risk aversion, which is described by a concave function that becomes more important as the scale increases. That's why Rabin & Thaler explained that loss aversion for small stakes can't be explained by risk aversion (whereas for larger stakes it can be interpreted as such).
G&R cite a paper that refutes loss aversion at low stakes: "Is loss-aversion magnitude-dependent? Measuring prospective affective judgments regarding gains and losses" by Mukherjee et al. (2017), and it is a blow to the whole theory, not just a reinterpretation. They also cite Katz (1964), who showed indifference to risk at small bets.
The other reference doesn't even use the small-stakes trick, but cleanly shows the same thing — that there is no discontinuity at the reference point:
It is basically the same choice with the reference point shifted by 1000; the fact that people are indifferent between the two contradicts prospect theory, which predicts dependence on the reference point, and is consistent with risk-aversion theory, which depends only on wealth.
G&R also discuss other explanations for real-world phenomena such as asymmetric demand elasticity and the equity premium.
But Scott discusses a further development of the story though: Mrkva et al., Loss Aversion Has Moderators, But Reports Of Its Death Are Greatly Exaggerated.
Another interesting point in the G&R paper was about decoupling loss aversion from action/inaction in the endowment effect.
I don't think this is just a change of perspective either, because avoiding action is clearly a rational choice, especially at low stakes, and it is important to distinguish it from loss aversion.
There is also another piece of prospect theory that we discussed — the reflection effect. It wasn't challenged at all in any of these papers; there are some other sources that criticize it, like this one, Prospect Theory's Reflection Hypothesis: A Critical Examination, but I didn't dig into it.
Anyway, I am not enough of an expert to read all the papers in the field and draw a comprehensive conclusion; it could be that all these refutations are wrong and the original findings of Kahneman and Tversky are right. But my point is that even for one of the most studied biases the discussion still seems to be ongoing, and that it is not only about reinterpretation but also about the correctness of the theory. Let's now look at the other examples a bit more quickly.
Other biases
There is another great article: "THE GREAT RATIONALITY DEBATE" by Philip E. Tetlock and Barbara A. Mellers, 2002.
They summarize the cognitive-bias research and draw a distinction between two kinds of counterargument. One is experimental: researchers try to reproduce results and fail, or tweak the conditions of an experiment a little and the result changes completely, and so on. The other is when researchers challenge what counts as normative, and whether biases are actually bad — I will call these reinterpretations.
A couple of their examples caught my eye:
But the paper is old, so I will have to add more examples myself, while sticking to their classification.
Experimental refutations
As a quick aside: in this witty and funny post by Scott Alexander, you will find out that parapsychology is the control group for science, and that the replication crisis is a mess; Beware The Man Of One Study is really good as well. So let's take all these meta-analyses and other shmalyses with a grain of salt.
Anyhow, the most obvious example of the first kind of counterargument is Priming and Contamination. I am sure everybody has heard about this, so I will not delve into it, but here is one source.
There is also a refutation of Bystander Apathy.
There is a good post by Kaj_Sotala that covered multiple examples of challenged biases: Are these cognitive biases, biases? A particularly interesting one is overconfidence (or, more precisely, the hard-easy effect).
Reinterpretations
One way to approach this is to criticize existing decision theory, which Jared Peterson discussed in his Biases Don't Exist, and Humans Are Not Irrational. I covered only one example, related to loss aversion, because I am not competent enough to understand the rest; if you are interested in more, please read his posts.
Another is to emphasise real-world success over mathematical correctness — the idea that simple heuristics, which make more errors but just work in most cases and consume less "compute", could be preferable. The main proponent of this approach is Gigerenzer. There is, again, a really good post by Kaj_Sotala about this: Fundamentally Flawed, or Fast and Frugal?
Gigerenzer basically says that people use simple heuristics which are very effective in real life and can create bugs in some edge cases, but that this is much better and more efficient than using correct Bayesian decision-making. There is also a point that our thinking works much better in an intuitive regime, and breaks much more easily when we have to make logical, conscious decisions — which I will discuss later.
Some social context can also change your incentives, and biases become rational: Tetlock (2000), Cognitive Biases and Organizational Correctives. For example, overconfidence or the fundamental attribution error can be necessary in some social environments — ones with high competition and high stakes, and a need for fast, heuristic judgments.
The next example of a bias is quite strong and I will discuss it using Gigerenzer's approach.
Conjunction fallacy
In Eliezer's Burdensome Details, and furthermore in Conjunction Controversy (Or, How They Nail It Down), he discusses the conjunction fallacy, where people prefer a detailed hypothesis to a more general one — for example, they assign more probability to “Russia invades Poland, followed by suspension of diplomatic relations between the USA and the USSR” than to “Suspension of diplomatic relations between the USA and the USSR”. Eliezer very convincingly argues that people just "substitute judgment of representativeness for judgment of probability".
This is actually a strong case. There is even a Kahneman paper that successfully disproves the claim that Gigerenzer's reformulation in frequencies instead of probabilities eliminates the effect (it does reduce it, though). But the question is whether it is a quirk of a specific setting or a ubiquitous bug.
The conjunction fallacy violates the fundamental law of probability that P(A&B) should not be greater than P(A). Also, if people always substitute representativeness for probability, that means they ignore priors: they basically compute how much the posterior probability of H given A — P(H|A) — is increased compared to P(H), instead of the posterior itself (Crupi, Fitelson & Tentori 2008; Tentori, Crupi & Russo). All of this seems quite unrealistic to me; people wouldn't survive at all if their cognition were that fucked up. Again, Tetlock says:
I think people, when it comes to real life, definitely account for priors at least in some situations. We realize that if someone is shouting that there is a dinosaur down the street, it is less likely to be true than if, in the same situation, someone were shouting that there is a bear
on a unicycle.The one real-life example I know of where people fail to account for the conjunction rule is in juries. But in this case you could argue that a detailed story increases the probability that it is not a lie, which is maybe one of the reasons this bias could be evolutionarily favourable.
There was a further discussion about these refutations in the comments, and Eliezer dismissed them in quite a sharp, accusatory tone:
Though Kahneman and Tversky conducted experiments on doctors as well, I think Eliezer is overstating things here when he says "The patient still dies". What Kahneman's experiments tested was the probability of getting certain symptoms given a known disease, which is not what actually kills patients. In real practice, doctors need to diagnose people who have multiple symptoms, and to judge how many diseases they have and which ones. And for that, doctors even have Hickam's dictum — "a patient can have as many diseases as they damn well pleases," which exists precisely because doctors sometimes over-apply Occam's razor. It is standard clinical teaching that the default norm is parsimony (one diagnosis explaining everything).
I know this might sound like cherry-picking — they did have a bias, right, so why does it matter in what scenario it appeared? It is true that Gigerenzer's point, that people are on average rational in real-life scenarios, is a dodgy one and a kind of goalpost-moving: we didn't succeed in refuting careful experiments, so we resort to the theory that everywhere except the lab setting we are rational, which sounds unfalsifiable. But, first of all, some of the biases are actually refuted, and the field of lab social experiments isn't one to be trusted without any doubt. Eliezer puts much more confidence in the science of such experiments than it deserves. And, more importantly, the question that interests us in the end is a real and falsifiable one: how biased are we in real life?
The next bias is another example when the experiment trapped people in unrealistic conditions, while in real life their strategy works well.
Confirmation bias
Well, do rationalists really play Zendo better than people ignorant of the Art? In "Confirmation, Disconfirmation, and Information in Hypothesis Testing" Joshua Klayman and Young-Won Ha explain that, at least in the case of Wason's 2-4-6 task, they probably don't. The simplest version of the game is as follows. The game master comes up with a rule for a sequence of three numbers and provides an example that satisfies the rule. A player can run tests — provide a sequence and ask the master whether it satisfies the rule or not — and should guess the rule with as few tests as possible. A usual experiment goes like this: the master invents the rule "just any three positive numbers" and says 1 3 5. Then the player tries 2 4 6, 6 8 10, 10 12 14, settles on the rule "three integers spaced two apart", and gets it wrong.
And we say it is confirmation bias! He just wanted to confirm his rule! But this is straight up wrong. We don't know what he wanted to do. What he did was provide three sequences of numbers that satisfy the rule he was trying to test.
There is a difference between positive/negative tests and falsification. A positive test is a test of a sequence that your theory predicts will be positive; a negative test is a test of a sequence that your theory predicts will be negative. A falsification is a test that contradicts your theory — i.e. your theory predicts one thing but the test output is another. But how can we know which test will turn out to be a falsification? Well, we can only do it probabilistically. Let's say we have a space of all possible states, where theories are subsets of this space, consisting of the states that satisfy them.
What's then the probability that a positive test will give us a falsification? It is the number of states that are INSIDE our theory but OUTSIDE the correct one, divided by all the states INSIDE our theory. It is proportional to
But now consider that in real environments the thing you're trying to pin down is almost always a sparse, specific pattern — one disease out of thousands, the single rule the game master happened to pick — so the true rule T occupies a tiny corner of the space, while the hypotheses (H) we actually entertain tend to be broader or at least similar in complexity.
That means that the sheer number of states outside our theory is usually much larger than the number inside, and because the correct theory also (we assume) should be quite small, a negative test has a very small probability of producing a falsification. On the other hand, if both theories are quite small, finding where they don't intersect is not so hard. In simple terms, most of the examples that you can present will be just negative for both rules/theories, so presenting a random negative example usually gives you nothing.
So the situation usually looks more like this.
Or even like this.
Again, we can't know for sure which test will lead to a falsification: in the first picture only negative examples could falsify our theory, in the third only positive ones can, but we don't know in advance which picture applies to our situation, so we have to use some priors. And natural experience tells humans that the 2nd and 3rd are more probable than the 1st. So usually you just want to use positive examples much more often than negative ones — unless someone has constructed an experiment specifically to catch a person using this heuristic, which is very useful on average. But when you play a real game, such a trick will be exhausted very fast, and you will start to come up with really complex theories (which have a small number of correct states); and if a rationalist still confuses negative examples with falsification, he could waste a couple of rounds on useless tests.
In Positive Bias: Look Into the Dark Eliezer makes this distinction right away:
But then he goes:
And:
If you constructed a theory that can explain anything, then positive examples would actually be the only way to falsify it. In the 3rd picture you would have H = U. It would be like saying my hypothesis is "just three numbers": then 1 2 3 is a positive example, and -9 pi sqrt(3) is positive as well and will falsify your theory, because it doesn't satisfy the target rule. It is still right to point out that phlogiston theory is a bad theory, but I don't think the reason is positive bias.
Of course, this is just the most famous example of confirmation bias, and refuting it doesn't refute confirmation bias in general. And from my personal experience this is the bias I have experienced most myself — it is really hard to change your opinion. Even though the Backfire effect is probably wrong. But I think there are plenty of rational reasons to hold your beliefs strongly that make it less dark than Eliezer presents. For example, people with a lot of experience are usually already exposed to a lot of information that confirms their view, and they could have tried to correct for this intentionally, but then they would sacrifice time they could spend developing expertise or building a reputation for their view — and that is fine in science, for example, where one person can't possibly gather all the evidence, and it is genuinely better if he works on his own theory. It is not good for his epistemic fairness, but who cares about that if he is successful and society benefits from it too. And I think it works in regular life as well: there are non-epistemic costs to switching beliefs, and always being unsure and flipping your beliefs at the slightest bit of new evidence hurts a lot.
Nevertheless, I think there is a very important kind of "bad" reason for holding on to your beliefs. There is a beautiful post by Scott: Guilt: Another Gift Nobody Wants. In short, it explains why it is very important to have a mechanism that convinces other people you are not incentivised to do bad stuff when no one is looking. One example is guilt: it is an intrinsic property of almost every human being, and people know other people have it. But roughly the same logic could work for holding strong beliefs. If you really need people to invest in your theory or vote for your party, you had better say you are a hundred percent sure it is the right thing — and when you say that, it is better to be honest about it, because people can notice when you are insincere. Obviously, for politicians it doesn't hold so well — they are the finest masters of lies — but for other people it seems to work quite well.
But, as I will discuss more later, in my opinion it is hard to disentangle a general bias — a rigid property of everyone's thinking — from other psychological effects or just random human errors.
Planning fallacy
Well, here I am not even going into research and papers, but will just appeal to my own experience, so you can rightfully skip this section right away.
I think it is quite apparent that we do actually underestimate the time we need to do things, especially at work, where we don't want to do anything at all, haha. There are some exceptions: I think being early or late differs dramatically across cultures, and in some cultures people are usually on time or only a little bit late. And most people are usually on time at airports.
The hypothesis that we just imagine how things can go right and run with it may well be true, but the alternative — thinking about all possible failures — is not necessarily always useful. First of all, there is only a finite number of ways things can go right, and an infinite number of ways everything can go wrong, and even keep going wrong indefinitely. So there is an asymmetry in the probability space towards infinity: the most probable outcome sits somewhere close to zero, and to the right there is a slowly decaying curve, like a Lévy distribution with no finite mean or variance. We still usually get things done, because most of the failures don't happen; but when we fail, we simply abandon the task. Also, it is not always the case that you need to predict everything in advance — quite the opposite. More often you will change things on the fly, other people will delay something, or the goal will shift slightly, so it won't matter that you miscalculated something. And it is not true that people never think about failures: we take medicine on a trip, we ask someone for help in advance when past experience tells us we won't manage on our own, we set reminders and alarms, we put our things in lockers, and so on. The only question is how much we think about failures; and though we systematically underestimate time and costs, that doesn't mean it isn't optimal behaviour. Given the unpredictable changes in circumstances and the infinite complexity of the possible failures we'd have to account for, it is perhaps too costly to always estimate correctly — or maybe it is even impossible not to underestimate on average, due to the divergent nature of the distribution.
So, do humans need fixing in the end?
We looked at several examples of cognitive biases: some are refuted, some are up for interpretation, and it does seem like people are biased somewhat — but the evidence that their condition is very serious, that it paralyzes all decisions and prevents humanity from functioning normally, is not overwhelming.
But wait, it seems apparent that people do very stupid things on the global scale — how can you argue with that? Let's first go back to where we started: we have to nail down what a bias is. We already know that a bias is a difference between a human decision and a rational theory.
But what's also important here is that we look only at the general rules of how the human mind functions. If one human makes a mistake and another doesn't, it is not a bias. But what if a lot of humans make this mistake, but not all? Take psychotherapy, for example: surely a lot of people have narcissistic disorder, autism, or ADHD; they could even gather in a community and declare that the human mind is susceptible to having too high an opinion of itself, and that we should fight that in all human beings. But what they actually need is to accept that it is their own peculiarity — not everybody has it — and it is good to trace the reason you acquired it and adapt accordingly.
In my opinion, a lot of evil in life comes from people with psychological problems and deviations, and definitely not all of it comes from cognitive biases, as Eliezer claimed (there is also the Moloch problem and other things — I'm obviously not saying all evil comes from not seeing a psychotherapist).
I am not talking about people who are just depressed or have an attention problem, but about people like Stalin, Mao Zedong, and Hitler. Maybe they could have used a little more rationality and a little less overconfidence — before you decide you can invent a way to grow rice twice as efficiently, only to have cannibalism spread across your country and more casualties than in World War II. But probably the reason they didn't have enough sanity was that they were narcissistic psychos.
How much of all evil comes from cognitive biases is not obvious to me. I think this is a very important thing to research before you put investment in the Art at the same level of ambition as saving humanity.
But there is also a point to be made that some irrational behaviours (including biases) don't need to be fixed because they are load-bearing — things that are irrational in theory but optimal given the engineering of our brains. Some rational actions cause too much stress and carry too much cost, given the limited ability of humans to process feelings and thoughts.
Some rational decisions will not account for the fragmented nature of our brain — you can consciously assemble a perfectly unbiased Bayesian argument while silencing the parts of your intuition that tell you it is wrong for some irrational reason, but forcing yourself to do it will come at a cost greater than the rational benefit.
It is similar to traumas — defensive mechanisms of our brain that shield away bad experiences we couldn't comprehend as kids and cause disharmony in our minds. But these were useful at some point as well.
You probably won't tell a guy with a fear of flying that it is irrational that he can't sit calmly on a plane. This art of accounting for our own design is quite different from the art of aligning yourself with an optimal Bayesian agent.
Rationalists will tell you that it is rational to account for all that as well, but, as with many things I will mention later, there is a pattern: when someone (usually Eliezer himself) points out some limits of rationality, rationalists will agree but keep insisting on the importance of the Art, just with this or that caveat. The problem here is that for some things it is not enough to merely account for them — some things just destroy the usefulness of your approach. Because once you account for this and for that, it is not obvious, in the end, whether you aren't just reinventing the wheel from scratch, doing all the same things but with a lot more effort.
In the end, as we already mentioned, Gigerenzer claims that, though biases are sometimes real mistakes in lab settings, they are also simplifications that work great in real life. And substituting more complicated thinking for them can backfire.
One such example is probabilities and Bayes' theorem. Bayes' theorem itself is not tractable even for modern computers; neural networks sometimes use simplifications like the ELBO, or derivatives of the posterior distribution, as in diffusion models. Maybe more useful in practice is Bayes' theorem in odds form, which — apart from some problems with priors — basically tells you to account for the control: don't only seek evidence that appears while your intervention is present, but also while it is absent. It is akin to positive bias, where people seek only to test examples that match the rule they invented, which is probably also the cause of many superstitions and of alternative medicine — though these are not necessarily irrational. Yes, not everything that can be destroyed by the truth is irrational: sometimes these are psychologically comforting beliefs, sometimes it is just an error on the side of caution (it is easier to do this ritual than to risk not doing it), and, as Scott Alexander pointed out, sometimes culture is wiser than individual people can explain. But I am not defending it outright — of course, in the modern world, things like controlled trials are blessings, and anti-vaxxers basically cause people to die. Which makes not forgetting about the control useful sometimes, even in everyday life.
But what about applying numerical probabilities?
Eliezer himself argued in When (Not) To Use Probabilities that we should only use numbers that are based on other numbers, and that we should calculate, then throw away the result and follow our intuition. Do you think rationalists followed this guidance? At least in my experience, in the Russian community many people rationalized their decision-making with made-up numbers. And many thought that following your intuition is wrong when the rational arguments say otherwise. The famous p-doom is also one of those cultural things that stuck despite Eliezer's warnings, and Eliezer himself uses it. But even if we treat this lightly and believe every word of the essay, the question remains: if one should throw away the math and follow intuition, then maybe you could just skip the math part altogether?
But I'm getting ahead of myself. Before we fully dive into the dissection of applied rationality practices, I need to discuss one more point that should be considered in understanding the usefulness of these practices.
On values
We already discussed that biases can be rational given some reinterpretation, which often involves assuming a different set of values.
You can interpret hyperbolic discounting as a mistake, or you can interpret it as a value or preference. You can basically make this flip-flop for every bias, and it can sound artificial, but I will argue that the negative framing of some biases stems from a misunderstanding of human values.
But btw sometimes people also say it is hyperbolic discounting when it is not, because they just didn't account for something that makes present more important, often it is just psychology. For example, your belly craves snacks because your consciousness punishes you for every wrong decision and tries to push you to the limit, so sweet treats compensate for that at least a little bit and slacken the tension — maybe not in an ideally healthy way, yes, but your conscious decisions can be much worse. Sometimes you also come across some new paper on how pears are actually carcinogenic, but you still want pears and eat them while complaining about hyperbolic discounting — when actually it is a good defensive mechanism of your brain resisting your conscious decision, because it has better priors and behaves conservatively. As in many cases, a person can unknowingly do something for rational reasons while believing his reasons are irrational, or while failing to understand his reasons at all.
Take Bryan Johnson, for example: he is trying to optimize his life with an evidence-based approach, without conservatism, without risk-avoidance, at his best conscious judgement — and I wouldn't want to live such a life (except that he is crazy rich, of course; that's a good bonus).
But what's more important is that sometimes instrumental convergence doesn't hold for us and we don't optimize for control over everything. We don't want a love affair with a puppet, and we don't want a climbing gym or a computer game to have a shortcut — an example that Eliezer himself mentions in Harmful Options.
Sometimes people play social games for the sake of the game, or argue in debates because they enjoy it. Rationalists often think our values are about winning a particular game or searching for truth, but actually it is also about fun and being in the process.
There is a great sequence of posts about what values the rationalist community is missing; the two most interesting to me are On green and On attunement.
Green is love of Nature, living as you are, living life, fully experiencing it, being in attunement. This is not necessarily the opposite of maximizing knowledge or pursuing goals in terms of your actions — you can find numerous justifications for Green in Eliezer's writings — but the posts argue that this is "green-according-to-white" or "green-according-to-blue", not Green itself. Being Green is a different state: not looking from above and merely modeling, but looking from the inside.
In The Unbearable Lightness of Being, Milan Kundera writes, "Happiness is the longing for repetition." Sometimes, calm, rural life with simple pleasures is better and happier than maximizing information, goods, and success. It is not always good to change everything, change Nature, change yourself, to optimize for something; there is a value to keeping everything as it is. Harry's urge to control everything in HPMOR is not a coincidence; it is inherent in rational ideology; it is the ambition to be powerful enough to extinguish a star to save one life. I don't think it is necessarily a bad thing, though. I am still an immortalist, but I do think that, applied to a simple life or your everyday cognition, this can become dangerous. And this is exactly the conflict that happens when you mix saving the world with self-help practices (Eliezer doesn't think the laws governing them are different but I do).
The point of bias is that it's a deviation from the utility-maximization theory of decision making, but fixing this also means that you want to control every aspect of your thinking to fit this abstract thing, rather than being yourself and giving up control.
I think it is very important to feel this distinction, and to shift your perspective on values away from this maximization paradigm, in order to decide which techniques feel right for you and which don't. But we've finally reached a point where I can't delay it any longer — it's time to discuss the mysterious beast: applied rationality itself!
On intuition vs conscious thinking
I think the notion that conscious thinking is more capable than intuition underlies almost all the techniques (I will avoid delving into too much detail here, like whether I mean System 1/2 or something else; I think in the end those details don't matter much for my point). I don't think it is completely wrong — conscious thinking is better for some things and worse for others — but what this notion misses is that the basic gears of cognition are actually better left untouched, because there is a great variety of subtleties that our intuition computes much better, and you will only make more mistakes without it.
One example is the goal-factoring technique. What it proposes is to start with some thing in question that we do or want to do, then list all its possible subgoals (using the button test to verify there is nothing more to it), and then find alternatives that cover all the subgoals without paying the costs of the thing in question that brought it to your attention in the first place. There is an example with grading students, where, after using the technique, our hero comes up with the idea of students grading each other. I think at this point a sane sceptic would raise a few questions: aren't we already using the same or a better algorithm by default, just unconsciously? Didn't our hero come up with his solution by simply inventing it with his intuition, and then justify it retrospectively via the technique? How often should we use this technique instead of default intuition?
The authors answered the first one in their opening section: quite often, when we are presented with two alternative options, we don't use our imagination to come up with a new alternative, but instead just weigh the pros and cons and choose. The key word here is "usually". Usually means there is an intuition in our brain that tells us whether we have to start searching for a new alternative or not, and usually it says "no". Sometimes we just come up with an alternative unconsciously; sometimes we realize that we want neither of the options and we search for a solution — it has happened to every person on Earth at least once in their life. But this isn't even how our brain works most of the time — launching the whole search and staying focused on it. It comes in pieces: some random thought, then another, stuck together across different times and contexts, and then a bright thought comes to mind that is the cumulative knowledge of all the alternatives, pros and cons, and desires we encountered, but computed in some distributed manner.
So, what does the technique give us? From the logic of the Sequences and rationality, you would assume there is a cognitive bias that creates some imbalance, and that we are already doing what the technique suggests, just not enough. But for a lot of techniques, including goal factoring, there isn't even a bias that they refer to. It seems like they have the assumption: we came up with some algorithm on a piece of paper that sounds right, so obviously humans in general do worse than that by default. Aversion Factoring "is not derived from any particular body of psych research, though it draws lightly on trigger-affect patterns and exposure therapy and takes advantage of the framework of reductionism." Internal Double Crux is not related to any cognitive bias; it is drawn from IFS, which is a therapy, and the approach to traumas should, I think, be more careful than a random rationality-retreat practice. TAPs are just from psychology as well, and even those I don't like. Of course, this doesn't hold for all of them: Bucket errors fight the representativeness heuristic (which is related to the conjunction fallacy we discussed before), Units of Exchange are related to the Sunk cost fallacy and loss aversion, and Murphyjitsu works specifically against the planning fallacy.
What would happen to our hero if he didn't apply the technique? I would guess that the idea of students grading each other had already crossed his mind, and that he probably noted it, would recall it again, and would subconsciously weigh all the factors that are listed in the technique — along with other ideas that he would dismiss. Because our brain does all of this by default. Yes, there is a possibility that he would miss this idea and continue to struggle without the technique, but there is always such a possibility. Making a mistake is not a bias, and not irrational, because the solution is apparent only after careful thinking, and thinking itself costs something. Obsessively applying this technique to everything you do can give you a neurosis, and applying it just a little can get you nowhere, because you would still miss important stuff. In other words, it is not "anecdotally strong" and obvious that the status quo is not the optimum — EVEN if there is some strong evidence of a particular bias (which is not always as convincing as we saw in the first section) — because, apart from knowing there is a bias, you would somehow have to guess in which situations it will manifest; otherwise you'd have to apply the technique everywhere and, again, get a neurosis.
And there is more that could go wrong. Decomposing something actually doesn't give you the full picture. Usually people can weigh all the factors and form an impression of something intuitively, not because there is a simple logical explanation for why it is so, but rather because of a complex, emergent (oh, c'mon) relation. And people on the spectrum do this worse than others, which could be one of the reasons such techniques are appealing. But the right way to fix this dissonance is to train your brain to form an attitude towards complex things — instead of using some ruminating tool, just experiment and teach your intuition through experience.
Again, even if the bias actually exists, it is not only about how often to use such a thinking method (which, fairly often, is already used by default), but when to use it. So just bluntly practicing it is not obviously going to help.
We can think about it by comparison with pharmaceuticals. They cure some illnesses — i.e. some function that is not activated when it should be, because something is broken in what the organism usually does well — and we just switch this thing on when we need it. We don't recreate all the circuits of the human cell from scratch; we just use some molecule to activate an already existing pathway. In contrast with that approach, there is anti-aging research. Aging is not something that happens to some people sometimes; it is a bug we all have from the beginning. And that's why it is hard, almost impossible, to fix with the existing approach. No matter how smart we think we are, we still can't even remotely design from scratch anything as complicated as the biological circuits our cells already have. The same thing applies to the basic functions of our cognition. Yet.
You can probably tell that the situation is similar with even simpler techniques, like "notice confusion more than you do in your usual life", "think about what can go wrong more than you already do", and the very famous "actually try very hard for at least 5 minutes, you lazy bastard!"
But wait — what about Do The Math, Then Burn The Math and Go With Your Gut? It's already been accounted for! (again). Well, the same question: even if you go with intuition in the end, it doesn't free you from the cost of the calculations and the wasted time (
and money on psychotherapy) — how do you know when these calculations are useful? Well, maybe you can teach your intuition to recognize such situations with practice, or insights from the math could leak into your intuition and fix biases permanently! This is not impossible to imagine.On practicality
But what does the practice actually say?
My impression is that almost all the CFAR techniques are useless, and I don't need them to succeed in life at all. Though I would say that some ideas from Street Epistemology are useful, and they are related to the Double Crux technique. With Eliezer's Sequences it is only a little bit better: Affective Death Spirals explained how you can get into a cult — though he didn't convince me that he isn't creating one. A Human's Guide to Words was a good explanation of some concepts for my autistic mind. Noticing Confusion and the similar Split and Commit I probably used a few times (and it happened on its own a hundred more). But what if I am just a lazy, narcissistic, AuDHD person who didn't get the Teaching? It is very possible, and I don't weight my experience too heavily. After all, I am really lazy and narcissistic, and I have AuDHD :)
Most of the people I know from the Russian community fall into three categories: people who just came for the fun, the community, and the philosophical conversations; people who tried the techniques and found they didn't help them, like me; and very few people who are still using the techniques. That last category is usually the people who also say that you should use Bayes' theorem for everything, think that everybody is stupid except rationalists, who deploy every CFAR technique in a single chat message just to argue with you, and so on.
Then again, the experience of other organizers is similar: Tanya Miropolskaya shared that she had a negative experience with applied rationality and the community in general, and Slava Matyukhin said the techniques didn't help him. And, again, Scott Alexander wrote Extreme Rationality: It's Not That Great a long time ago.
There is a study that CFAR conducted to determine the effects of their program. The results are below. All variables are coded such that positive numbers indicate an improvement from pre-workshop to post-workshop, with (R) indicating that this involved reverse scoring. Effect size is the standardized mean difference using the pre-workshop standard deviation. † p<.10, * p<.05, ** p<.01, *** p<.001.
Improvement in Cognitive Biases was not significant; interestingly, they tested them with Stanovich questions. But, unfortunately, this is not great evidence for my point, because the study is terrible — they didn't have enough statistical power to detect even three of the four biases.
Anyway, another piece of evidence that CFAR wasn't a huge success is that they said so themselves and closed their workshops. There is a summary of what happened by Anna Salamon: my low-quality thoughts on why CFAR didn't get farther with a "real/efficacious art of rationality". There is another project that follows in CFAR's footsteps, but it looks even worse.
Another problem, which Scott also mentioned in his essay 17 years ago, is that rationality is a dual-purpose technology. It was developed for exploring AI Safety and then reoriented to help people with their biases. This simultaneously damaged our understanding of human cognition, by leaning towards artificial models instead of actual humans, and damaged the project itself, by shifting its focus from the epistemic exploration of these ideas towards saving humanity right now. CFAR were actively collaborating with MIRI and convincing people to switch to AI Safety. Sometimes people even say that the whole point of Effective Altruism was to bring people to AI Safety, and Robin Hanson said this about the whole idea of rationality (which is actually quite obvious from the Sequences, but I hadn't thought about it before).
There are some posts and videos by Brienne Yudkowsky that go to another level — not just practicing techniques to get some insights, but installing them as a habit of thought via TAPs. I tried to use this a long time ago, and it made me feel really bad about all of it. It is a kind of uncanny feeling — that rationalists still treat people like Spocks, even though they say otherwise.
I think another reason for this could be that neurodivergent people created and practiced all of it. A lot of neurodivergent people like me need help with basic things that are not as problematic for normal people. And they tried to use these applied-rationality techniques to solve those problems, when they actually should have used therapy. It feels better to use applied rationality, because you are ahead of everybody: you are not fixing your specific problems, you are fixing basic human biases! But that's not what you should be doing when you have akrasia, anxiety, or painful inner conflicts and self-blame.
I don't think therapy is a panacea; there are some problems with its effect sizes, as there are with medications. But the point of therapy is much more humble — helping a person with their unique problems, as opposed to solving all the problems of humanity. And in my experience it just works much better.
On overpromising
I think people with a lot of narcissistic ambition (like me) are affected by Eliezer the most. When I first met him on the pages of HPMOR, he was in the costume of a magical boy who can do anything using rationality and science, and conquer a world of stupid people who have biases while he doesn't (ok, ok, he does, but he very masterfully accounts for that, whatever). I went to my friends and started bragging about what I had found: this is the best book I have ever read. One of them said he had started it but couldn't stand the pretentiousness; I was furious and tried to explain to him that he understood nothing. Now I think he was right, and though HPMOR is still a really good story, it is actually quite strange morally. Eliezer intentionally made Harry rational instead of brave and loving, and, to disqualify these character traits even further, instead of standing up for Ron when Malfoy basically said he is stupid and pathetic, Harry casually agreed with him. Where Rowling praised love, Eliezer says: if you are stupid, you are not even interesting to me. But I am not going to spoil the whole video for you — just go and watch it.
And then there are the Sequences. From Scott's original Extreme Rationality: It's Not That Great:
Beisutsukai is another example of fiction, along with HPMOR and Three Worlds Collide, where Yudkowsky depicts rationalists as having superpowers. And these tales are built into the Sequences. He writes in A Sense That More Is Possible:
In Crisis of Faith:
Btw Robin Hanson apparently disagreed with that: Rational Me or We?
In Human Evil and Muddled Thinking:
So yeah, Eliezer never said that rationality is going to make you a superhuman, and that rationalists are warriors of Light against Evil in the world..
He also never wrote a few hundred posts listing his conversations with irrational strangers who disagreed with him, from which he learnt the Art and realized how stupid people really are and that we should help them, because Your Rationality is My Business. In each post, he didn't cite himself a hundred times with lots of pretentious words, and didn't talk about his own Enlightenment. And he didn't write a book about how everyone else could be wrong and you are right.
Though he wrote about this problem a little bit as well.
I am not saying that wanting a lot from yourself and speaking pretentiously is always wrong; what I am saying is that, conditioned on the fact that the practical results are quite questionable, and that even the cognitive biases are in some cases questionable, it becomes overpromising. And urging yourself to just Shut up and do the impossible! is not quite healthy.
And, btw, did he write The Bottom Line — claiming that people are desperately biased — back then, based on one Tversky study? (I am half-joking here)
From We Change Our Minds Less Often Than We Think:
And Yes, We Have Noticed The Skulls — so he wrote the whole sequence on how to not become a cult. That didn't save the community from a feeling of being exceptional and a responsibility to save humanity. Robin Hanson's opinion on this again, and some discussions of EA cultishness; and there are some people speaking out about psychologically unhealthy experiences at MIRI as well.
In the context of AI Safety, I genuinely think his arguments are good and his contributions are very important (I believe we are quite possibly screwed and everything), but that doesn't mean they are at the level where he can state that the apocalypse is 99% certain, and that everyone who says otherwise, or disagrees about the reasons, is just irrational, and that everyone who proposes at least something to solve it but different from what he thinks should be done is stupid. Even if he is a genius, I don't think it is rational to weight your own opinion so highly. He also encouraged people to ignore politics for a long time, and when people started to do good things there, he burst in saying you are doing everything wrong and started to propose radical things. He has also stated some strange things on unrelated topics, and made some strange public appearances.
So, in the end, I don't think he is a bad person of course — he has done a lot of good for humanity and his intentions are sincere — but it doesn't look to me like he is so much better at overcoming human biases than anybody else, as people might infer from his writings.
Conclusion
So, what do we have? Some biases were mostly refuted, either by experiment or by logical reinterpretation; debates over some are still hot; and some are quite solid, but it is still debatable how large their impact is, and whether it is useful to try to fix them given the constraints of our brains, and so on.
Our values are not about winning some game or achieving some goals, but rather about enjoying the process; so it is not always that important for us to know the truth, or to plan without mistakes, to put our lives on a pedestal of rationality.
And finally, the techniques don't seem to work all that well: after a couple of decades, we don't see very successful rationalists who use them in real life, and CFAR stopped running rationality workshops.
And that is what you would expect, I think, if you look impartially at the field of cognitive biases — without an unhealthy world-saving ambition, but with curiosity. What Eliezer proposed in his A Sense That More Is Possible made a lot of sense:
When you have so little data on how to fight biases, such poor-quality science in the field because of the replication crisis, and ongoing debates over whether biases make sense at all, it is quite hard to imagine that people will just be able to invent a full set of techniques from scratch, without any tests, and have it be effective. You would expect rationalists to be very careful with every fact, and to test each cognitive bias and intervention one by one. Instead, rationality tried to be more than science, but I think it turned out to be less effective than science in the end. Hiding behind the Bayesian approach, rationalists slapped the epistemic status "anecdotally strong" on everything, and developed a lot of random things not even related to biases — let alone tested whether the techniques actually fix biases. It is crazy that "anecdotally strong" is enough to claim that thinking really hard for an extra 5 minutes is going to improve your life (no problem with that at all). People did talk about opposition to Kahneman and Tversky, and about problems with biases, here on LessWrong — but very little. If you measured the publication bias favoring biases here against the amount of scientific evidence and debate, I think it would not come out in LessWrong's favour, which is strange, because you would expect a community that celebrates the crisis of faith to be really careful with the facts that underlie its basic beliefs. And instead, Eliezer writes such posts as the one about the conjunction fallacy, full of contempt towards everybody who dared to doubt REAL SCIENTISTS.
So I do think this project could be useful, but with much less overpromising and more care for proofs and scientific facts. Still, all of the above doesn't mean I think the rationalist community completely failed; I think it is still a very creative community, with very interesting values and philosophy, which is worth nurturing and saving. The rationalist community is not only about fixing cognitive biases. There is also just philosophy, culture, and adjacent things like effective altruism and AI Safety. How are things going there?
I want to leave a special place here for Astral Codex Ten — Scott Alexander's writings, which, though not directly affiliated with rationality, are still very much influenced by it. And a lot of them are probably among the best posts I have ever read on the internet. Apart from his professional insights and his broad knowledge, I like his intellectual honesty, which I think taught me about the spirit of rationality by personal example — maybe even more than abstract philosophy did. And I think this spirit of unaffiliated, honest blogging is one of the most valuable things about LessWrong.
I think the obsession of effective altruism and 80k Hours with pursuing the most efficient career — and 80k Hours' advice not to follow your passion — just sucks. It is the thing that affected me most in a bad way, I think, and only recently did I realize that what I want to do is completely unrelated to what is important to do on a humanity-wide scale. And not only that — it also plants a seed of anxiety, "I should save the world", which is a really unhealthy approach to your career, even if you happen to like it; and it forces you to always think about which direction is optimal, encouraging a switch of career, which is a new fashion but is actually very damaging to your expertise, especially if you have problems with productivity and aren't a genius. Switching careers is really bad for your expertise; it is not just about sunk costs, it is about value to lock in.
But I think that, over the years, the focus is shifting. I visited the recent EA Global conference, and though I didn't like most of the topics myself, I met a lot of nice, chill people who are really passionate about pursuing their careers, and who mostly discuss specific and useful things, like economics, AI, animal welfare, and so on. In the end, I think EA is useful while it stays healthy — with altruism and passion first, and efficiency second. Pledges to donate I especially endorse in that sense.
AI Safety is definitely the most important thing right now, and though, thankfully, it is not limited to LessWrong, the impact of Eliezer and other rationalists on the field was crucial. Even if Yudkowsky thinks we are all doomed and that everyone proposes only hopeless ideas, they propose them based on Yudkowsky's writings. Maybe someday someone like Yoshua Bengio will solve the problem, only because Eliezer turned him to it.
So, the community itself still has a lot of value, and its overall effect on humanity — and, even more importantly, on the people here — is net positive. But still, maybe there is a lesson from the story of cognitive biases and rationality that will help the rationalist community grow up?
tl;dr
I spent years deep in applied rationality and came out skeptical. Some biases are refuted, some are debated, some hold ground — but the scope of the effect can still be overstated. The values the Sequences are built on come from AI research and are tied to maximization and instrumental convergence, which is wrong for simple everyday life. CFAR closed its workshops in the end, and their techniques weren't even all based on biases, let alone tested to have any effect. The project overpromised, partly because it inherited a world-saving, AI-safety mission that pulled it away from careful science.
The culture, the writing (especially Scott Alexander), EA while it stays healthy, and AI safety are important and valuable. But the promise of getting Beisutsukai powers deserved far more testing and scepticism than it got.