Against (some of) rationality

lionrock

TL;DR: Before fighting a bias, ask yourself why it's there.

Epistemic status: Uncertain

At first, it seems almost self evident that if something feels more “rational”, it’s better. However, in reality, this heuristic leads to a lot of incorrect actions. A classic example is a someone thinking revenge is irrational, and suppressing his wish to take revenge. This will probably harm him, as people will be less reluctant to take actions that would get others to take revenge.

Some of the sequences are posts along the lines of “Here is an example of a bias: When one group of people are asked to estimate P(A) and another group are asked to estimate P(something that can only happen if A is true), people in the first group sometimes predictably give a lower probability than the second. Therefore, people are irrational.”

However, I think this is somewhat closer to bounded rationality. People use heuristics that work most of the time, and you can find experiments to show that people sometimes make choices that aren't the best choice, but it's much harder to figure out which end of the rope to pull or if there is even irrationality at all.

The claim I am trying to make here is that snuffing out biases is a Chesterton's fence situation. Often, we may think we found a proof of irrationality (e.g. the revenge example) but in reality the mechanism behind the supposed irrationality is there for a good reason.

I am not suggesting that we just say "oh, there's a bias, probably there for the best". What I am suggesting is instead of thinking "here is a failure mode in our rationality, we should avoid this", we should aim for "here is a failure mode. It's there because of X. X is bad, therefore we should avoid this failure mode"

Examples:

When people do something bad to me, I want to do something bad to them. Why is that? Oh, probably so they won't do bad things in the future. I should continue plotting revenge.
I want to take [addictive action] even though it's seems negative. Why is that? Probably because the addiction hacks my feedback loop. I should try to avoid it.
When I see a drowning child^[1] next to me, I jump in to save them, but when they are drowning far away, I don't care as much. Why is that? Maybe it's because I'm cooperating with people near me to save each other's children? Maybe I want to save children near me as a virtue signal? In this case, I should probably save drowning children near me even if it will cost as much as saving two drowning children far away. Maybe I just have a hard time thinking about children far away, even though I care as much. In this case, I should focus a lot of resources on far away drowning children. And perhaps, because I'm having a hard time figuring it out, I should balance the two options.
Why do people assign higher probabilities to events with more details? Perhaps^[2] it's because when playing two truths and a lie, people overdetail the true claims. Maybe even trivia quizzes do the same, in which case more detail IS evidence of being more likely. However, we should not let this heuristic leak into non-trivia questions. Or perhaps it's because the details let us see how the event could happen. I would give a lower probability to P=NP than to (P=NP and [proof of P=NP that I went over rigorously] works). In this case, we should think about how events could happen before assigning them a probability.

^{^}
If someone has a better link to the drowning child thought experiment, please tell me.
^{^}
It seems hard to determine the difference between "humans have the heuristic that more details is more likely to be true in trivia" and "humans are biased to believe things are more likely if they have more details". But maybe.

Would it be a good summary of this post to say that our biases come from heuristics that are sometimes correct? That, at least, is true.

It does not follow that rationality is bad. Recognizing when our heuristics are or are not valid matters for rationality, of course, and usually we can tell when using a certain heuristic is acceptable.

I don't disagree that "in reality the mechanism behind the supposed irrationality is there for a good reason" (at least, in a lot of cases), but we don't often run into Chesterton's fence just by countering biases.

At the same time, Chesterton's fence is a useful heuristic that we need to consider as part of rationality.

I claim we DO run into Chesterton's when counter most biases, and that naively countering a bias is not always good (though I have never found it to be devastating).
I didn't claim rationality is bad. I claimed some parts of rationality are bad. I am currently trying to purge the bad parts from my thinking process (although I think rationality was net positive even before I did so)
I think a good summary of this post would be "we sometimes call heuristics biases, and you should think to yourself why a bias is there before countering it". I guess I believe that evolution would remove most biases so we better have a good explanation why one sunk through

“Or perhaps it’s because the details let us see how the event could happen. I would give a lower probability to P=NP than to (P=NP and [proof of P=NP that I went over rigorously] works). In this case, we should think about how events could happen before assigning them a probability.”

I believe what you really meant to say was that you expect

Otherwise (as I’m sure you know), this breaks the conjunction rule. This tweak retains your argument without breaking probability: more information may increase your assignment of probabilities.

What do you mean breaks the conjunction rule? conjunction fallacy is assigning a lower probability to A than to A and B. A = (P=NP), B = [proof] works. If I saw B, I would believe A to be true, but I need to see B.

For example, in one experiment, 68% of the subjects ranked it more likely that “Reagan will provide federal support for unwed mothers and cut federal support to local governments” than that “Reagan will provide federal support for unwed mothers.”

This is partially because thinking about cutting support to local governments as an option... OK I'm probably nitpicking here.