We distinguish epistemic rationality (making your beliefs more accurate) from instrumental rationality (succeeding at getting what you want).
“rationality is about forming true beliefs[1] and making winning decisions”
Main tools for this: probability theory (bayesianism), and decision theory.
Feeling Rational
Being rational does not mean (contrary to traditional and popular expectations) being unemotional.
Emotions, including strong ones, can be appropriate responses to how things are.
Why Truth? And…
There are different reasons for seeking truth: intellectual curiosity, accomplishing goals, and morality.
Yudowsky generally likes the first two (curiosity is good even if it’s an emotion and trying to manipulate the world is what got us to science and helps in getting external verification), but is wary of the third, which can lead to internalizing wrong truth-seeking strategies.
Overcoming biases is the way to improve rationality.
… What’s a Bias, Again?
Biases are some of the obstacles we encounter when we’re trying to get to the truth.
Specifically, they are those obstacles that arise from our own mental machinery, which did not evolve to be maximally truth-seeking. They are ‘humanly universal’ and really hard (perhaps impossible?) to fully correct.
Availability
The availability heuristic consists of judging how frequent we consider an event to be based on how easily we can remember examples of it happening.
More vivid or publicized events (like murders or accidents) stick into our minds easily and make us misjudge their frequency viz-à-viz suicides or illnesses.
Related to it is the absurdity bias: “it has never happened so it can’t happen”. This also works with events one can’t recall or which happened long ago (examples related to flood damage and insurance).
Some Thoughts of mine
Yud’s two types of rationality are familiar to me, and also bring to mind his reference in IABED to intelligence as a combo of ‘predicting’ and ‘steering’, which map pretty well onto this.
I think he’s right about emotions not being irrational, but I think the argument for the low status of emotionalism is a complex and fraught topic, where ‘being capable of repressing your emotions’ is taken as a signal of general coolness and self-control. I suspect strong emotional judgments tend to be less truth-tracking than cooler-headed ones.
I am tempted to say that I value truth so much for moral reasons (second place, curiosity, third place, accomplishing goals. Maybe this is apartly a result of a strong humanistic education), but I fail to see how that would make it bad for arriving to truth-seeking strategies, at least in my flavor of truth-ethics[2].
No issues with the explanation of biases, except to caveat my acceptance of them with significant skepticism on my part about how much we can overcome them, even with great effort and by applying good tools. I suspect the expected alpha from doing this is pretty low: my superficial acquaintance with what many rationalists I’ve heard of have achieved seems pretty low (but then again… availability heuristic at play?). And a lot of their success seems to be mostly a result of selection effects from the population of smart nerds who get attracted to this stuff in the first place. There is no Royal Road to Rationality?
Absurdity bias obviously brings to mind Yud’s (and the rat’s) main hobbyhorse, i.e., the AGI Apocalypse. I need to study probability sometime soon, but a priori, I am very much inclined to just ruthlessly cut down to zero or to infinitesimals the probabilities of highly unlikely events that have never happened. I think I have mentioned more than once in my Substack that I really, really, really dislike the idea of and the arguments behind Pascalian wagers. Is this just my absurdity bias speaking? Perhaps. But for the moment, I am still going to be unreceptive to being cajoled into the moral trap of assigning highly speculative, highly unlikely events enough positive or negative expected value to make them urgent. I feel that one needs to tightly bound this kind of reasoning[3].
Note: I used ChatGPT for minimal proofreading/copyediting - fixing typos, grammar mistakes and suggestions of awkward or unidiomatic phrasing. The text, views, and responsibility for the post are mine.
Which is complicated. I am at the moment a contractarian anti-realist in morality, so my passion for truth is something I perceive as just… my strongest and most terminal conscious preference, with quasi-religious overtones (distorting the truth as a crime against Reality).
As I said, I need to study probability, but the superficial acquaintance I have with the field is of the type that makes me feel uncomfortable with back of the envelope calculations and made-up probabilities. At the moment I am reading Annie Duke’s Thinking in Bets - perhaps it will update me more positively in this regard.
Crossposted (with small tweaks) from my Substack.
What Do I Mean By “Rationality”?
Feeling Rational
Why Truth? And…
… What’s a Bias, Again?
Availability
Some Thoughts of mine
Note: I used ChatGPT for minimal proofreading/copyediting - fixing typos, grammar mistakes and suggestions of awkward or unidiomatic phrasing. The text, views, and responsibility for the post are mine.
where truth is explained as correspondence of beliefs with reality.
Which is complicated. I am at the moment a contractarian anti-realist in morality, so my passion for truth is something I perceive as just… my strongest and most terminal conscious preference, with quasi-religious overtones (distorting the truth as a crime against Reality).
As I said, I need to study probability, but the superficial acquaintance I have with the field is of the type that makes me feel uncomfortable with back of the envelope calculations and made-up probabilities. At the moment I am reading Annie Duke’s Thinking in Bets - perhaps it will update me more positively in this regard.