LESSWRONG
Rationality: A-Z
LW

Rationality: A-Z
Map and Territory
Predictably Wrong
Fake Beliefs
Noticing Confusion
Mysterious Answers
How to Actually Change Your Mind
Overly Convenient Excuses
Politics and Rationality
Against Rationalization
Against Doublethink
Seeing with Fresh Eyes
Death Spirals
Letting Go
The Machine in the Ghost
The Simple Math of Evolution
Fragile Purposes
A Human's Guide to Words
Mere Reality
Lawful Truth
Reductionism 101
Joy in the Merely Real
Physicalism 201
Quantum Physics and Many Worlds
Science and Rationality
Mere Goodness
Fake Preferences
Value Theory
Quantified Humanism
Becoming Stronger
Yudkowsky's Coming of Age
Challenging the Difficult
The Craft and the Community

Rationality: A-Z

Rationality: A-Z (or "The Sequences") is a series of blog posts by Eliezer Yudkowsky on human rationality and irrationality in cognitive science. It is an edited and reorganized version of posts published to Less Wrong and Overcoming Bias between 2006 and 2009. This collection serves as a long-form introduction to formative ideas behind Less Wrong, the Machine Intelligence Research Institute, the Center for Applied Rationality, and substantial parts of the effective altruist community. Each book also comes with an introduction by Rob Bensinger and a supplemental essay by Yudkowsky.

The first two books, Map and Territory and How to Actually Change Your Mind, are available on Amazon (printed and e-book version).

The entire collection is available as an e-book and audiobook. A number of alternative reading orders for the essays can be found here, and a compilation of all of Eliezer's blogposts up to 2010 can be found here.

Start Reading

Map and Territory

0 / 49 posts readlog in to track progress

What is a belief, and what makes some beliefs work better than others? These four sequences explain the Bayesian notions of rationality, belief, and evidence. A running theme: the things we call “explanations” or “theories” may not always function like maps for navigating the world. As a result, we risk mixing up our mental maps with the other objects in our toolbox.

Predictably Wrong

This, the first book of "Rationality: AI to Zombies" (also known as "The Sequences"), begins with cognitive bias. The rest of the book won’t stick to just this topic; bad habits and bad ideas matter, even when they arise from our minds’ contents as opposed to our minds’ structure.

It is cognitive bias, however, that provides the clearest and most direct glimpse into the stuff of our psychology, into the shape of our heuristics and the logic of our limitations. It is with bias that we will begin.

47 min read
Preface
Biases: An Introduction
Scope Insensitivity
The Martial Art of Rationality
Availability
What's a Bias?
Burdensome Details
What Do We Mean By "Rationality"?
Planning Fallacy
Why Truth?
Feeling Rational
The Lens That Sees Its Flaws
Fake Beliefs

An account of irrationality would be incomplete if it provided no theory about how rationality works—or if its “theory” only consisted of vague truisms, with no precise explanatory mechanism. This sequence asks why it’s useful to base one’s behavior on “rational” expectations, and what it feels like to do so.

33 min read
Making Beliefs Pay Rent (in Anticipated Experiences)
A Fable of Science and Politics
Belief in Belief
Religion's Claim to be Non-Disprovable
Professing and Cheering
Belief as Attire
Pretending to be Wise
Applause Lights
Noticing Confusion
40 min read
Focus Your Uncertainty
What is Evidence?
Scientific Evidence, Legal Evidence, Rational Evidence
How Much Evidence Does It Take?
Einstein's Arrogance
Occam's Razor
Your Strength as a Rationalist
Absence of Evidence Is Evidence of Absence
Conservation of Expected Evidence
Hindsight Devalues Science
Illusion of Transparency: Why No One Understands You
Expecting Short Inferential Distances
Mysterious Answers

This sequence asks whether science resolves the problems raised so far. Scientists base their models on repeatable experiments, not speculation or hearsay. And science has an excellent track record compared to anecdote, religion, and . . . pretty much everything else. Do we still need to worry about “fake” beliefs, confirmation bias, hindsight bias, and the like when we’re working with a community of people who want to explain phenomena, not just tell appealing stories?

83 min read
Fake Explanations
Guessing the Teacher's Password
Science as Attire
Fake Causality
Semantic Stopsigns
Mysterious Answers to Mysterious Questions
The Futility of Emergence
Say Not "Complexity"
Positive Bias: Look Into the Dark
Lawful Uncertainty
My Wild and Reckless Youth
Failing to Learn from History
Making History Available
Explain/Worship/Ignore?
"Science" as Curiosity-Stopper
Truly Part Of You
Interlude
The Simple Truth

How to Actually Change Your Mind

0 / 81 posts readlog in to track progress

This truth thing seems pretty handy. Why, then, do we keep jumping to conclusions, digging our heels in, and recapitulating the same mistakes? Why are we so bad at acquiring accurate beliefs, and how can we do better? These seven sequences discuss motivated reasoning and confirmation bias, with a special focus on hard-to-spot species of self-deception and the trap of “using arguments as soldiers”.

Overly Convenient Excuses

This sequence focuses on questions that are as probabilistically clear-cut as questions get. The Bayes-optimal answer is often infeasible to compute, but errors like confirmation bias can take root even in cases where the available evidence is overwhelming and we have plenty of time to think things over.

65 min read
Rationality: An Introduction
Tsuyoku Naritai! (I Want To Become Stronger)
The Proper Use of Humility
Tsuyoku vs. the Egalitarian Instinct
The Third Alternative
Lotteries: A Waste of Hope
New Improved Lottery
But There's Still A Chance, Right?
The Fallacy of Gray
Absolute Authority
How to Convince Me That 2 + 2 = 3
Infinite Certainty
0 And 1 Are Not Probabilities
Your Rationality is My Business
Politics and Rationality

Now we move into a murkier area. Mainstream national politics, as debated by TV pundits, is famous for its angry, unproductive discussions. On the face of it, there’s something surprising about that. Why do we take political disagreements so personally, even when the machinery and effects of national politics are so distant from us in space or in time? For that matter, why do we not become more careful and rigorous with the evidence when we’re dealing with issues we deem important?

36 min read
Politics is the Mind-Killer
Policy Debates Should Not Appear One-Sided
The Scales of Justice, the Notebook of Rationality
Correspondence Bias
Are Your Enemies Innately Evil?
Reversed Stupidity Is Not Intelligence
Argument Screens Off Authority
Hug the Query
Rationality and the English Language
Human Evil and Muddled Thinking
Against Rationalization

The last sequence focused in on how feeling tribal often distorts our ability to reason. Now we'll explore one particular cognitive mechanism that causes this: much of our reasoning process is really rationalization—story-telling that makes our current beliefs feel more coherent and justified, without necessarily improving their accuracy.

50 min read
Knowing About Biases Can Hurt People
Update Yourself Incrementally
One Argument Against An Army
The Bottom Line
What Evidence Filtered Evidence?
Rationalization
A Rational Argument
Avoiding Your Belief's Real Weak Points
Motivated Stopping and Motivated Continuation
Fake Justification
Is That Your True Rejection?
Entangled Truths, Contagious Lies
Of Lies and Black Swan Blowups
Dark Side Epistemology
Against Doublethink

This short sequence explores another cognitive pattern that hinders our ability to update on evidence: George Orwell's 'doublethink' - the attempt to deceive oneself.

17 min read
Doublethink (Choosing to be Biased)
No, Really, I've Deceived Myself
Belief in Self-Deception
Moore's Paradox
Don't Believe You'll Self-Deceive
Seeing with Fresh Eyes

On the challenge of recognizing evidence that doesn’t fit our expectations and assumptions.

34 min read
Anchoring and Adjustment
Priming and Contamination
Do We Believe Everything We're Told?
Cached Thoughts
Original Seeing
The Virtue of Narrowness
Stranger Than History
The Logical Fallacy of Generalization from Fictional Evidence
We Change Our Minds Less Often Than We Think
Hold Off On Proposing Solutions
The Genetic Fallacy
Death Spirals

Leveling up in rationality means encountering a lot of interesting and powerful new ideas. In many cases, it also means making friends who you can bounce ideas off of and finding communities that encourage you to better yourself. This sequence discusses some important hazards that can afflict groups united around common interests and amazing shiny ideas, which will need to be overcome if we’re to get the full benefits out of rationalist communities.

68 min read
The Affect Heuristic
Evaluability (And Cheap Holiday Shopping)
Unbounded Scales, Huge Jury Awards, & Futurism
The Halo Effect
Superhero Bias
Affective Death Spirals
Resist the Happy Death Spiral
Uncritical Supercriticality
Evaporative Cooling of Group Beliefs
When None Dare Urge Restraint
Every Cause Wants To Be A Cult
Two Cult Koans
Asch's Conformity Experiment
On Expressing Your Concerns
Lonely Dissent
Cultish Countercultishness
Letting Go

Our natural state isn’t to change our minds like a Bayesian would. Getting the people in opposing tribes to notice what they’re really seeing won’t be as easy as reciting the axioms of probability theory to them. As Luke Muehlhauser writes, in The Power of Agency:

You are not a Bayesian homunculus whose reasoning is “corrupted” by cognitive biases.

You just are cognitive biases.

Confirmation bias, status quo bias, correspondence bias, and the like are not tacked on

...
44 min read
Singlethink
The Importance of Saying "Oops"
The Crackpot Offer
Just Lose Hope Already
The Proper Use of Doubt
You Can Face Reality
The Meditation on Curiosity
No One Can Exempt You From Rationality's Laws
Leave a Line of Retreat
Crisis of Faith
The Ritual

The Machine in the Ghost

0 / 53 posts readlog in to track progress

Why haven’t we evolved to be more rational? Even taking into account resource constraints, it seems like we could be getting a lot more epistemic bang for our evidential buck. To get a realistic picture of how and why our minds execute their biological functions, we need to crack open the hood and see how evolution works, and how our brains work, with more precision. These three sequences illustrate how even philosophers and scientists can be led astray when they rely on intuitive, non-technical evolutionary or psychological accounts. By locating our minds within a larger space of goal-directed systems, we can identify some of the peculiarities of human reasoning and appreciate how such systems can “lose their purpose”.

The Simple Math of Evolution

The first sequence of The Machine in the Ghost aims to communicate the dissonance and divergence between our hereditary history, our present-day biology, and our ultimate aspirations. This will require digging deeper than is common in introductions to evolution for non-biologists, which often restrict their attention to surface-level features of natural selection.

83 min read
Minds: An Introduction
The Power of Intelligence
An Alien God
The Wonder of Evolution
Evolutions Are Stupid (But Work Anyway)
No Evolutions for Corporations or Nanodevices
Evolving to Extinction
The Tragedy of Group Selectionism
Fake Optimization Criteria
Adaptation-Executers, not Fitness-Maximizers
Evolutionary Psychology
An Especially Elegant Evpsych Experiment
Superstimuli and the Collapse of Western Civilization
Thou Art Godshatter
Fragile Purposes

This sequence abstracts from human cognition and evolution to the idea of minds and goal-directed systems at their most general. These essays serve the secondary purpose of explaining the author’s general approach to philosophy and the science of rationality, which is strongly informed by his work in AI.

72 min read
Belief in Intelligence
Humans in Funny Suits
Optimization and the Intelligence Explosion
Ghosts in the Machine
Artificial Addition
Terminal Values and Instrumental Values
Leaky Generalizations
The Hidden Complexity of Wishes
Anthropomorphic Optimism
Lost Purposes
A Human's Guide to Words

This sequence discusses the basic relationship between cognition and concept formation. 37 Ways That Words Can Be Wrong is a guide to the sequence.

165 min read
The Parable of the Dagger
The Parable of Hemlock
Words as Hidden Inferences
Extensions and Intensions
Similarity Clusters
Typicality and Asymmetrical Similarity
The Cluster Structure of Thingspace
Disguised Queries
Neural Categories
How An Algorithm Feels From Inside
Disputing Definitions
Feel the Meaning
The Argument from Common Usage
Empty Labels
Taboo Your Words
Replace the Symbol with the Substance
Fallacies of Compression
Categorizing Has Consequences
Sneaking in Connotations
Arguing "By Definition"
Where to Draw the Boundary?
Entropy, and Short Codes
Mutual Information, and Density in Thingspace
Superexponential Conceptspace, and Simple Words
Conditional Independence, and Naive Bayes
Words as Mental Paintbrush Handles
Variable Question Fallacies
37 Ways That Words Can Be Wrong
Interlude
An Intuitive Explanation of Bayes's Theorem

Mere Reality

0 / 78 posts readlog in to track progress

What kind of world do we live in? What is our place in that world? Building on the previous sequences’ examples of how evolutionary and cognitive models work, these six sequences explore the nature of mind and the character of physical law. In addition to applying and generalizing past lessons on scientific mysteries and parsimony, these essays raise new questions about the role science should play in individual rationality.

Lawful Truth

Just as it was useful to contrast humans as goal-oriented systems with inhuman processes in evolutionary biology and artificial intelligence, it will be useful in the coming sequences of essays to contrast humans as physical systems with inhuman processes that aren’t mind-like.

We humans are, after all, built out of inhuman parts. The world of atoms looks nothing like the world as we ordinarily think of it, and certainly looks nothing like the world’s conscious denizens as we ordinarily think of them. As Giulio Giorello put the point in an interview with Daniel Dennett: “Yes, we have a soul. But it’s made of lots of tiny robots."

We start with a sequence on the basic links between physics and human cognition.

56 min read
The World: An Introduction
Universal Fire
Universal Law
Is Reality Ugly?
Beautiful Probability
Outside the Laboratory
The Second Law of Thermodynamics, and Engines of Cognition
Perpetual Motion Beliefs
Searching for Bayes-Structure
Reductionism 101
51 min read
Dissolving the Question
Wrong Questions
Righting a Wrong Question
Mind Projection Fallacy
Probability is in the Mind
The Quotation is not the Referent
Qualitatively Confused
Think Like Reality
Chaotic Inversion
Reductionism
Explaining vs. Explaining Away
Fake Reductionism
Savanna Poets
Joy in the Merely Real

..Do not all charms fly

At the mere touch of cold philosophy?

There was an awful rainbow once in heaven:

We know her woof, her texture; she is given

In the dull catalogue of common things.

—John Keats, Lamia
43 min read
Joy in the Merely Real
Joy in Discovery
Bind Yourself to Reality
If You Demand Magic, Magic Won't Help
Mundane Magic
The Beauty of Settled Science
Amazing Breakthrough Day: April 1st
Is Humanism A Religion-Substitute?
Scarcity
The Sacred Mundane
To Spread Science, Keep It Secret
Initiation Ceremony
Physicalism 201

Can we ever know what it’s like to be a bat? Traditional dualism, with its immaterial souls freely floating around violating physical laws, may be false; but what about the weaker thesis, that consciousness is a “further fact” not fully explainable by the physical facts? A number of philosophers and scientists have found this line of reasoning persuasive. If we feel this argument’s intuitive force, should we grant its conclusion and ditch physicalism?

We certainly shouldn’t

...
113 min read
Hand vs. Fingers
Angry Atoms
Heat vs. Motion
Brain Breakthrough! It's Made of Neurons!
When Anthropomorphism Became Stupid
A Priori
Reductive Reference
Zombies! Zombies?
Zombie Responses
The Generalized Anti-Zombie Principle
GAZP vs. GLUT
Belief in the Implied Invisible
Zombies: The Movie
Excluding the Supernatural
Psychic Powers
Quantum Physics and Many Worlds

Quantum mechanics is our best mathematical model of the universe to date, powerfully confirmed by a century of tests. However, interpreting what the experimental results mean - how and when the Schrödinger equation and Born's rule interact - is a topic of much contention, with the main disagreement being between the Everett and the Copenhagen interpretations.

Yudkowsky uses this scientific controversy as a proving ground for some central ideas from previous sequences: map-territory distinctions, mysterious answers, Bayesianism, and Occam’s Razor.

115 min read
Quantum Explanations
Configurations and Amplitude
Joint Configurations
Distinct Configurations
Collapse Postulates
Decoherence is Simple
Decoherence is Falsifiable and Testable
Privileging the Hypothesis
Living in Many Worlds
Quantum Non-Realism
If Many-Worlds Had Come First
Where Philosophy Meets Science
Thou Art Physics
Many Worlds, One Best Guess
Science and Rationality

The final sequence in this book tie these ideas together, and draws some conclusions on the strength of our scientific institutions.

160 min read
The Failures of Eld Science
The Dilemma: Science or Bayes?
Science Doesn't Trust Your Rationality
When Science Can't Help
Science Isn't Strict Enough
Do Scientists Already Know This Stuff?
No Safe Defense, Not Even Science
Changing the Definition of Science
Faster Than Science
Einstein's Speed
That Alien Message
My Childhood Role Model
Einstein's Superpowers
Class Project
Interlude
A Technical Explanation of Technical Explanation

Mere Goodness

0 / 36 posts readlog in to track progress

What makes something valuable—morally, or aesthetically, or prudentially? These three sequences ask how we can justify, revise, and naturalize our values and desires. The aim will be to find a way to understand our goals without compromising our efforts to actually achieve them. Here the biggest challenge is knowing when to trust your messy, complicated case-by-case impulses about what’s right and wrong, and when to replace them with simple exceptionless principles.

Fake Preferences

On failed attempts at theories of value.

35 min read
Ends: An Introduction
Not for the Sake of Happiness (Alone)
Fake Selfishness
Fake Morality
Fake Utility Functions
Detached Lever Fallacy
Dreams of AI Design
The Design Space of Minds-In-General
Value Theory

On obstacles to developing a new theory, and some intuitively desirable features of such a theory.

106 min read
Where Recursive Justification Hits Bottom
My Kind of Reflection
No Universally Compelling Arguments
Created Already In Motion
Sorting Pebbles Into Correct Heaps
2-Place and 1-Place Words
What Would You Do Without Morality?
Changing Your Metaethics
Could Anything Be Right?
Morality as Fixed Computation
Magical Categories
The True Prisoner's Dilemma
Sympathetic Minds
High Challenge
Serious Stories
Value is Fragile
The Gift We Give To Tomorrow
Quantified Humanism

On the tricky question of how we should apply such theories to our ordinary moral intuitions and decision-making.

79 min read
One Life Against the World
The Allais Paradox
Zut Allais!
Feeling Moral
The "Intuitions" Behind "Utilitarianism"
Ends Don't Justify Means (Among Humans)
Ethical Injunctions
Something to Protect
When (Not) To Use Probabilities
Newcomb's Problem and Regret of Rationality
Interlude
Twelve Virtues of Rationality

Becoming Stronger

How can individuals and communities put all this into practice? These three sequences begin with an autobiographical account of Yudkowsky’s own biggest philosophical blunders, with advice on how he thinks others might do better. The book closes with recommendations for developing evidence-based applied rationality curricula, and for forming groups and institutions to support interested students, educators, researchers, and friends.

Yudkowsky's Coming of Age

This sequence provides a last in-depth illustration of the dynamics of irrational belief, this time spotlighting the author’s own intellectual history.

85 min read
Beginnings: An Introduction
My Childhood Death Spiral
My Best and Worst Mistake
Raised in Technophilia
A Prodigy of Refutation
The Sheer Folly of Callow Youth
That Tiny Note of Discord
Fighting a Rearguard Action Against the Truth
My Naturalistic Awakening
The Level Above Mine
The Magnitude of His Own Folly
Beyond the Reach of God
My Bayesian Enlightenment
Challenging the Difficult

This sequences asks what it takes to solve a truly difficult problem—including demands that go beyond epistemic rationality.

45 min read
Trying to Try
Use the Try Harder, Luke
On Doing the Impossible
Make an Extraordinary Effort
Shut up and do the impossible!
Final Words
The Craft and the Community

Discusses rationality groups and group rationality, raising the questions:

  • Can rationality be learned and taught?
  • If so, how much improvement is possible?

How can we be confident we're seeing a real effect in a rationality intervention, and picking out the right cause?

  • What community norms would make this process of bettering ourselves easier?
  • Can we effectively collaborate on large-scale problems without sacrificing our freedom of thought and conduct?

Above all: What’s missing? What should be in the next generation of rationality primers—the ones that replace this text, improve on its style, test its prescriptions, supplement its content, and branch out in altogether new directions?

114 min read
Raising the Sanity Waterline
A Sense That More Is Possible
Epistemic Viciousness
Schools Proliferating Without Evidence
3 Levels of Rationality Verification
Why Our Kind Can't Cooperate
Tolerate Tolerance
Your Price for Joining
Can Humanism Match Religion's Output?
Church vs. Taskforce
Rationality: Common Interest of Many Causes
Helpless Individuals
Money: The Unit of Caring
Purchase Fuzzies and Utilons Separately
Bystander Apathy
Collective Apathy and the Internet
Incremental Progress and the Valley
Bayesians vs. Barbarians
Beware of Other-Optimizing
Practical Advice Backed By Deep Theories
The Sin of Underconfidence
Go Forth and Create the Art!