LESSWRONG
LW

Books of LessWrong
A Map That Reflects the Territory (2018)
Epistemology
Agency
Coordination
Curiosity
Alignment
The Engines of Cognition (2019)
Trust
Modularity
Incentives
Failure
The Carving of Reality (2020)
Reality & Reason
Coordination & Constraint
Alignment & Agency
Takeoff & Takeover
A Moderate Update to your Priors (2021)
A Moderate Update to your Organic Priors
A Moderate Update to your Artificial Priors

Books of LessWrong

Each December since 2018, the LessWrong community comes together to look at the best posts from the previous year, and reflect on which of them have stood the test of time. We hold 2-month-long annual review process. At the end, the thousands of posts from a given year are narrowed down into approximately 50 posts.

Those posts are collected in these Best Of LessWrong sequences. If you'd like to get a sense of what intellectual work LessWrong has done in the past few years, this is a good place to start.


A Map That Reflects the Territory
(Best of 2018)

Buy on Amazon

The Engines of Cognition
Best of 2019

Buy on Amazon

      

The Carving of Reality
Best of 2020

Buy on Amazon

 

Start Reading

A Map That Reflects the Territory

Best of 2018
0 / 42 posts readlog in to track progress
Epistemology

This first book is about epistemology, the study of how we come to know the world.

A healthy epistemology is not only about avoiding sins like self-deception and obfuscation. It is also about living out virtues like curiosity and empiricism, and taking active steps to dissolve confusion and see the world more clearly over time.

We do not understand how many parts of reality work. We also have many motives to not look directly at it or...

132 min read
A Sketch of Good Communication
Babble
Local Validity as a Key to Sanity and Civilization
The Loudest Alarm Is Probably False
Varieties Of Argumentative Experience
More Babble
Naming the Nameless
Toolbox-thinking and Law-thinking
Prune
Towards a New Impact Measure
Agency

A mind must be created already in motion.
There is no argument so compelling
that it will give dynamics to a static thing.
There is no computer program so persuasive
that you can run it on a rock.

 

This second book is about agency, the ability to take action in the world and control the future.

There’s something very strange about being human. We do not have a single simple goal, but rather a weird amalgam of desires and wants that...

74 min read
Being a Robust Agent
Noticing the Taste of Lotus
The Tails Coming Apart As Metaphor For Life
Meta-Honesty: Firming Up Honesty Around Its Edge-Cases
My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms
Coordination

"Indeed. Moving from bad equilibria to better equilibria is the whole point of having a civilization in the first place."

Coordination is the ability of multiple agents to work together.

This is the difficult question of how to work with other agents in the world, and how to have healthy reasoning and decision-making processes within a group.

If many agents in a group are working on different projects, how do we decide on resource allocation between those projects? If...

116 min read
Anti-social Punishment
The Costly Coordination Mechanism of Common Knowledge
The Intelligent Social Web
Prediction Markets: When Do They Work?
Spaghetti Towers
On the Loss and Preservation of Knowledge
A voting theory primer for rationalists
The Pavlov Strategy
Inadequate Equilibria vs. Governance of the Commons
Curiosity

"A burning itch to know is higher than a solemn vow to pursue truth."

This fourth book is about curiosity, which is the desire to understand how some corner of the world works. 

Curiosity is diving into Wikipedia. It’s running a survey to get data from your friends. It’s dropping balls from different heights and measuring how long they take to fall. Empiricism, scholarship, googling, introspection, data-gathering, science: all these are examples of the curious impulse in...

51 min read
Is Science Slowing Down?
Research: Rescuers during the Holocaust
An Untrollable Mathematician Illustrated
Why did everything take so long?
Is Clickbait Destroying Our General Intelligence?
What makes people intellectually active?
Open question: are minimal circuits daemon-free?
Beyond Astronomical Waste
Historical mathematicians exhibit a birth order effect too
Birth order effect found in Nobel Laureates in Physics
Alignment

This fifth book is about alignment, the problem of aligning the thoughts and goals of artificial intelligences with those of humans.

This relates to the art of human rationality in two ways.

First, the laws of reasoning and decision-making apply similarly to all agents, whether they are human or artificial. Many of the deepest insights about rationality at LessWrong have come directly from users’ attempts to grapple with the problem of understanding and aligning artificial intelligences.

Second, the...

84 min read
Arguments about fast takeoff
Specification gaming examples in AI
The Rocket Alignment Problem
Embedded Agents
Paul's research agenda FAQ
Challenges to Christiano’s capability amplification proposal
Robustness to Scale
Coherence arguments do not entail goal-directed behavior

The Engines of Cognition

Best of 2019
0 / 58 posts readlog in to track progress
Trust

The first book is about trust, the belief in something in the absence of understanding.

The Royal Society was founded in 1660, with the motto Nullius in Verba, "On No One's Word". The promise of science was that it would be a method for discovering truth without needing to trust anyone: because experiments can and would be replicated, incorrect theories and incorrect information would be !ltered out. By exploring systems that could be understood in full,...

196 min read
Rule Thinkers In, Not Out
Gears vs Behavior
Book Review: The Secret Of Our Success
Reason isn't magic
"Other people are wrong" vs "I am right"
In My Culture
Chris Olah’s views on AGI safety
Understanding “Deep Double Descent”
How to Ignore Your Emotions (while also thinking you're awesome at emotions)
Paper-Reading for Gears
Book summary: Unlocking the Emotional Brain
Noticing Frame Differences
Propagating Facts into Aesthetics
Do you fear the rock or the hard place?
Mental Mountains
Steelmanning Divination
Modularity

The second book is about modularity. Well designed or evolved structures are often not just made of parts, but made of parts with simple interfaces. These interfaces allow the parts to be reused in alternative contexts, and thus recombined in different ways.

One of the most important benefits is inspection and debugging. Modular systems are easier to understand, because their components are self-contained. However, this can also be a curse; when components have boundaries that are...

182 min read
Book Review: Design Principles of Biological Circuits
Reframing Superintelligence: Comprehensive AI Services as General Intelligence
Building up to an Internal Family Systems model
Being the (Pareto) Best in the World
The Schelling Choice is "Rabbit", not "Stag"
Literature Review: Distributed Teams
Gears-Level Models are Capital Investments
Evolution of Modularity
You Get About Five Words
Coherent decisions imply consistent utilities
Alignment Research Field Guide
Forum participation as a research strategy
The Credit Assignment Problem
Selection vs Control
Incentives

The third book is about incentives, which are the patterns of what is rewarded and what is punished.

The dynamics of what is incentivized and disincentivized in a group is a primary force in determining how that group behaves. When those incentives are aligned with what the individuals actually want, the output can be far greater than the sum of the parts; when they are misaligned with what the individuals want, the results can be far...

145 min read
Asymmetric Justice
Unconscious Economics
Power Buys You Distance From The Crime
Seeking Power is Often Convergently Instrumental in MDPs
Yes Requires the Possibility of No
Mistakes with Conservation of Expected Evidence
Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists
Excerpts from a larger discussion about simulacra
Moloch Hasn’t Won
Integrity and accountability are core parts of rationality
The Real Rules Have No Exceptions
Simple Rules of Law
The Amish, and Strategic Norms around Technology
Risks from Learned Optimization: Introduction
Gradient hacking
Failure

The fourth book is about failure. It’s what happens when a system behaves differently from how we expected it to, with adverse consequences for those who were relying on the success of that system. Failure is often as much about misunderstanding how a system works, as it is about the lack of effort or plan to bring the system into a successful configuration.

Almost nothing humans build ever works on the first try. Failure is part...

177 min read
The Parable of Predict-O-Matic
Blackmail
Bioinfohazards
What failure looks like
AI Safety "Success Stories"
Reframing Impact
The strategy-stealing assumption
Is Rationalist Self-Improvement Real?
The Curse Of The Counterfactual
human psycholinguists: a critical appraisal
[Answer] Why wasn't science invented in China?
Make more land
Rest Days vs Recovery Days

The Carving of Reality

Best of 2020
0 / 49 posts readlog in to track progress
Reality & Reason

Reality is that which doesn't go away when you stop believing it. Reason is the process by which we get our beliefs to more accurately reflect reality. 

This collection of essays explores the relationship between the two.

196 min read
The First Sample Gives the Most Information
What Money Cannot Buy
Reality-Revealing and Reality-Masking Puzzles
To listen well, get curious
The Felt Sense: What, Why and How
The Solomonoff Prior is Malign
The Treacherous Path to Rationality
Anti-Aging: State of the Art
Search versus design
Radical Probabilism
Introduction To The Infra-Bayesianism Sequence
When Money Is Abundant, Knowledge Is The Real Wealth
Coordination & Constraint
166 min read
Coordination as a Scarce Resource
Transportation as a Constraint
Interfaces as a Scarce Resource
microCOVID.org: A tool to estimate COVID risk from common activities
Seeing the Smoke
Pain is not the unit of Effort
Studies On Slack
Is Success the Enemy of Freedom? (Full)
Simulacra Levels and their Interactions
Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems
Swiss Political System: More than You ever Wanted to Know (I.)
"Can you keep this confidential? How do you know?"
Motive Ambiguity
Can crimes be discussed literally?
Why haven't we celebrated any major achievements lately?
Alignment & Agency
203 min read
An Orthodox Case Against Utility Functions
The Pointers Problem: Human Values Are A Function Of Humans' Latent Variables
Alignment By Default
An overview of 11 proposals for building safe advanced AI
The ground of optimization
Search versus design
Inner Alignment: Explain like I'm 12 Edition
Inaccessible information
AGI safety from first principles: Introduction
Is Success the Enemy of Freedom? (Full)
Takeoff & Takeover

Imagine an advanced society, with complex structures more intricate and intelligent than anything that exists today – a society which nevertheless lacks any being that is conscious, or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland with no children.

– Nick Bostrom

315 min read
Draft report on AI timelines
The date of AI Takeover is not the day the AI takes over
An overview of 11 proposals for building safe advanced AI
Cortés, Pizarro, and Afonso as Precedents for Takeover
Nuclear war is unlikely to cause human extinction
Biology-Inspired AGI Timelines: The Trick That Never Works
Reply to Eliezer on Biological Anchors
Against GDP as a metric for timelines and takeoff speeds
Some AI research areas and their relevance to existential safety
My computational framework for the brain
How uniform is the neocortex?
Discontinuous progress in history: an update

A Moderate Update to your Priors

Best of 2021
0 / 45 posts readlog in to track progress
A Moderate Update to your Organic Priors
399 min read
Strong Evidence is Common
“PR” is corrosive; “reputation” is not.
Your Cheerful Price
This Can't Go On
Rationalism before the Sequences
Lies, Damn Lies, and Fabricated Options
How To Write Quickly While Maintaining Epistemic Rigor
Science in a High-Dimensional World
How factories were made safe
Cryonics signup guide #1: Overview
Making Vaccine
Taboo "Outside View"
All Possible Views About Humanity's Future Are Wild
Split and Commit
There’s no such thing as a tree (phylogenetically)
Trapped Priors As A Basic Problem Of Rationality
Slack Has Positive Externalities For Groups
The Rationalists of the 1950s (and before) also called themselves “Rationalists”
Ruling Out Everything Else
Leaky Delegation: You are not a Commodity
Feature Selection
Cup-Stacking Skills (or, Reflexive Involuntary Mental Motions)
Self-Integrity and the Drowning Child
Working With Monsters
Simulacrum 3 As Stag-Hunt Strategy
Elephant seal 2
Lars Doucet's Georgism series on Astral Codex Ten
Catching the Spark
Shoulder Advisors 101
Notes from "Don't Shoot the Dog"
Why has nuclear power been a flop?
A Moderate Update to your Artificial Priors
395 min read
ARC's first technical report: Eliciting Latent Knowledge
Fun with +12 OOMs of Compute
What 2026 looks like
Ngo and Yudkowsky on alignment difficulty
Another (outer) alignment failure story
What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)
The Plan
Finite Factored Sets
Selection Theorems: A Program For Understanding Agents
My research methodology
larger language models may disappoint you [or, an eternally unfinished draft]
Comments on Carlsmith's “Is power-seeking AI an existential risk?”
EfficientZero: How It Works
Specializing in Problems We Don't Understand