LESSWRONG
The Best of LessWrong
LW

1434

The Best of LessWrong

When posts turn more than a year old, the LessWrong community reviews and votes on how well they have stood the test of time. These are the posts that have ranked the highest for all years since 2018 (when our annual tradition of choosing the least wrong of LessWrong began).

For the years 2018, 2019 and 2020 we also published physical books with the results of our annual vote, which you can buy and learn more about here.
+

Rationality

Eliezer Yudkowsky
Local Validity as a Key to Sanity and Civilization
Buck
"Other people are wrong" vs "I am right"
Mark Xu
Strong Evidence is Common
TsviBT
Please don't throw your mind away
Raemon
Noticing Frame Differences
johnswentworth
You Are Not Measuring What You Think You Are Measuring
johnswentworth
Gears-Level Models are Capital Investments
Hazard
How to Ignore Your Emotions (while also thinking you're awesome at emotions)
Scott Garrabrant
Yes Requires the Possibility of No
Ben Pace
A Sketch of Good Communication
Eliezer Yudkowsky
Meta-Honesty: Firming Up Honesty Around Its Edge-Cases
Duncan Sabien (Inactive)
Lies, Damn Lies, and Fabricated Options
Scott Alexander
Trapped Priors As A Basic Problem Of Rationality
Duncan Sabien (Inactive)
Split and Commit
Duncan Sabien (Inactive)
CFAR Participant Handbook now available to all
johnswentworth
What Are You Tracking In Your Head?
Mark Xu
The First Sample Gives the Most Information
Duncan Sabien (Inactive)
Shoulder Advisors 101
Scott Alexander
Varieties Of Argumentative Experience
Eliezer Yudkowsky
Toolbox-thinking and Law-thinking
alkjash
Babble
Zack_M_Davis
Feature Selection
abramdemski
Mistakes with Conservation of Expected Evidence
Kaj_Sotala
The Felt Sense: What, Why and How
Duncan Sabien (Inactive)
Cup-Stacking Skills (or, Reflexive Involuntary Mental Motions)
Ben Pace
The Costly Coordination Mechanism of Common Knowledge
Jacob Falkovich
Seeing the Smoke
Duncan Sabien (Inactive)
Basics of Rationalist Discourse
alkjash
Prune
johnswentworth
Gears vs Behavior
Elizabeth
Epistemic Legibility
Daniel Kokotajlo
Taboo "Outside View"
Duncan Sabien (Inactive)
Sazen
AnnaSalamon
Reality-Revealing and Reality-Masking Puzzles
Eliezer Yudkowsky
ProjectLawful.com: Eliezer's latest story, past 1M words
Eliezer Yudkowsky
Self-Integrity and the Drowning Child
Jacob Falkovich
The Treacherous Path to Rationality
Scott Garrabrant
Tyranny of the Epistemic Majority
alkjash
More Babble
abramdemski
Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems
Raemon
Being a Robust Agent
Zack_M_Davis
Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists
Benquo
Reason isn't magic
habryka
Integrity and accountability are core parts of rationality
Raemon
The Schelling Choice is "Rabbit", not "Stag"
Diffractor
Threat-Resistant Bargaining Megapost: Introducing the ROSE Value
Raemon
Propagating Facts into Aesthetics
johnswentworth
Simulacrum 3 As Stag-Hunt Strategy
LoganStrohl
Catching the Spark
Jacob Falkovich
Is Rationalist Self-Improvement Real?
Benquo
Excerpts from a larger discussion about simulacra
Zvi
Simulacra Levels and their Interactions
abramdemski
Radical Probabilism
sarahconstantin
Naming the Nameless
AnnaSalamon
Comment reply: my low-quality thoughts on why CFAR didn't get farther with a "real/efficacious art of rationality"
Eric Raymond
Rationalism before the Sequences
Owain_Evans
The Rationalists of the 1950s (and before) also called themselves “Rationalists”
Raemon
Feedbackloop-first Rationality
LoganStrohl
Fucking Goddamn Basics of Rationalist Discourse
Raemon
Tuning your Cognitive Strategies
johnswentworth
Lessons On How To Get Things Right On The First Try
+

Optimization

So8res
Focus on the places where you feel shocked everyone's dropping the ball
Jameson Quinn
A voting theory primer for rationalists
sarahconstantin
The Pavlov Strategy
Zvi
Prediction Markets: When Do They Work?
johnswentworth
Being the (Pareto) Best in the World
alkjash
Is Success the Enemy of Freedom? (Full)
johnswentworth
Coordination as a Scarce Resource
AnnaSalamon
What should you change in response to an "emergency"? And AI risk
jasoncrawford
How factories were made safe
HoldenKarnofsky
All Possible Views About Humanity's Future Are Wild
jasoncrawford
Why has nuclear power been a flop?
Zvi
Simple Rules of Law
Scott Alexander
The Tails Coming Apart As Metaphor For Life
Zvi
Asymmetric Justice
Jeffrey Ladish
Nuclear war is unlikely to cause human extinction
Elizabeth
Power Buys You Distance From The Crime
Eliezer Yudkowsky
Is Clickbait Destroying Our General Intelligence?
Spiracular
Bioinfohazards
Zvi
Moloch Hasn’t Won
Zvi
Motive Ambiguity
Benquo
Can crimes be discussed literally?
johnswentworth
When Money Is Abundant, Knowledge Is The Real Wealth
GeneSmith
Significantly Enhancing Adult Intelligence With Gene Editing May Be Possible
HoldenKarnofsky
This Can't Go On
Said Achmiz
The Real Rules Have No Exceptions
Lars Doucet
Lars Doucet's Georgism series on Astral Codex Ten
johnswentworth
Working With Monsters
jasoncrawford
Why haven't we celebrated any major achievements lately?
abramdemski
The Credit Assignment Problem
Martin Sustrik
Inadequate Equilibria vs. Governance of the Commons
Scott Alexander
Studies On Slack
KatjaGrace
Discontinuous progress in history: an update
Scott Alexander
Rule Thinkers In, Not Out
Raemon
The Amish, and Strategic Norms around Technology
Zvi
Blackmail
HoldenKarnofsky
Nonprofit Boards are Weird
Wei Dai
Beyond Astronomical Waste
johnswentworth
Making Vaccine
jefftk
Make more land
jenn
Things I Learned by Spending Five Thousand Hours In Non-EA Charities
Richard_Ngo
The ants and the grasshopper
So8res
Enemies vs Malefactors
Elizabeth
Change my mind: Veganism entails trade-offs, and health is one of the axes
+

World

Kaj_Sotala
Book summary: Unlocking the Emotional Brain
Ben
The Redaction Machine
Samo Burja
On the Loss and Preservation of Knowledge
Alex_Altair
Introduction to abstract entropy
Martin Sustrik
Swiss Political System: More than You ever Wanted to Know (I.)
johnswentworth
Interfaces as a Scarce Resource
eukaryote
There’s no such thing as a tree (phylogenetically)
Scott Alexander
Is Science Slowing Down?
Martin Sustrik
Anti-social Punishment
johnswentworth
Transportation as a Constraint
Martin Sustrik
Research: Rescuers during the Holocaust
GeneSmith
Toni Kurz and the Insanity of Climbing Mountains
johnswentworth
Book Review: Design Principles of Biological Circuits
Elizabeth
Literature Review: Distributed Teams
Valentine
The Intelligent Social Web
eukaryote
Spaghetti Towers
Eli Tyre
Historical mathematicians exhibit a birth order effect too
johnswentworth
What Money Cannot Buy
Bird Concept
Unconscious Economics
Scott Alexander
Book Review: The Secret Of Our Success
johnswentworth
Specializing in Problems We Don't Understand
KatjaGrace
Why did everything take so long?
Ruby
[Answer] Why wasn't science invented in China?
Scott Alexander
Mental Mountains
L Rudolf L
A Disneyland Without Children
johnswentworth
Evolution of Modularity
johnswentworth
Science in a High-Dimensional World
Kaj_Sotala
My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms
Kaj_Sotala
Building up to an Internal Family Systems model
Steven Byrnes
My computational framework for the brain
Natália
Counter-theses on Sleep
abramdemski
What makes people intellectually active?
Bucky
Birth order effect found in Nobel Laureates in Physics
zhukeepa
How uniform is the neocortex?
JackH
Anti-Aging: State of the Art
Vaniver
Steelmanning Divination
KatjaGrace
Elephant seal 2
Zvi
Book Review: Going Infinite
Rafael Harth
Why it's so hard to talk about Consciousness
Duncan Sabien (Inactive)
Social Dark Matter
Eric Neyman
How much do you believe your results?
Malmesbury
The Talk: a brief explanation of sexual dimorphism
moridinamael
The Parable of the King and the Random Process
Henrik Karlsson
Cultivating a state of mind where new ideas are born
+

Practical

alkjash
Pain is not the unit of Effort
benkuhn
Staring into the abyss as a core life skill
Unreal
Rest Days vs Recovery Days
Duncan Sabien (Inactive)
In My Culture
juliawise
Notes from "Don't Shoot the Dog"
Elizabeth
Luck based medicine: my resentful story of becoming a medical miracle
johnswentworth
How To Write Quickly While Maintaining Epistemic Rigor
Duncan Sabien (Inactive)
Ruling Out Everything Else
johnswentworth
Paper-Reading for Gears
Elizabeth
Butterfly Ideas
Eliezer Yudkowsky
Your Cheerful Price
benkuhn
To listen well, get curious
Wei Dai
Forum participation as a research strategy
HoldenKarnofsky
Useful Vices for Wicked Problems
pjeby
The Curse Of The Counterfactual
Darmani
Leaky Delegation: You are not a Commodity
Adam Zerner
Losing the root for the tree
chanamessinger
The Onion Test for Personal and Institutional Honesty
Raemon
You Get About Five Words
HoldenKarnofsky
Learning By Writing
GeneSmith
How to have Polygenically Screened Children
AnnaSalamon
“PR” is corrosive; “reputation” is not.
Ruby
Do you fear the rock or the hard place?
johnswentworth
Slack Has Positive Externalities For Groups
Raemon
Limerence Messes Up Your Rationality Real Bad, Yo
mingyuan
Cryonics signup guide #1: Overview
catherio
microCOVID.org: A tool to estimate COVID risk from common activities
Valentine
Noticing the Taste of Lotus
orthonormal
The Loudest Alarm Is Probably False
Raemon
"Can you keep this confidential? How do you know?"
mingyuan
Guide to rationalist interior decorating
Screwtape
Loudly Give Up, Don't Quietly Fade
+

AI Strategy

paulfchristiano
Arguments about fast takeoff
Eliezer Yudkowsky
Six Dimensions of Operational Adequacy in AGI Projects
Ajeya Cotra
Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
paulfchristiano
What failure looks like
Daniel Kokotajlo
What 2026 looks like
gwern
It Looks Like You're Trying To Take Over The World
Daniel Kokotajlo
Cortés, Pizarro, and Afonso as Precedents for Takeover
Daniel Kokotajlo
The date of AI Takeover is not the day the AI takes over
Andrew_Critch
What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)
paulfchristiano
Another (outer) alignment failure story
Ajeya Cotra
Draft report on AI timelines
Eliezer Yudkowsky
Biology-Inspired AGI Timelines: The Trick That Never Works
Daniel Kokotajlo
Fun with +12 OOMs of Compute
Wei Dai
AI Safety "Success Stories"
Eliezer Yudkowsky
Pausing AI Developments Isn't Enough. We Need to Shut it All Down
HoldenKarnofsky
Reply to Eliezer on Biological Anchors
Richard_Ngo
AGI safety from first principles: Introduction
johnswentworth
The Plan
Rohin Shah
Reframing Superintelligence: Comprehensive AI Services as General Intelligence
lc
What an actually pessimistic containment strategy looks like
Eliezer Yudkowsky
MIRI announces new "Death With Dignity" strategy
KatjaGrace
Counterarguments to the basic AI x-risk case
Adam Scholl
Safetywashing
habryka
AI Timelines
evhub
Chris Olah’s views on AGI safety
So8res
Comments on Carlsmith's “Is power-seeking AI an existential risk?”
nostalgebraist
human psycholinguists: a critical appraisal
nostalgebraist
larger language models may disappoint you [or, an eternally unfinished draft]
Orpheus16
Speaking to Congressional staffers about AI risk
Tom Davidson
What a compute-centric framework says about AI takeoff speeds
abramdemski
The Parable of Predict-O-Matic
KatjaGrace
Let’s think about slowing down AI
Daniel Kokotajlo
Against GDP as a metric for timelines and takeoff speeds
Joe Carlsmith
Predictable updating about AI risk
Raemon
"Carefully Bootstrapped Alignment" is organizationally hard
KatjaGrace
We don’t trade with ants
+

Technical AI Safety

paulfchristiano
Where I agree and disagree with Eliezer
Eliezer Yudkowsky
Ngo and Yudkowsky on alignment difficulty
Andrew_Critch
Some AI research areas and their relevance to existential safety
1a3orn
EfficientZero: How It Works
elspood
Security Mindset: Lessons from 20+ years of Software Security Failures Relevant to AGI Alignment
So8res
Decision theory does not imply that we get to have nice things
Vika
Specification gaming examples in AI
Rafael Harth
Inner Alignment: Explain like I'm 12 Edition
evhub
An overview of 11 proposals for building safe advanced AI
TurnTrout
Reward is not the optimization target
johnswentworth
Worlds Where Iterative Design Fails
johnswentworth
Alignment By Default
johnswentworth
How To Go From Interpretability To Alignment: Just Retarget The Search
Alex Flint
Search versus design
abramdemski
Selection vs Control
Buck
AI Control: Improving Safety Despite Intentional Subversion
Eliezer Yudkowsky
The Rocket Alignment Problem
Eliezer Yudkowsky
AGI Ruin: A List of Lethalities
Mark Xu
The Solomonoff Prior is Malign
paulfchristiano
My research methodology
TurnTrout
Reframing Impact
Scott Garrabrant
Robustness to Scale
paulfchristiano
Inaccessible information
TurnTrout
Seeking Power is Often Convergently Instrumental in MDPs
So8res
A central AI alignment problem: capabilities generalization, and the sharp left turn
evhub
Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research
paulfchristiano
The strategy-stealing assumption
So8res
On how various plans miss the hard bits of the alignment challenge
abramdemski
Alignment Research Field Guide
johnswentworth
The Pointers Problem: Human Values Are A Function Of Humans' Latent Variables
Buck
Language models seem to be much better than humans at next-token prediction
abramdemski
An Untrollable Mathematician Illustrated
abramdemski
An Orthodox Case Against Utility Functions
Veedrac
Optimality is the tiger, and agents are its teeth
Sam Ringer
Models Don't "Get Reward"
Alex Flint
The ground of optimization
johnswentworth
Selection Theorems: A Program For Understanding Agents
Rohin Shah
Coherence arguments do not entail goal-directed behavior
abramdemski
Embedded Agents
evhub
Risks from Learned Optimization: Introduction
nostalgebraist
chinchilla's wild implications
johnswentworth
Why Agent Foundations? An Overly Abstract Explanation
zhukeepa
Paul's research agenda FAQ
Eliezer Yudkowsky
Coherent decisions imply consistent utilities
paulfchristiano
Open question: are minimal circuits daemon-free?
evhub
Gradient hacking
janus
Simulators
LawrenceC
Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research]
TurnTrout
Humans provide an untapped wealth of evidence about alignment
Neel Nanda
A Mechanistic Interpretability Analysis of Grokking
Collin
How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme
evhub
Understanding “Deep Double Descent”
Quintin Pope
The shard theory of human values
TurnTrout
Inner and outer alignment decompose one hard problem into two extremely hard problems
Eliezer Yudkowsky
Challenges to Christiano’s capability amplification proposal
Scott Garrabrant
Finite Factored Sets
paulfchristiano
ARC's first technical report: Eliciting Latent Knowledge
Diffractor
Introduction To The Infra-Bayesianism Sequence
TurnTrout
Towards a New Impact Measure
LawrenceC
Natural Abstractions: Key Claims, Theorems, and Critiques
Zack_M_Davis
Alignment Implications of LLM Successes: a Debate in One Act
johnswentworth
Natural Latents: The Math
TurnTrout
Steering GPT-2-XL by adding an activation vector
Jessica Rumbelow
SolidGoldMagikarp (plus, prompt generation)
So8res
Deep Deceptiveness
Charbel-Raphaël
Davidad's Bold Plan for Alignment: An In-Depth Explanation
Charbel-Raphaël
Against Almost Every Theory of Impact of Interpretability
Joe Carlsmith
New report: "Scheming AIs: Will AIs fake alignment during training in order to get power?"
Eliezer Yudkowsky
GPTs are Predictors, not Imitators
peterbarnett
Labs should be explicit about why they are building AGI
HoldenKarnofsky
Discussion with Nate Soares on a key alignment difficulty
Jesse Hoogland
Neural networks generalize because of this one weird trick
paulfchristiano
My views on “doom”
technicalities
Shallow review of live agendas in alignment & safety
Vanessa Kosoy
The Learning-Theoretic Agenda: Status 2023
ryan_greenblatt
Improving the Welfare of AIs: A Nearcasted Proposal
201820192020202120222023All
RationalityWorldOptimizationAI StrategyTechnical AI SafetyPracticalAll
#2
Focus on the places where you feel shocked everyone's dropping the ball

If you're looking for ways to help with the whole “the world looks pretty doomed” business, here's my advice: look around for places where we're all being total idiots. Look around for places where something seems incompetently run, or hopelessly inept, and where some part of you thinks you can do better.

Then do it better.

#3
When Money Is Abundant, Knowledge Is The Real Wealth

As resources become abundant, the bottleneck shifts to other resources. Power or money are no longer the limiting factors past a certain point; knowledge becomes the bottleneck. Knowledge can't be reliably bought, and acquiring it is difficult. Therefore, investments in knowledge (e.g. understanding systems at a gears-level) become the most valuable investments.

by johnswentworth
#3
Significantly Enhancing Adult Intelligence With Gene Editing May Be Possible

The author argues that it may be possible to significantly enhance adult intelligence through gene editing. They discuss potential delivery methods, editing techniques, and challenges. While acknowledging uncertainties, they believe this could have a major impact on human capabilities and potentially help with AI alignment. They propose starting with cell culture experiments and animal studies.

by GeneSmith
#4
Being the (Pareto) Best in the World

John Wentworth argues that becoming one of the best in the world at *one* specific skill is hard, but it's not as hard to become the best in the world at the *combination* of two (or more) different skills. He calls this being "Pareto best" and argues it can circumvent the generalized efficient markets principle. 

by johnswentworth
#5
This Can't Go On

We're used to the economy growing a few percent per year. But this is a very unusual situation. Zooming out to all of history, we see that growth has been accelerating, that it's near its historical high point, and that it's faster than it can be for all that much longer. There aren't enough atoms in the galaxy to sustain this rate of growth for even another 10,000 years!

What comes next – stagnation, explosion, or collapse?

by HoldenKarnofsky
#6
Rule Thinkers In, Not Out

"Some of the people who have most inspired me have been inexcusably wrong on basic issues. But you only need one world-changing revelation to be worth reading."

Scott argues that our interest in thinkers should not be determined by their worst idea, or even their average idea, but by their best ideas. Some of the best thinkers in history believed ludicrous things, like Newton believing in Bible codes.

by Scott Alexander
#7
The Tails Coming Apart As Metaphor For Life

The "tails coming apart" is a phenomenon where two variables can be highly correlated overall, but at extreme values they diverge. Scott Alexander explores how this applies to complex concepts like happiness and morality, where our intuitions work well for common situations but break down in extreme scenarios. 

by Scott Alexander
#8
Asymmetric Justice

According to Zvi, people have a warped sense of justice. For any harm you cause, regardless of intention and or motive,  you earn "negative points" that merit punishment. At least implicitly, however, people only want to reward good outcomes a person causes only if their sole goal was being altruistic. Curing illness to make profit? No "positive points" for you!

by Zvi
#10
Things I Learned by Spending Five Thousand Hours In Non-EA Charities

Jenn spent 5000 hours working at non-EA charities, and learned a number of things that may not be obvious to effective altruists, when working with more mature organizations in more mature ecosystems.

by jenn
#13
Prediction Markets: When Do They Work?

Prediction markets are a potential way to harness wisdom of crowds and incentivize truth-seeking. But they're tricky to set up correctly. Zvi Mowshowitz, who has extensive experience with prediction markets and sports betting, explains the key factors that make prediction markets succeed or fail.

by Zvi
#13
How factories were made safe

Back in the early days of factories, workplace injury rates were enormous. Over time, safety engineering took hold, various legal reforms were passed (most notably liability law), and those rates dramatically dropped. This is the story of how factories went from death traps to relatively safe. 

by jasoncrawford
#14
Coordination as a Scarce Resource

Many of the most profitable jobs and companies are primarily about solving coordination problems. This suggests "coordination problems" are an unusually tight bottleneck for productive economic activity. John explores implications of looking at the world through this lens. 

by johnswentworth
#15
Making Vaccine

John made his own COVID-19 vaccine at home using open source instructions. Here's how he did it and why.

by johnswentworth
#17
A voting theory primer for rationalists

Democratic processes are important loci of power. It's useful to understand the dynamics of the voting methods used real-world elections. My own ideas of ethics and of fun theory are deeply informed by my decades of interest in voting theory

by Jameson Quinn
#17
All Possible Views About Humanity's Future Are Wild

It's wild to think that humanity might expand throughout the galaxy in the next century or two. But it's also wild to think that we definitely won't. In fact, all views about humanity's long-term future are pretty wild when you think about it. We're in a wild situation!

by HoldenKarnofsky
#19
The ants and the grasshopper

One winter a grasshopper, starving and frail, approaches a colony of ants drying out their grain in the sun to ask for food, having spent the summer singing and dancing.

Then, various things happen.

by Richard_Ngo
#22
Nonprofit Boards are Weird

Nonprofit boards have great power, but low engagement, unclear responsibility, and no accountability. There's also a shortage of good guidance on how to be an effective board member. Holden gives recommendations on how to do it well, but the whole structure is inherently weird and challenging. 

by HoldenKarnofsky
#23
Moloch Hasn’t Won

Scott Alexander's "Meditations on Moloch" paints a gloomy picture of the world being inevitably consumed by destructive forces of competition and optimization. But Zvi argues this isn't actually how the world works - we've managed to resist and overcome these forces throughout history. 

by Zvi
#24
Is Success the Enemy of Freedom? (Full)

Success is supposed to open doors and broaden horizons. But often it can do the opposite - trapping people in narrow specialties or roles they've outgrown. This post explores how success can sometimes be the enemy of personal freedom and growth, and how to maintain flexibility as you become more successful.

by alkjash
#27
Inadequate Equilibria vs. Governance of the Commons

A book review examining Elinor Ostrom's "Governance of the Commons", in light of Eliezer Yudkowsky's "Inadequate Equilibria." Are successful local institutions for governing common pool resources possible without government intervention? Under what circumstances can such institutions emerge spontaneously to solve coordination problems?

by Martin Sustrik
#27
Make more land

Jeff argues that people should fill in some of the San Francisco Bay, south of the Dumbarton Bridge, to create new land for housing. This would allow millions of people to live closer to jobs, reducing sprawl and traffic. While there are environmental concerns, the benefits of dense urban housing outweigh the localized impacts. 

by jefftk
#28
Why haven't we celebrated any major achievements lately?

Crawford looks back on past celebrations of achievements like the US transcontinental railroad, the Brooklyn Bridge, electric lighting, the polio vaccine, and the Moon landing. He then asks: Why haven't we celebrated any major achievements lately? He explores some hypotheses for this change.

by jasoncrawford
#29
The Pavlov Strategy

You've probably heard about the "tit-for-tat" strategy in the iterated prisoner's dilemma. But have you heard of the Pavlov strategy? The simple strategy performs surprisingly well in certain conditions. Why don't we talk about Pavlov strategy as much as Tit-for-Tat strategy?

by sarahconstantin
#31
Bioinfohazards

A thoughtful exploration of the risks and benefits of sharing information about biosecurity and biological risks. The authors argue that while there are real risks to sharing sensitive information, there are also important benefits that need to be weighed carefully. They provide frameworks for thinking through these tradeoffs. 

by Spiracular
#32
Beyond Astronomical Waste

What if our universe's resources are just a drop in the bucket compared to what's out there? We might be able to influence or escape to much larger universes that are simulating us or can otherwise be controlled by us. This could be a source of vastly more potential value than just using the resources in our own universe. 

by Wei Dai
#32
Discontinuous progress in history: an update

AI Impacts investigated dozens of technological trends, looking for examples of discontinuous progress (where more than a century of progress happened at once). They found ten robust cases, such as the first nuclear weapons, and the Great Eastern steamship. 

They hope the data can inform expectations about discontinuities in AI development.

by KatjaGrace
#33
Enemies vs Malefactors

Harmful people often lack explicit malicious intent. It’s worth deploying your social or community defenses against them anyway. 

by So8res
#36
Is Clickbait Destroying Our General Intelligence?

It might be some elements of human intelligence (at least at the civilizational level) are culturally/memetically transmitted. All fine and good in theory. Except the social hypercompetition between people and intense selection pressure of ideas online might be eroding our world's intelligence. Eliezer wonders if he's only who he is because he grew up reading old science fiction from before the current era's memes.

by Eliezer Yudkowsky
#36
Motive Ambiguity

A counterintuitive concept: Sometimes people choose the worse option, to signal their loyalty or values in situations where that loyalty might be in question. Zvi explores this idea of "motive ambiguity" and how it can lead to perverse incentives. 

by Zvi
#36
Working With Monsters

A person wakes up from cryonic freeze in a post-apocalyptic future. A "scissor" statement – an AI-generated statement designed to provoke maximum controversy – has led to massive conflict and destruction. The survivors are those who managed to work with people they morally despise.

by johnswentworth
#38
The Amish, and Strategic Norms around Technology

The Amish relationship to technology is not "stick to technology from the 1800s", but rather "carefully think about how technology will affect your culture, and only include technology that does what you want." Raemon explores how these ideas could potentially be applied in other contexts.

by Raemon
#38
What should you change in response to an "emergency"? And AI risk

You might feel like AI risk is an "emergency" that demands drastic changes to your life. But is this actually the best way to respond? Anna Salamon explores what kinds of changes actually make sense in different types of emergencies, and what that might mean for how to approach existential risk.

by AnnaSalamon
#39
Power Buys You Distance From The Crime

Power allows people to benefit from immoral acts without having to take responsibility or even be aware of them. The most powerful person in a situation may not be the most morally culpable, as they can remain distant from the actual "crime". If you're not actively looking into how your wants are being met, you may be unknowingly benefiting from something unethical.

by Elizabeth
#39
Nuclear war is unlikely to cause human extinction

You've probably heard that a nuclear war between major powers would cause human extinction. This post argues that while nuclear war would be incredibly destructive, it's unlikely to actually cause human extinction. The main risks come from potential climate effects, but even in severe scenarios some human populations would likely survive.

by Jeffrey Ladish
#40
Can crimes be discussed literally?

All sorts of everyday practices in the legal system, medicine, software, and other areas of life involve stating things that aren't true. But calling these practices "lies" or "fraud" seems to be perceived as an attack rather than a straightforward description. This makes it difficult to discuss and analyze these practices without provoking emotional defensiveness. 

by Benquo
#40
Lars Doucet's Georgism series on Astral Codex Ten

An in-depth overview of Georgism, a school of political economy that advocates for a Land Value Tax (LVT), aiming to discourage land speculation and rent-seeking behavior; promote more efficient use of land, make housing more affordable, and taxes more efficient.

by Lars Doucet
#42
The Real Rules Have No Exceptions

Said argues that there's no such thing as a real exception to a rule. If you find an exception, this means you need to update the rule itself. The "real" rule is always the one that already takes into account all possible exceptions.

by Said Achmiz
#42
Studies On Slack

Under conditions of perfectly intense competition, evolution works like water flowing down a hill – it can never go up even the tiniest elevation. But if there is slack in the selection process, it's possible for evolution to escape local minima. "How much slack is optimal" is an interesting question, Scott explores in various contexts.

by Scott Alexander
#43
Change my mind: Veganism entails trade-offs, and health is one of the axes

Elizabeth argues that veganism comes with trade-offs, including potential health issues, that are often downplayed or denied by vegan advocates. She calls for more honesty about these challenges from the vegan community. 

by Elizabeth
#45
Blackmail

Smart people are failing to provide strong arguments for why blackmail should be illegal. Robin Hanson is explicitly arguing it should be legal. Zvi Mowshowitz argues this is wrong, and gives his perspective on why blackmail is bad.

by Zvi
#45
Why has nuclear power been a flop?

Nuclear power once seemed to be the energy of the future, but has failed to live up to that promise. Why? Jason Crawford summarizes Jack Devanney's book "Why Nuclear Power Has Been a Flop", which blames overregulation driven by unrealistic radiation safety models.

by jasoncrawford
#47
The Credit Assignment Problem

The credit assignment problem – the challenge of figuring out which parts of a complex system deserve credit for good or bad outcomes – shows up just about everywhere. Abram Demski describes how credit assignment appears in areas as diverse as AI, politics, economics, law, sociology, biology, ethics, and epistemology. 

by abramdemski
#50
Simple Rules of Law

Robin Hanson asked "Why do people like complex rules instead of simple rules?" and gave 12 examples.

Zvi responds with a detailed analysis of each example, suggesting that the desire for complex rules often stems from issues like Goodhart's Law, the Copenhagen Interpretation of Ethics, power dynamics, and the need to consider factors that can't be explicitly stated.

by Zvi
11Daniel Kokotajlo
This is one of those posts, like "pain is not the unit of effort," that combines a memorable and informative and very useful and important slogan with a bunch of argumentation and examples to back up that slogan. I think this type of post is great for the LW review. When I first read this post, I thought it was boring and unimportant: trivially, there will be some circumstances where knowledge is the bottleneck, because for pretty much all X there will be some circumstances where X is the bottleneck. However, since then I've ended up saying the slogan "when money is abundant, knowledge is the real wealth" probably about a dozen separate times when explaining my career decisions, arguing with others at CLR about what our strategy should be, and even when deliberating to myself about what to do next. I guess longtermist EAs right now do have a surplus of money and a shortage of knowledge (relative to how much knowledge is needed to solve the problems we are trying to solve...) so in retrospect it's not surprising that this slogan was practically applicable to my life so often. I do think there are ways the post could be expanded and improved. Come to think of it, I'll make a mini-comment right here to gesture at the stuff I would add to it if I could: 1. List of other ideas for how to invest in knowledge. For example, building a community with good epistemic norms. Or paying a bunch of people to collect data / info about various world developments and report on them to you. Or paying a bunch of people to write textbooks and summaries and explainer videos and make diagrams illustrating cutting-edge knowledge (yours and others'). 2. Arguments that in fact, right now, longtermist EAs and/or AI-risk-reducers are bottlenecked on knowledge (rather than money, or power/status) --My own experience doing cost-benefit analyses is that interventions/plans vary in EV by OOMs and that it's common to find new considerations or updated models that flip the sign entirely, or ad
36abramdemski
I really like this post. I think it points out an important problem with intuitive credit-assignment algorithms which people often use. The incentive toward inaction is a real problem which is often encountered in practice. While I was somewhat aware of the problem before, this post explains it well. I also think this post is wrong, in a significant way: asymmetric justice is not always a problem and is sometimes exactly what you want. in particular, it's how you want a justice system (in the sense of police, judges, etc) to work. The book Law's Order explains it like this: you don't want theft to be punished in keeping with its cost. Rather, in order for the free market to function, you want theft to be punished harshly enough that theft basically doesn't happen. Zvi speaks as if the purpose of the justice system is to reward positive externalities and punish negative externalities, to align everyone's incentives. While this is a noble goal, Law's Order sees it as a goal to be taken care of by other parts of society, in particular the free market. (Law's Order is a fairly libertarian book, so it puts a lot of faith in the free market.) The purpose of the justice system is to enforce the structure such that those other institutions can do their jobs. The free market can't optimize people's lives properly if theft and murder are a constant and contracts cannot be enforced. So, it makes perfect sense for a justice system to be asymmetric. Its role is to strongly disincentivize specific things, not to broadly provide compensatory incentives. (For this reason, scales are a pretty terrible symbol for justice.) In general, we might conclude that credit assignment systems need two parts: 1. A "symmetric" part, which attempts to allocate credit in as calibrated a way as it can, rewarding good work and punishing bad. 2. An "asymmetric" part, which harshly enforces the rules which ensure that the symmetric part can function, ensuring that those rules are followed fr
13Bird Concept
Elicit Prediction (elicit.org/binary/questions/2b3PzqXn9) I took some liberties in operationalising what seemed to me a core thesis underlying the post. Let me know if you think it doesn't really capture the important stuff! (You can find a list of all review poll questions here.)
11Zvi
The central point here seems strong and important. One can, as Scott notes, take it too far, but mostly yes one should look where there are very interesting things even if the hit rate is not high, and it's important to note that. Given the karma numbers involved and some comments sometimes being included I'd want assurance that we wouldn't include any of that with regard to particular individuals.  That comment section, though, I believe has done major harm and could keep doing more even in its current state, so I still worry about bringing more focus on this copy of the post (as opposed to the SSC copy). Also, I worry about this giving too much of a free pass to what it calls "outrage culture" - there's an implicit "yeah, it's ok to go all essentialist and destroy someone for one statement that breaks your outrage mob's rules, I can live with that and please don't do it to me here, but let's not extend that to things that are merely stupid or wrong." I don't think you can do that, it doesn't work that way. Could be fixed with an edit if Scott wanted it fixed. 
37Zac Hatfield-Dodds
I remain both skeptical some core claims in this post, and convinced of its importance. GeneSmith is one of few people with such a big-picture, fresh, wildly ambitious angle on beneficial biotechnology, and I'd love to see more of this genre. One one hand on the object level, I basically don't buy the argument that in-vivo editing could lead to substantial cognitive enhancement in adults. Brain development is incredibly important for adult cognition, and in the maybe 1%--20% residual you're going well off-distribution for any predictors trained on unedited individuals. I too would prefer bets that pay off before my median AI timelines, but biology does not always allow us to have nice things. On the other, gene therapy does indeed work in adults for some (much simpler) issues, and there might be valuable interventions which are narrower but still valuable. Plus, of course, there's the nineteen-ish year pathway to adults, building on current practice. There's no shortage of practical difficulties, but the strong or general objections I've seen seem ill-founded, and that makes me more optimistic about eventual feasibility of something drawing on this tech tree. I've been paying closer attention to the space thanks to Gene's posts, to the point of making some related investments, and look forward to watching how these ideas fare on contact with biological and engineering reality over the next few years.
16Eigil Rischel
This post introduces a potentially very useful model, both for selecting problems to work on and for prioritizing personal development. This model could be called "The Pareto Frontier of Capability". Simply put: 1. By an efficient markets-type argument, you shouldn't expect to have any particularly good ways of achieving money/status/whatever - if there was an unusually good way of doing that, somebody else would already be exploiting it. 2. The exception to this is that if only a small amount of people can exploit an opportunity, you may have a shot. So you should try to acquire skills that only a small number of people have. 3. Since there are a lot of people in the world, it's incredibly hard to become among the best in the world at any particular skill. 4. This means you should position yourself on the Pareto Frontier - you should seek out a combination of skills where nobody else is better than you at everything. Then you will have the advantage in problems where all these skills matter. It might be important to contrast this with the economical term comparative advantage, which is often used informally in a similar context. But its meaning is different. If we are both excellent programmers, but you are also a great writer, while I suck at writing, I have a comparative advantage in programming. If we're working on a project together where both writing and programming are relevant, it's best if I do as much programming as possible while you handle as much as the writing as possible - even though you're as good at me as programming, if someone has to take off time from programming to write, it should be you. This collaboration can make you more effective even though you're better at everything than me (in the economics literature this is usually conceptualized in terms of nations trading with each other). This is distinct from the Pareto optimality idea explored in this post. Pareto optimality matters when it's important that the same person does both the
13SebastianG
“The Tails Coming Apart as a Metaphor for Life” should be retitled “The Tails Coming Apart as a Metaphor for Earth since 1800.” Scott does three things, 1) he notices that happiness research is framing dependent, 2) he notices that happiness is a human level term, but not specific at the extremes, 3) he considers how this relates to deep seated divergences in moral intuitions becoming ever more apparent in our world. He hints at why moral divergence occurs with his examples. His extreme case of hedonic utilitarianism, converting the entire mass of the universe into nervous tissue experiencing raw euphoria, represents a ludicrous extension of the realm of the possible: wireheading, methadone, subverting factory farming. Each of these is dependent upon technology and modern economies, and presents real ethical questions. None of these were live issues for people hundreds of years ago. The tails of their rival moralities didn’t come apart – or at least not very often or in fundamental ways. Back then Jesuits and Confucians could meet in China and agree on something like the “nature of the prudent man.” But in the words of Lonergan that version of the prudent man, Prudent Man 1.0, is obsolete: “We do not trust the prudent man’s memory but keep files and records and develop systems of information retrieval. We do not trust the prudent man’s ingenuity but call in efficiency experts or set problems for operations research. We do not trust the prudent man’s judgment but employ computers to forecast demand,” and he goes on. For from the moment VisiCalc primed the world for a future of data aggregation, Prudent Man 1.0 has been hiding in the bathroom bewildered by modern business efficiency and moon landings. Let’s take Scott’s analogy of the Bay Area Transit system entirely literally, and ask the mathematical question: when do parallel lines come apart or converge? Recall Euclid’s Fifth Postulate, the one saying that parallel lines will never intersect. For almost a couple
31philh
I think I agree with the thrust of this, but I think the comment section raises caveats that seem important. Scott's acknowledged that there's danger in this, and I hope an updated version would put that in the post. But also... This seems like a strange model to use. We don't know, a priori, what % are false. If 50% are obviously false, probably most of the remainder are subtly false. Giving me subtly false arguments is no favor. Scott doesn't tell, us, in this essay, what Steven Pinker has given him / why Steven Pinker is ruled in. Has Steven Pinker given him valuable insights? How does Scott know they're valuable? (There may have been some implicit context when this was posted. Possibly Scott had recently reviewed a Pinker book.) Given Anna's example, I find myself wondering, has Scott checked Pinker's straightforwardly checkable facts? I wouldn't be surprised if he has. The point of these questions isn't to say that Pinker shouldn't be ruled in, but that the questions need to be asked and answered. And the essay doesn't really acknowledge that that's actually kind of hard. It's even somewhat dismissive; "all you have to do is *test* some stuff to *see if it’s true*?" Well, the Large Hadron Collider cost €7.5 billion. On a less extreme scale, I recently wanted to check some of Robert Ellickson's work; that cost me, I believe, tens of hours. And that was only checking things close to my own specialty. I've done work that could have ruled him out and didn't, but is that enough to say he's ruled in? So this advice only seems good if you're willing and able to put in the time to find and refute the bad arguments. Not only that, if you actually will put in that time. Not everyone can, not everyone wants to, not everyone will do. (This includes: "if you fact-check something and discover that it's false, the thing doesn't nevertheless propagate through your models influencing your downstream beliefs in ways it shouldn't".) If you're not going to do that... I don
15DirectedEvolution
If coordination services command high wages, as John predicts, this suggests that demand is high and supply is limited. Here are some reasons why this might be true: 1. Coordination solutions scale linearly (because the problem is a general one) or exponentially (due to networking effects). 2. Coordination is difficult, unpleasant, risky work. 3. Coordination relies on further resources that are themselves in limited supply or on information that has a short life expectancy, such as involved personal relationships, technical knowhow that depends on a lot of implicit knowledge, familiarity with language and culture, access to user bases and communities, access to restricted communication channels and information, trust, credentials, charisma, money, land, or legal privileges. 4. Coordination is most intensively needed in innovative, infrastructure-development work, which is a high-risk area with long-term payoffs.  5. Coordination is neglected due to systematic biases on an individual and/or institutional level. Perhaps coordination is easy to learn, but is difficult to train in an educational context, and as such is frequently neglected by the educational system. Students are therefore mis-incentivized and don’t engage in developing their coordination skills to anywhere near the possible and optimal level. Alternatively, it might be that we teach coordination in the context of centrally coordination-focused careers (MBAs, for example), but that many other careers less obviously centrally focused on coordination (bench scientists) would also benefit - a problem of interdisciplinary neglect. Note that, if the argument in my review of interfaces as scarce resources is correct, then coordination can also be viewed as a subtype of interface - a way of translating between what a user wants and how they express that desire, into the internal language or structure of a complex system. This makes sense. Google translates natural-language queries into the PageRank algo
14Viliam
Two years later, I suppose we know more than we did when the article was written. I would like to read some postscript explaining how well this article has aged.
11jasoncrawford
Since writing this, I've run across even more examples: * The transatlantic telegraph was met with celebrations similar to the transcontinental railroad, etc. (somewhat premature as the first cable broke after two weeks). Towards the end of Samuel Morse's life and/or at his death, he was similarly feted as a hero. * The Wright Brothers were given an enormous parade and celebration in their hometown of Dayton, OH when they returned from their first international demonstrations of the airplane. I'd like to write these up at some point. Related: The poetry of progress (another form of celebration, broadly construed)
11Zac Hatfield-Dodds
I think Elizabeth is correct here, and also that vegan advocates would be considerably more effective with higher epistemic standards: The post unfortunately suffers for its length, detailed explanations, and rebuttal of many motivated misreadings - many of which can be found in the comments, so it's unclear whether this helped. It's also well-researched and cited, well organized, offers cruxes and anticipates objections - vegan advocates are fortunate to have such high-quality criticism. This could have been a shorter post, which was about rather than engaged in epistemics and advocacy around veganism, with less charitable assumptions. I'd have shared that shorter post more often, but I don't think it would be better.
10DirectedEvolution
There's a lot of attention paid these days to accommodating the personal needs of students. For example, a student with PTSD may need at least one light on in the classroom at all times. Schools are starting to create mechanisms by which a student with this need can have it met more easily. Our ability to do this depends on a lot of prior work. The mental health community had to establish PTSD as a diagnosis; the school had to create a bureaucratic mechanism to normalize accommodations of this kind; and the student had to spend a significant amount of time figuring out what accommodations alleviated their PTSD symptoms and how to get them addressed through the school's bureaucracy. This points in a direction of something like "transitions research," an attempt to identify and economically address the specific barriers that skew individuals toward immediate modest-productivity strategies and away from long-term high-productivity strategies. Imagine if there was a well-known "diagnosis" of "status-loss anxiety," in which a person who's achieved some professional success notices themselves avoiding situations that would be likely to enhance their growth, yet come with a threat of loss of status. It's like the depressed person who resists mental health unseling because it implies there's something wrong with them. Being able to identify that precise reaction, label it, raise awareness of it, and find means and messages to address it would be helpful to overcome a barrier to mental health treatment. In economics jargon, what's going on here is not so much the sunk cost fallacy as a combination of aging, opportunity cost and diminishing returns. Learning takes time, aging us, and this means we have less time to profit off a new long-term investment in skill-building. Increased skill raises the opportunity cost of learning new skills. Diminishing returns means that, if we learn a skill that increases our profit from A + B to A + 2B, that this is less intrinsically valu
10Raemon
This a first pass review that's just sort of organizing my thinking about this post. This post makes a few different types of claims: * Hyperselected memes may be worse (generally) than weakly selected ones * Hyperselected memes may specifically be damaging our intelligence/social memetic software * People today are worse at negotiating complex conflicts from different filter bubbles * There's a particular set of memes (well represented in 1950s sci-fi) that was particularly important, and which are not as common nowadays. It has a question which is listed although not focused on too explicitly on its own terms: * What do you do if you want to have good ideas? (i.e. "drop out of college? read 1950s sci-fi in your formative years?") It prompts me to separately consider the questions: * What actually is the internet doing to us? It's surely doing something. * What sorts of cultures are valuable? What sorts of cultures can be stably maintained? What sorts of cultures cause good intellectual development? ... Re: the specific claim of "hypercompetition is destroying things", I think the situation is complicated by the "precambrian explosion" of stuff going on right now. Pop music is defeating classical music in relative terms, but, like, in absolute terms there's still a lot more classical music now than in 1400 [citation needed?]. I'd guess this is also true of for tribal FB comments vs letter-to-the-editor-type writings.  * [claim by me] Absolute amounts of thoughtful discourse is probably still increasing My guess is that "listens carefully to arguments" has just always been rare, and that people have generally been dismissive of the outgroup, and now that's just more prominent. I'd also guess that there's more 1950s style sci-fi today than in 1950. But it might not be, say, driving national projects that required a critical mass of it. (And it might or might not be appearing on bestseller lists?) If so, the question is less "are things being destro
33DirectedEvolution
The referenced study on group selection on insects is "Group selection among laboratory populations of Tribolium," from 1976. Studies on Slack claims that "They hoped the insects would evolve to naturally limit their family size in order to keep their subpopulation alive. Instead, the insects became cannibals: they ate other insects’ children so they could have more of their own without the total population going up."  This makes it sound like cannibalism was the only population-limiting behavior the beetles evolved. According to the original study, however, the low-population condition (B populations) showed a range of population size-limiting strategies, including but not limited to higher cannibalism rates. "Some of the B populations enjoy a higher cannibalism rate than the controls while other B populations have a longer mean developmental time or a lower average fecundity relative to the controls. Unidirectional group selection for lower adult population size resulted in a multivarious response among the B populations because there are many ways to achieve low population size." Scott claims that group selection can't work to restrain boom-bust cycles (i.e. between foxes and rabbits) because "the fox population has no equivalent of the overarching genome; there is no set of rules that govern the behavior of every fox." But the empirical evidence of the insect study he cited shows that we do in fact see changes in developmental time and fecundity. After all, a species has considerable genetic overlap between individuals, even if we're not talking about heavily inbred family members, as we'd be seeing in the beetle study. Wikipedia's article on human genetic diversity cites a Nature article and says "as of 2015, the typical difference between an individual's genome and the reference genome was estimated at 20 million base pairs (or 0.6% of the total of 3.2 billion base pairs)." An explanation here is that the inbred beetles of the study are becoming progressiv
12DirectedEvolution
This post is based on the book Moral Mazes, which is a 1988 book describing "the way bureaucracy shapes moral consciousness" in US corporate managers. The central point is that it's possible to imagine relationship and organization structures in which unnecessarily destructive behavior, to self or others, is used as a costly signal of loyalty or status. Zvi titles the post after what he says these behaviors are trying to avoid, motive ambiguity. He doesn't label the dynamic itself, so I'll refer to it here as "disambiguating destruction" (DD). Before proceeding, I want to emphasize that DD is referring to truly pointless destruction for the exclusive purpose of signaling a specific motive, and not to an unavoidable tradeoff. This raises several questions, which the post doesn't answer. 1. Do pointlessly destructive behaviors typically succeed at reducing or eliminating motive ambiguity? 2. Do they do a better job of reducing motive ambiguity than alternatives? 3. How common is DD in particular types of institutions, such as relationships, cultures, businesses, and governments? 4. How do people manage to avoid feeling pressured into DD? 5. What exactly are the components of DD, so that we can know what to look for when deciding whether to enter into a certain organization or relationship? 6. Are there other explanations for the components of DD, and how would we distinguish between DD and other possible interpretations of the component behaviors? We might resort to a couple explanations for (4), the question of how to avoid DD. One is the conjunction of empathy and act utilitarianism. My girlfriend says she wouldn't want to go to a restaurant only she loves, even if the purpose was to show I love her. Part of her enjoyment is my enjoyment of the experience. If she loved the restaurant only she loves so much that she was desperate to go, then she could go with someone else. She finds the whole idea of destructive disambiguation of love to be distinctly unapp
23Martin Sustrik
Author here. I still believe this article is a important addition to the discussion of inadequate equilibria. While Scott Alexander's Moloch post and Eliezer Yudkowsky's book are great for introduction and discussion of the topic, both of them fail, in my opinion, to convey the sheer complexity of the problem as it occurs in the real world. That, I think, results in readers thinking about the issue in simple malthusian or naive game-theoretic terms and eventually despairing about inescapability of suboptimal Nash equilibria. What I try to present is a world that is much more complex but also much less hopeless. Everything is an intricate mess of games played on different levels and interacting in complex and unpredictable ways. What, at the first glance, looks like a simple tragedy-of-the-commons problem is in fact a complex dynamic system with many inputs and many intertwined interests. To solve it, one may just have to step back a bit and consider other forces and mechanisms at play. One idea that is expressed in the article and that I often come back to is (my wording, but the idea is very much implicitly present in Ostrom's book): Another one that still feels important in the hindsight is the attaching of a price tag to a coordination failure ("this can be solved for $1M") which turns the semi-mystical work of Moloch into a boring old infrastructure project, very much like building a dam. This may have implications for Effective Altruism. Solving a coordination failure may often be the most efficient way to spend money in a specific area.
21Vanessa Kosoy
This essay provides some fascinating case studies and insights about coordination problems and their solutions, from a book by Elinor Ostrom. Coordination problems are a major theme in LessWrongian thinking (for good reasons) and the essay is a valuable addition to the discussion. I especially liked the 8 features of sustainable governance systems (although I wish we got a little more explanation for "nested enterprises"). However, I think that the dichotomy between "absolutism (bad)" and "organically grown institutions (good)" that the essay creates needs more nuance or more explanation. What is the difference between "organic" and "inorganic" institutions? All institutions "grew" somehow. The relevant questions are e.g. how democratic is the institution, whether the scope of the institution is the right scope for this problem, whether the stakeholders have skin in the game (feature 3) et cetera. The 8 features address some of that, but I wish it was more explicit. Also, It's notable that all examples focus on relatively small scale problems. While it makes perfect sense to start by studying small problems before trying to understand the big problems, it does make me wonder whether going to higher scales brings in qualitatively new issues and difficulties. Paying to officials with parcels in the tail end works for water conflicts, but what is the analogous approach to global warming or multinational arms races?
13Bird Concept
I'm trying out making some polls about posts for the Review (using the predictions feature). You can answer by hovering over the scale and clicking a number to indicate your agreement with the claim.  Making more land out of the about 50mi^2 shallow water in the San Francisco Bay, South of the Dumbarton Bridge, would...  Elicit Prediction (elicit.org/binary/questions/KkqpSr5rW) Elicit Prediction (elicit.org/binary/questions/qzzNzEfa9) Elicit Prediction (elicit.org/binary/questions/csYlcNdhZ) Elicit Prediction (elicit.org/binary/questions/RwtAoMlnP) Elicit Prediction (elicit.org/binary/questions/xGIZipvb-) Elicit Prediction (elicit.org/binary/questions/zAtqSgbnS) For some of these questions, I tried to operationalise them to be less ambiguous than Jeff's original formulation. 
18TurnTrout
This will not be a full review—it's more of a drive-by comment which I think is relevant to the review process. I am extremely skeptical of and am not at all confident in this conclusion. Ellsberg's The Doomsday Machine describes a horribly incentivized military establishment which pursued bloodthirsty and senseless policies, deceiving their superiors (including several presidents), breaking authentication protocols, refusing to adopt plans which didn't senselessly destroy China in a conflict in the Soviet Union, sub-delegation of nuclear launch authority to theater commanders and their subordinates (no, it's not operationally true that the US president has to authorize an attack!), lack of controls against false alarms, and constant presidential threats of first-use. The USAF would manipulate presidential officials in order to secure funding, via tactics such as inflating threat estimates or ignoring evidence that the Soviet Union had less nuclear might than initially thought. And Ellsberg stated that he didn't think much had changed since his tenure in the 50s-70s. While individual planners might be aware of the nuclear winter risks, the overall US military establishment seems insane to me around nuclear policy—and what of those in other nuclear powers?  However, The Doomsday Machine is my only exposure to these considerations, and perhaps I'm missing a broader perspective. If so, I think that case should be more clearly spelled out, because as far as I can tell, nuclear policy seems like yet another depravedly inadequate facet of our current civilization. 
14David Hornbein
This sort of thing is exactly what Less Wrong is supposed to produce. It's a simple, straightforward and generally correct argument, with important consequences for the world, which other people mostly aren't making. That LW can produce posts like this—especially with positive reception and useful discussion—is a vindication of this community's style of thought.
16Bucky
The post claims: This review aims to assess whether having read the post I can conclude the same. The review is split into 3 parts: * Epistemic spot check * Examining the argument * Outside the argument Epistemic spot check Claim: There are 14,000 nuclear warheads in the world. Assessment: True Claim: Average warhead yield <1 Mt, probably closer to 100kt Assessment: Probably true, possibly misleading. Values I found were: * US * W78 warhead: 335-350kt * W87 warhead: 300 or 475 kt * Russia * R-36 missile: 550-750 kt * R29 missile: 100 or 500kt The original claim read to me that 100kT was probably pretty close and 1Mt was a big factor of safety (~x10) but whereas the safety factor was actually less than that (~x3). However that’s the advantage of having a safety factor – even if it’s a bit misleading there still is a safety factor in the calculations. I found the lack of links slightly frustrating here – it would have been nice to see where the OP got the numbers from. Examining the argument The argument itself can be summarized as: 1. Kinetic destruction can’t be big enough 2. Radiation could theoretically be enough but in practice wouldn’t be 3. Nuclear winter not sufficient to cause extinction One assumption in the arguments for 1 & 2 is that the important factor is the average warhead yield and that e.g. a 10Mt warhead doesn’t have an outsized effect. This seems likely and a comment suggests that going over 500kt doesn’t make as much difference as might be thought and that is why warheads are the size that they are. Arguments 1 & 2 seem very solid. We have done enough tests that our understanding of kinetic destruction is likely to be fairly good so I don’t have much concerns there. Similarly, radiation is well understood and dispersal patterns seem kinda predictable in principle and even if these are wrong the total amount of radiation doesn't change, just the where it is. Climate change is less easy to model, especially giv
28johnswentworth
ETA 1/12: This review is critical and at times harsh, not because I want to harshly criticize the post or the author, but because I did not consider harshness of criticism when writing. I still think the post is positive-net-value, and might even vote it up in the review. I especially want to emphasize that I do not think it is in any way useful to blame or punish the author for the things I complain about below; this is intended as a "pointing out a problematic habit which a lot of people have and society often encourages" criticism, not a "bad thing must be punished" criticism. When this post first came out, I said something felt off about it. The same thing still feels off about it, but I no longer endorse my original explanation of what-felt-off. So here's another attempt. First, what this post does well. There's a core model which says something like "people with the power to structure incentives tend get the appearance of what they ask for, which often means bad behavior is hidden". It's a useful and insightful model, and the post presents it with lots of examples, producing a well-written and engaging explanation. The things which the post does well more than outweigh the problems below; it's a great post. On to the problem. Let's use the slave labor example, because that's the first spot where the problem comes up: ... so far, so good. This is generally solid analysis of an interesting phenomenon. But then we get to the next sentence: ... and this where I want to say NO. My instinct says DO NOT EVER ASK THAT QUESTION, it is a WRONG QUESTION, you will be instantly mindkilled every time you ask "who should be blamed for X?". ... on reflection, I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective. Asking "who should we blame?" is always engaging in a status fight. Status fights are generally mindkillers, and should be kept strictly separate from modelling and epistemics
26ryan_b
I think this post should be included in the best posts of 2018 collection. It does an excellent job of balancing several desirable qualities: it is very well written, being both clear and entertaining; it is informative and thorough; it is in the style of argument which is preferred on LessWrong, by which I mean makes use of both theory and intuition in the explanation. This post adds to the greater conversation by displaying rationality of the kind we are pursuing directed at a big societal problem. A specific example of what I mean that distinguishes this post from an overview that any motivated poster might write is the inclusion of Warren Smith's results; Smith is a mathematician from an unrelated field who has no published work on the subject. But he had work anyway, and it was good work which the author himself expanded on, and now we get to benefit from it through this post. This puts me very much in mind of the fact that this community was primarily founded by an autodidact who was deeply influenced by a physicist writing about probability theory. A word on one of our sacred taboos: in the beginning it was written that Politics is the Mindkiller, and so it was for years and years. I expect this is our most consistently and universally enforced taboo. Yet here we have a high-quality and very well received post about politics, and of the ~70 comments only one appears to have been mindkilled. This post has great value on the strength of being an example of how to address troubling territory successfully. I expect most readers didn't even consider that this was political territory. Even though it is a theory primer, it manages to be practical and actionable. Observe how the very method of scoring posts for the review, quadratic voting, is one that is discussed in the post. Practical implications for the management of the community weigh heavily in my consideration of what should be considered important conversation within the community. Carrying on from that
11Unnamed
It seems like the core thing that this post is doing is treating the concept of "rule" as fundamental.  If you have a general rule plus some exceptions, then obviously that "general rule" isn't the real process that is determining the results. And noticing that (obvious once you look at it) fact can be a useful insight/reframing. The core claim that this post is putting forward, IMO, is that you should think of that "real process" as being a rule, and aim to give it the virtues of good rules such as being simple, explicit, stable, and legitimate (having legible justifications). An alternative approach is to step outside of the "rules" framework and get in touch with what the rule is for - what preferences/values/strategy/patterns/structures/relationships/etc. it serves. Once you're in touch with that purpose, then you can think about both the current case, and what will become of the "general rule", in that light. This could end up with an explicitly reformulated rule, or not. It seems like treating the "real process" as a rule is more fitting in some cases than others, a better fit for some people's style of thinking than for other people's, and also something that a person could choose to aim for more or less. I think I'd find it easier to think through this topic if there was a long, diverse list of brief examples.
12hamnox
Biorisk - well wouldn't it be nice if we'd all been familiar with the main principles of biorisk before 2020? i certainly regretted sticking my head in the sand. > If concerned, intelligent people cannot articulate their reasons for censorship, cannot coordinate around principles of information management, then that itself is a cause for concern. Discussions may simply move to unregulated forums, and dangerous ideas will propagate through well intentioned ignorance. Well. It certainly sounds prescient in hindsight, doesn't it? Infohazards in particular cross my mind: so many people operate on extremely bad information right now. Conspiracies theories abound, and I imagine the legitimate coordination for secrecy surrounding the topic do not help in the least. What would help? Exactly this essay. A clear model of *what* we should expect well-intentioned secrecy to cover, so we can reason sanely over when it's obviously not. Y'all done good. This taxonomy clarifies risk profiles better than Gregory Lewis' article, though I think his includes a few vivid-er examples. I opened a document to experiment tweaking away a little dryness from the academic tone. I hope you don't take offense. Your writing represents massive improvements in readability in its examples and taxonomy, and you make solid, straightforward choices in phrasing. No hopelessly convoluted sentence trees. I don't want to discount that. Seriously! Good job. As I read I had a few ideas spark on things that could likely get done at a layman level, in line with spiracular's comment. That comment could use some expansion, especially in the direction of "Prefer to discuss this over that, or discuss in *this way* over *that way" for bad topics. Very relevantly, I think basic facts should get added to some the good discussion topics, since they represent information it's better to disseminate! we seek to review basic facts under the good discussion topics, since they represent information it's better to diss
11Drake Morrison
A great example of taking the initiative and actually trying something that looks useful, even when it would be weird or frowned upon in normal society. I would like to see a post-review, but I'm not even sure if that matters. Going ahead and trying something that seems obviously useful, but weird and no one else is doing is already hard enough. This post was inspiring. 
13fiddler
This review is more broadly of the first several posts of the sequence, and discusses the entire sequence.  Epistemic Status: The thesis of this review feels highly unoriginal, but I can't find where anyone else discusses it. I'm also very worried about proving too much. At minimum, I think this is an interesting exploration of some abstract ideas. Considering posting as a top-level post. I DO NOT ENDORSE THE POSITION IMPLIED BY THIS REVIEW (that leaving immoral mazes is bad), AND AM FAIRLY SURE I'M INCORRECT. The rough thesis of "Meditations on Moloch" is that unregulated perfect competition will inevitably maximize for success-survival, eventually destroying all value in service of this greater goal. Zvi (correctly) points out that this does not happen in the real world, suggesting that something is at least partially incorrect about the above mode, and/or the applicability thereof. Zvi then suggests that a two-pronged reason can explain this: 1. most competition is imperfect, and 2. most of the actual cases in which we see an excess of Moloch occur when there are strong social or signaling pressures to give up slack.  In this essay, I posit an alternative explanation as to how an environment with high levels of perfect competition can prevent the destruction of all value, and further, why the immoral mazes discussed later on in this sequence are an example of highly imperfect competition that causes the Molochian nature thereof.  First, a brief digression on perfect competition: perfect competition assumes perfectly rational agents. Because all strategies discussed are continuous-time, the decisions made in any individual moment are relatively unimportant assuming that strategies do not change wildly from moment to moment, meaning that the majority of these situations can be modeled as perfect-information situations.  Second, the majority of value-destroying optimization issues in a perfect-competition environment can be presented as prisoners dilemmas: both