Summary for regular readers: The epistemic content of The Sequences — i.e., the advice on finding true beliefs — has a different epistemic status than the instrumental content — i.e., the advice on how to behave.  Specifically, the epistemic content is based upon techniques from logic, probability, statistics, and causal inference that have already been heavily explored and vetted, both formally and empirically, in a wide range of disciplines and application areas.  This is not true of the how-to-behave content of the sequences, such as the stances presented on individual heroism and what does or not not constitute humble behavior.  Indeed, the analogous underpinning fields — namely, decision theory, game theory, ethics, meta-ethics, and political theory — are not nearly as well explored-and-vetted as the epistemic underpinnings. As such, the epistemic content of the sequences should be made available in a separate compendium, curated for its special epistemic status.   This post is one attempt at that, which I'd be happy to have replaced or superseded by a more official attempt from, say, the LessWrong team, or Eliezer Yudkowsky.

Followed by: What's next for instrumental rationality?

Introduction for newcomers

The good

The "Epistemic Sequences" curated below are taken from the writings of Eliezer Yudkowsky.  Yudkowsky's writings cover some of the best introductions I've seen for helping individual people to start forming true beliefs about the world, or at least, beliefs that become progressively less wrong over time.  The Epistemic Sequences describe processes to follow to form true beliefs, rather than merely conclusions to reach, and the prescribed processes are well backed by logic, statistics, and causal inference.  They draw inspiration from well-written, tried-and-true textbooks like these:

• (2000) H. B. Enderton: A Mathematical Introduction to Logic
• (2003) E.T. Jaynes: Probability Theory: The Logic of Science
• (2000/2009) J. Pearl: Causality: Models, Reasoning, and Inference*

(* J. Pearl: Causal Inference in Statistics: A Primer is a more recent and friendlier intro.)

The questionable

Epistemic status: broadly verifiable claims about fields of theory and practice and their relation to the LessWrong Sequences.

For better or worse, the broader LessWrong Sequences and Sequence Highlights also contain a lot of strong stances on how to behave, i.e., what decisions and actions to take given one's beliefs, including considerations of ethics, meta-ethics, and human values.  There are strong themes of unilateral heroism, how to be humble, and how to best protect and serve humanity.  Compared to the epistemic aspects of the sequences, these how-to-behave aspects are not founded in well-established technical disciplines like logic and statistics, even though they have a tendency to motivate people to form true beliefs and attempt to help the world.  The analogous underpinning fields — including decision theory, game theory, ethics, meta-ethics, political theory — are nowhere near as well explored-and-vetted as logic and statistics.

My take

Epistemic status: my opinion as a researcher professionally dedicated to helping the world.

Feel free to skip this part if you're not particularly interested in my opinion.

That said: I think the how-to-behave themes of the LessWrong Sequences are at best "often wrong but sometimes motivationally helpful because of how they inspire people to think as individuals and try to help the world", and at worst "inspiring of toxic relationships and civilizational disintegration."  I'm not going to try to argue this position here, though, because I think it would distract from the goodness of the epistemic content, which I'd like to see promoted in its purest possible form.  

Also, I'd like to add that Eliezer Yudkowsky can't be "blamed" for the absence of ideal technical underpinnings for the how-to-behave aspects of the sequences.  In fact, he and his colleagues at MIRI have made world-class attempts to improve these foundations through "Agent Foundations" research.  E.g., Scott Garrabrant's discovery of Logical Inductors (on which I had the privilege of serving as his co-author) was a research direction that was inspired by Yudkowsky and funded by MIRI.  

Importantly, the strength of my dislike for what I consider the 'toxic' aspects of the sequences is also not based on tried-and-true technical underpinnings, any more than Eliezer's original writings.  Like Eliezer, I've tried to advance research on foundations of individual and collective decision-making — e.g., negotiable reinforcement learning, fair division algorithms, Lobian cooperation, expert aggregation criteria, and equilibria of symmetric games — but even new technical insights I've made along this journey do not have the tried-everywhere-and-definitely-helpful properties that logic, probability, statistics, and causal inference have. 

Nonetheless, for the purpose of building a community of people who collectively pursue the truth and sometimes attempt to take collective actions, I think it's important to call out the questionability of the LessWrong Sequences for its how-to-behave content, and promote the epistemic content as being valuable despite this critique.

The Epistemic Sequences, list v.0.1

The content below is taken from the Sequence Highlights created by the LessWrong team, with only strikethroughs and italicized interjections from me marked by carets (^).

Thinking Better on Purpose

Part 1 of 6 from the Sequence Highlights. 

Humans can not only think, but think about our own thinking. This makes it possible for us to recognize the shortcomings of our default reasoning and work to improve it – the project of human rationality. 

<33 min read

• The Lens That Sees Its Flaws
• What Do We Mean By "Rationality"?
• Humans are not automatically strategic
  • ^ Flag: this was almost removed for its how-to-behave content, but kept because it's more framed as observation-about-what-happens rather than what-you-should-do.
• Use the Try Harder, Luke
  • ^ Removed for strong how-to-behave content.
• Your Strength as a Rationalist
• The Meditation on Curiosity
• The Importance of Saying "Oops"
• The Martial Art of Rationality
• Twelve Virtues of Rationality
  • ^ Flag: this was almost removed because the 'humility' virtue is a particular prescription for how to behave, and because the framing of the post sounds like a treatise on how to live, but actually it's mostly about beliefs.

Pitfalls of Human Cognition

Part 2 of 6 from the Sequence Highlights.

A major theme of the Sequences is the ways in which human reasoning goes astray. This sample of essays describes a number of failure modes and invokes us to do better.

34 min read

• The Bottom Line
• Rationalization
• You Can Face Reality
• Is That Your True Rejection?
• Avoiding Your Belief's Real Weak Points
• Belief as Attire
  • ^ Flag: this was almost removed for strong overtones about whether people should or should not engage in certain kinds of signaling, but kept because these overtones are not super-explicit, and because much of the post is observational.
• Cached Thoughts
• The Fallacy of Gray
• Lonely Dissent
• Positive Bias: Look Into the Dark
  • ^ Flag: this was almost removed for have a strong overtone of encouraging pessimism (a bias), but kept because its explicit content is more about confirmation bias than social positivity (presumptions of good faith).
• Knowing About Biases Can Hurt People
• Politics is the Mind-Killer
  • ^ Flag: this was almost removed for having a strong overtone encouraging people not to get involved in politics, but kept because the explicit content is mostly observational, and most of the advice is still about how to practice true-belief-formation.

The Laws Governing Belief

Part 3 of 6 from the Sequence Highlights.

While beliefs are subjective, that doesn't mean that one gets to choose their beliefs willy-nilly. There are laws that theoretically determine the correct belief given the evidence, and it's towards such beliefs that we should aspire.

82 min read

• Making Beliefs Pay Rent (in Anticipated Experiences)
• What is Evidence?
• Scientific Evidence, Legal Evidence, Rational Evidence
• How Much Evidence Does It Take?
• Absence of Evidence Is Evidence of Absence
• Conservation of Expected Evidence
  • ^ Flag: this was almost removed because it's often misapplied, i.e., around a dozen readers that I've met have cited this post as saying "you can't predict the direction of your belief updates", which is wrong, but that's not what the post actually says.  Conclusion: this post needs to be read carefully to avoid confusion.
• Argument Screens Off Authority
• An Intuitive Explanation of Bayes's Theorem
• The Second Law of Thermodynamics, and Engines of Cognition
• Toolbox-thinking and Law-thinking
• Local Validity as a Key to Sanity and Civilization
  • ^ Removed for making a lot of claims about multi-agent coordination that are not well founded in logic, statistics, and causality (rather, in game theory, which is not as well vetted for real-world-use).

Science Isn't Enough

Part 4 of 6 from the Sequence Highlights.

While far better than what came before, "science" and the "scientific method" are still crude, inefficient, and inadequate to prevent you from wasting years of effort on doomed research directions.

18 min read

• When Science Can't Help
  • ^ Flag: almost removed for a strong overtone of encouraging cryonics (a behavior) , but kept because the explicit content is about whether it works or not (a belief).
• Faster Than Science
• Science Doesn't Trust Your Rationality
  • ^ Removed for not drawing actionable belief-forming advice from logic / statistics / causality, while arguably being a pro-libertarian how-to-behave piece.
• No Safe Defense, Not Even Science
  • ^ Removed for not drawing actionable belief-forming advice from logic / statistics / causality, while encouraging emotional distrust in the sanity of other people.
  • ^ Worth reading as a contextualization of Yukdowsky's other writings.
     

Connecting Words to Reality

Part 5 of 6 from the Sequence Highlights.

To understand reality, especially on confusing topics, it's important to understand the mental processes involved in forming concepts and using words to speak about them.

33 min read

• Taboo Your Words
• Dissolving the Question
• Diseased thinking: dissolving questions about disease
• Hug the Query
• Say Not "Complexity"
• Mind Projection Fallacy
• How An Algorithm Feels From Inside
• Expecting Short Inferential Distances
• Illusion of Transparency: Why No One Understands You

Why We Fight

^ Flag: This (Part 6) was almost removed entirely due to not carrying much in the way of belief-formation advice.  Perhaps it should be removed from a more curated "Epistemic Sequences" compendium.

Part 6 of 6 from the Sequence Highlights.

The pursuit of rationality and that of doing better on purpose, can in fact be rather hard. You have to get the motivation for that from somewhere.

31 min read

• Something to Protect
  • ^ Flag: almost removed for encouraging unilateral heroism (a behavior), and for not carrying much belief-formation advice, but kept because many people find it motivating.  Perhaps this should still be removed from a more curated "Epistemic Sequences" compendium.
• The Gift We Give To Tomorrow
  • ^ Flag: almost removed for not carrying much belief-formation advice, but kept because many people find it motivating.  Perhaps this should still be removed from a more curated "Epistemic Sequences" compendium.
• On Caring
  • ^ Removed for not carrying much belief-formation advice, and for explicitly advising on how to feel about about relate to other people (emotions are not quite behavior, but intermediate between belief and behavior).
• Tsuyoku Naritai! (I Want To Become Stronger)
  • ^ Removed for not carrying much belief formation advice, and for explicitly advising on how to relate to others (behavior).
• A Sense That More Is Possible
  • ^ Flag: almost removed for not carrying much belief-formation advice, but kept because many people find it motivating.

What's next?

I'm not sure!  Perhaps an official "Epistemic Sequences" compendium could someday be be produced that focusses entirely on epistemics, with the potential upsides of

• more strongly emphasizing the most-well-founded aspects of the Sequences;
• avoiding turning off readers who find the how-to-behave aspects of the sequences off-putting (either because the Sequences are wrong and those readers can sense it, or just because the arguments aren't strong enough, or both); and
• yielding a larger and broader community of people who can agree on beliefs and good belief formation practices, even if they don't (yet) agree on how to treat each other or the rest of the world.

For now, I'll just content myself to link interested readers to this post if they ask me which parts of the LessWrong sequences are most worth reading and why.

Followed by: What's next for instrumental rationality?