This post is part of my Hazardous Guide To Rationality. I don't expect this to be new or exciting to frequent LW people, and I would super appreciate comments and feedback in light of intents for the sequence, as outlined in the above link. Also, note this is a STRUCTURE post, again see the above link for what that means.


Communication can be hard. Thinking can be hard. Trying to get another person to understand what you mean can be hard. Trying to explain to yourself what you mean can be hard. This sequence draws out in detail some of the things that happen in human brains that leads to these hardships, and proposes a few tactics to make it less hard.

Some Problems

When a tool becomes familiar enough, it can become forgotten. You forget a black box is a black box, and see it as a lever of reality.

What you meant is not what you said, but also, what you meant is not what you said.

I was expressing two distinct thoughts there. It wasn't very obvious. The first sentence meant, "You don't always say words that make other people think of the same things you're thinking about". The second meant, "What you mean is a different type of thing from what words you produce, and they should not be mistaken for each other".

What you meant is not what you said; the language you use is not the same thing as the cognition that produced the language. Apologies if you're getting bored.

In this sequence I'm going to make some claims about how we think and how language works, and then point to a bunch of pit-falls that can lead to confusion, never ending argument, and bad times. I believe these pitfalls have to do with inaccurate models of how language works and subtly forgetting that cognition and language are different.

(aside: what if after reading this sequence you think "Well, it's not really inaccurate models and subtly forgetting that is the cause of these problems, it's XYZ". Oh really? Why would it matter if it was the case?)

Cognition, Classification, and Inference

Your brain's job is to keep you alive. This takes a lot of guess work. Is this edible or poisons? Is that blur a tiger on its way to kill me? The usual. It sure would be useful if the brain could turn sensory input into the exact set of motor movements needed to not die. This is what the brain does. You take the outside world and make categories. Once categorized, you are use various "if... then..." rules that you previously associated with the category. If all goes well, you hear a rustle, categorize as predator, and activate "run!" This is not the only thing the brain does, but it is something the brain does.

These neural categories predates language, though they can be effected by language.

Some neural categories are loose, some are tight. Some categories immediately spawn action. The tightness, triggers, and consequences of a category are all learned through [opaque process]. Humans are basically the same considering how from more than nothing we share. Humans are nothing alike considering how far from exactly alike we are. No one will ever know your inner soul, yet you can still drive to work without being killed 100 times over in a car crash.

Remember, up to now, we have not talked about language, just the cognition of categorization. Later we talk about how language can effect how you categorize.

Sometimes, you goof. I hear a dog barking, think there's a dog around the corner, but it turns out it was just someone's ring tone. I think my new manager is going to be an ass-hole because of the face they didn't smile at me, turns out they just literally never smile and are super nice.

Pre Language seed of error [under construction]

I want to make a point about what an error in cognition feels like versus an error in modeling language.

*Ugna is relaxing under a tree when Targno appears over the hill being chased by *

[some dialoge about a caveman bringing up a shrieking pet with big teeth, Ugna say's it's "scary" Targno says it isn't]

Ugna is probably going to feel uneasy about not running. But something Ugna won't do is say, "Targno, scary by definition means something that makes me feel fear and activates my parasympathetic nervous system, this is absolutely scary!"

Ugna's mind has a quick categorization where loud shrieks get put in the "shit I should run away from". Ugna might aslo use large fangs, being huge, or moving really fast as triggers to put things in the "shit I should run away from" category. In their primitive cave language, "scary" is used to communicate "this is in the shit you should run away from category!"

For the very first time, Ugna has encountered something that shrieks loudly, has fangs, and yet isn't going to try and kill her.

It takes something special for Targno to think that Ugna is thinking about if the pet moves fast, as opposed to whether or not it can kill her.

They might still disagree, but they are disagreeing where it matters. They both really care about how to not die. There are a lot of arguments about what is scary that don't look like this.


Language is more broad than spoken and written languages like English. Take sign language as an example. Despite this, I'll mostly talk about language in terms of words, which will be me thinking of things like "catch" and "derivative".

Words are not just a 1 to 1 mapping from sounds to categories in our heads. "the" and "is" are not categories. Language has it's own logic and structure to it. Verbs and nouns, subject agreement, something something I never payed attention in grade school nor took a linguistics class.

The point: cognition has it's own internal structure. Language has it's own internal structure. Language is used to try to poke in another persons brain and produce similar patterns of cognition as you did. Telepathy exists, it's just not what you expected.

This works well, a lot of the time. This fucks up a lot of the time. A deep see marine biologist has trouble talking to a mathematician because there are some concepts that just aren't in the other persons brain (how you build new concepts in a mind from old ones?). The French don't understand the Japanese because the symbols used are completely different (there are likely also conceptual differences between the cultures). A teacher has a hard time understanding kids, because even though they use the same symbol set, kids quickly make slang. Kids grew up differently, and thus a lot of words might relate to dif cognition in subtle or gross ways.

We are all so different. We are all so similar.

Bad Theories and How they got there

Nows the fun part. How do things break? Having a bad model of what language is what leads to arguing about the definition of "scary" instead of figuring out if this thing is going to kill us.

The child thinks that words have power. When I shout "hungry" the humans bring me food. You get confused when your dog doesn't stop when you tell it to. You lack theory of mind. Words not only mean things, words have power. There is no reflection.

The student thinks that words mean something. It may be the case that you don't know what they mean, but there is a right answer to this question. You know because your teacher gives you a bad score if you can't answer it "correctly".

The Greek thinks that words have meaning, and are the same thing as thoughts. There is reflection. All words and all thoughts correspond to perfectly cut logical categories. The world is syllogisms.

The poet learns that meaning is in people, not the words. A word means whatever you want it to mean, as it gets its meaning from how it's used. You see through the illusion.

The engineer learns that some words have real meanings. If they learn these meanings deeply enough, they can remake the world in their image. It is forbidden to use the words that do not mean something, and only talk about the ones that do.

The lawyer learns that once again, words have power. Not from magic, but from social contract. If you can get the right people to accept the right words, you can force other things that you want. This is all a game. Be careful not to say the wrong words when someone is listening, they might force you to make a move you don't like.

These names themselves do not form strict boundaries. In the following examples, I'll often refer to a mistake as "poet behavior" or "student behavior". That's mostly to help you have a richer web of mnemonic connections and hopefully remember this stuff more. This post is the framework, the rest will be worked examples and more subtle points. Examples of conversation, diagnostics of what went wrong, and lessons learned. Discrete skills will be pointed out, and there will be links to practice exercises to hammer in those skills. "I don't need to practice remembering that other people have different minds". Hmmm, maybe. Hard to say.

(aside: did you disagree with the names of the groups? Do you think I think something wrong? Do you not like that I unwittingly made you think of a concept you don't actually think I'm talking about? Do you think it was irresponsible to name the groups this way given common usage? The difference is interesting.)

New to LessWrong?

New Comment
7 comments, sorted by Click to highlight new comments since: Today at 3:22 AM

Hah, I was thinking along the same lines as you—I have two pieces of advice that I give to everyone, that would solve most of their problems, at that nobody ever takes: meditate daily, and read the goddamn Sequences. So naturally, I'm gonna write up my own, heavily condensed version of the Sequences, and then my friends will have to read them because it's rude not to read something if your friend wrote it (right?)

My version will sit in my brain for at least a few more years, potentially forever, but I did take a few notes on structure at least. I thought I'd dump them here because they might be of interest to you (?) They might not be very legible, but they're for my own consumption mostly, I'm dumping them as-is just to give an idea what I found important in the Sequences (and elsewhere).


  • introduction: something to protect
  • theoretical epistemology
    • probability, evidence, betting
      • bayesian inference
        • the importance of priors
      • privileging the hypothesis
    • causality, causal relationship between reality and beliefs
      • ontology, what we mean by "real"
        • "supernatural"
    • how to use language, dissolving questions
      • predictions, expectations
  • human brains/neural nets
    • evolution, how to theorize about evolution (tooby & cosmides)
    • evopsych, the political mind
      • escaping the paradigm is impossible, but there's some slack
    • moral psychology
    • brain, neural nets
      • all the GPT-2 shit
    • predictive processing and world models, memory
    • tinted lenses
    • wanting/liking/goals/values/motivation
    • the Self, identity
    • the press secretary, introspection
    • fake explanations (elan vital)
    • fake beliefs
      • the dragon in the garage
  • mental movements/habits
    • noticing
    • noticing confusion
      • noticing "makes sense", armchair logic (e.g. 80k's reasoning about how to have impact)
    • noticing resistance to ideas (e.g. [redacted personal example])
    • noticing already knowing in advance the conclusion (in a debate, as soon as your opponent opens their mouth, you know that there's going to be something wrong with their argument)
    • noticing the pull to lash out in an argument with a loved one
    • noticing when you do motte and bailey
    • feeling certainty (e.g. private property = good, taxes and coercion = bad)
    • being stuck in a loop (e.g. trying to get out of bed in a semi-dreamy state)
    • ? suffering
    • practical epistemology (perception tinted by e.g. depression, fear, but mainly when you tie your Self to an ideology)
      • everything is a lens, there is no "lens-less" view
      • caching, normalization, prediction overrides perception ?
    • tracking debate propositions ?
    • scout mindset
  • practical topics
    • politics vs policy
      • ethics, consequentialism vs. other stances
      • goals and values vs. methods, goal factoring
      • beware grand unified theories
      • though experiments, double crux, counterfactual thinking
      • charity, compassion, empathy, we're all in this together, kumbaya
        • "how would I end up with these beliefs/behavior?"
        • addressing the cases when extending charity is already an act of desertion
      • cooperative discussion
    • adversarial optimization: Russian bots, scammers, marketing, manufactured addiction, manufactured outrage
    • social dynamics; purity spirals; meta-norms
    • commitment, identity, choosing which status game you play, leaving yourself/others a line of retreat
      • personality, character, mask, the web
    • philosophy case studies
      • chinese room, free will, teleportation/personal identity
    • footguns: "I am a rationalist, I couldn't possibly be making trivial cognitive mistakes", "all conspiracy theories are bullshit, I am smart", "I am better than others because I am more rational and smarter", "I know NVC, therefore I cannot get into dumb fights", church of Rationality; isolated demands for rigor, the fallacy fallacy
      • remember there are underlying determinants of e.g. open-mindedness etc.
  • [interesting biases]
    • status quo
    • just/non-horrible universe
    • mind projection fallacy (e.g. "the meaning of life")
  • [practical considerations]
    • ? read this book with other people, discuss

I'm planning a somewhat similar project, but it's going to be more about "Here are some good ideas I've heard". I just am sad thinking that my formulations would be lost forever if I was hit by a bus instead of written down so that if any are useful they can be stolen and used.

"Different Minds (You're Concepts Formed Differently From Mine)" should probably be "Different Minds (Your Concepts Formed Differently From Mine)."

I recommend adding header formatting to create more breaks in the essay structure. I mean like how "Bad Theories and How they got there" isn't set off from the text any.

I liked this post a lot. There are several fun bits, but I liked your categories describing how people use words. There are parts in there where I can see an underlying theory you may have used, but it's really written in your language, making it interesting and a bit fresh.


Probably typo: "affected"

> it's own

Typo for "its own" (this occurs in multiple locations)

> deep see
Should be "deep-sea"

> other persons brain
"other person's brain"

I found this part more meandering and less clear than the previous ones. It's still useful of course, but it seems to jump from topic to topic without a cohesive flow, and it's not clear if there is a central theme to follow (that's distinct from those of previous parts).

Yeah, that seems to be the main theme of the feedback I've gotten. I'm working on V2 and it's shaping up to be a lot more sensical and cohesive. Thanks for the feedback, and stay tuned!