(This is probably covering subjects already widely discussed here (haven't read LW much recently), but I wanted to get this down quickly and this seemed an appropriate place to post it.)

Here's a problem to which I don't have a solution. How do you get large numbers of people past "valleys of bad rationality" -- conveying a position other than the one they might naturally jump to?

The relevance here is to things like taking ideas seriously, which, as has been often discussed (here's a good recent post by Ozy, or here's perhaps the canonical old one), can be a bit dangerous.

This seems to be pretty hard! Raemon famously stated that you get about five words. Fortunately, I think depending on what you want, it is possible to do a bit better than this. Like, that's if you want to coordinate large numbers of people; if you don't need them to coordinate per se, I think things aren't quite that dire. Eliezer wrote the sequences, those are pretty long, but they're pretty good and a lot of us have read them!

But, there are plenty of people who have come into the rationality community (online or IRL) and have got the "5 words" version of things. A lot of people dislike LW because they don't actually know what it says and are just applying stereotypes... but it's also true that a lot of people like LW for those same mistaken reasons, which can be a pain when they show up and start arguing based on such ideas.

(Here in New York at least we seem to be doing pretty well on this front, in that we've got enough people who've actually taken the time to understand things that when someone shows up at a meetup with the stereotyped understanding we can correct them (as I've seen occur); at least, we can if they state it. But also, it does occur, rather than being completely unnecessary. I don't know how things are elsewhere!)

Note, by the way, that I say "non-stereotyped position", rather than "nuanced position". Nuanced positions are certainly difficult to convey! But the problem is even more general -- even a simple, non-nuanced, extreme positions can sometimes be difficult to get through to people if it it isn't what they're expecting (see: rounding to the nearest cliche). That said, conveying nuanced positions is a big part of the problem, so some of this will focus on that.

So why is this so hard? Well, there are a lot of reasons, but there's a few I want to discuss.

One is that groups naturally emphasize what makes them distinctive -- not the things they have in common with everyone else. When you only get a few words, conditions or restrictions on your distinctive ideas are going to get dropped. "We believe in X, but not to the extent of Y" -- yeah, just about everyone disagrees with Y, so that's naturally not going to get emphasized!

Now to some extent this can be mitigated with community, as Phil Goetz discusses -- yeah, there's the written law, sure, but then there's the oral law, y'know? This solution obviously has problems though. Firstly, if it's not publicly written down, it won't spread well. Secondly, such things likely aren't argued over properly -- they're likely just social norms rather than thought-out elaborations of the position, actual answers to the question. Such social norms can be useful for preventing disaster, but we're the sort of people who believe in taking apart social norms -- a proper solution is still needed!

One of the reasons I really like the sequences is because they do actually cover their bases pretty well -- they actually do a lot to ward off a lot of the obvious mistakes and misinterpretations one might make. (Thus the joke that a postrationalist is someone who insists they disagree with Yudkowsky without actually doing so.) There certainly are some things in the sequences I consider to be pretty wrong, but the mistakes don't really take the form of Eliezer just asserting a stereotyped incorrect position without thinking about it.

So, you can indeed put things in writing. That though raises the problem of making this writing visible, which means that we've reduced the problem compared to pure oral law, but not in any way solved it. (And even if it becomes highly visible, you're still going to bump up against "you get about 5 words".)

But another problem is that awareness of such nuances can be nonuniform and patchy above the individual level; even when things are put in writing, the failure to reach certain people may become failure to reach certain whole groups.

Or so I infer, anyway. I'm not very into EA, but I remember one EA complaining of EA's demandingness -- how it encourages one to burn oneself out, etc. Which is odd, because I'd certainly seen, prior to that, plenty of discussion to the effect of, yeah when EA started this was a big problem, but since then we've discussed this a lot and written a lot on this subject and these days we really don't encourage that sort of thing. So you'd think this wouldn't be a problem! What's going on?

Well, same things as always, really; but also, I have to infer that there must have been some EA group that somehow ended up isolated from all that and never got the message, such that its oral law failed to protect its members. Troublesome! (Or maybe it's possible that this person wasn't actually involved in such a group at all? I'm not sure, that was a while ago, so I'm going by memory and impressions here.)

And then we get to more fundamental difficulties in communication...

Hitting a target vs pushing in a direction

One thing I've said a bunch on the internet is that you should try to hit a target, not just push in a direction. (If the direction you are pushing in is simply "what is good", then sure, go ahead and just push arbitrarily in that direction. But otherwise you should not.) And I continue to think this is the right way! However, I've come to appreciate this is harder than I realized -- it's not something one just does.

There are two problems here: A problem of thought, and a problem of communication. The problem of thought is that many people just aren't really thinking about a desired end state -- they're only really thinking about the local landscape, not in any more global sense. Thus, in cases where the notional target is a distant one, they can only push in a direction. Now I'm guessing most people who are interested in rationality are not that likely to fall into this trap. And for those that are, hopefully just telling people about this problem will help them get out of it.

But the problem of communication is much harder to avoid. Language just isn't easily built for being used this way. Specifying a target can be a lot of work and require providing worked examples. A lot of language just isn't that precise, while the world itself is very detailed, and sometimes those details are relevant, and distinguishing examples of what's desired from what's not via language gets really tricky.

One big problem is that, absent a standard of measurement, lots of terms are implicitly relative to a baseline. This is obvious when it comes to words like "tall" and "short", but this applies very widely -- what is "confident", what is "aggressive", what is "annoying"... etc.

(Also, lots of things people say have an implicit "substantial[ly]" before them. Sure wish I'd realized that when I was younger!)

This baseline-dependence becomes a real problem because people use their own experience and expectations as a baseline, resulting in frequent miscommunication. So, if I say "you should take ideas seriously" -- having in mind a particular target that is somewhere in the take-ideas-seriously direction from what I consider to be the default -- someone who is already at that target will instead interpret it as pointing to some spot beyond the target I had in mind. Because why would I bother saying it, if I was just saying they should do what they're already doing? That's just baseline, they think!

Effectively, attempts to communicate targets are understood as communicating directions instead, unless substantial effort is put into prevent this! Very troublesome!

But, as stated above, I don't really have a solution to any of this. I just wanted to make some notes on the problem.