I want to discuss a particular failure mode of communication and thinking in general. I think it affects our thinking about AI Alignment too.

Communication. A person has a vague, but useful idea (P). This idea is applicable on one level of the problem. It sounds similar to another idea (T), applicable on a very different level of the problem. Because of the similarity nobody can understand the difference between (P) and (T). People end up overestimating the vagueness of (P) and not considering it. Because people aren't used to mapping ideas to "levels" of a problem. Information that has to give more clarity (P is similar to T) ends up creating more confusion. I think this is irrational, it's a failure of dealing with information.

Thinking in general. A person has a specific idea (T) applicable on one level of a problem. The person doesn't try to apply a version of this idea on a different level. Because (1) she isn't used to it (2) she considers only very specific ideas, but she can't come up with a specific idea for other levels. I think this is irrational: rationalists shouldn't shy away from vague ideas and evidence. It's a predictable way to lose.

A comical example of this effect:

• A: I got an idea. We should cook our food in the oven. Using the oven itself. I haven't figured out all the details yet, but...
• B: We already do this. We put the food in the oven. Then we explode the oven. You can't get more "itself" than this.
• A: I have something else on my mind. Maybe we should touch the oven in multiple places or something. It may turn it on.
• B: I don't want to blow up with the oven!
• A: We shouldn't explode the oven at all.
• B: But how does the food get cooked?
• A: I don't know the exact way it happens... but I guess it gets heated.
• B: Heated but not exploded? Sounds like a distinction without a difference. Come back when you have a more specific idea.
• A: But we have only 2 ovens left, we can't keep exploding them! We have to try something else!

B can't understand A, because B thinks about the problem on the level of "chemical reactions". On that level it doesn't matter what heats the food, so it's hard to tell the difference between exploding the oven and using the oven in other ways.

Bad news is that "taboo technique" (replacing a concept with its components: "unpacking" a concept) may fail to help. Because A doesn't know the exact way to turn on the oven or the exact way the oven heats the food. Her idea is very useful if you try it, but it doesn't come with a set of specific steps.

And the worst thing is that A may not be there in the first place. There may be no one around to even bother you to try to use your oven differently.

I think rationality doesn't have a general cure for this, but this may actually be one of the most important problems of human reasoning. I think the entire human knowledge is diseased with this. Our knowledge is worse than swiss cheese and we don't even try to fill the gaps.

Any good idea that was misunderstood and forgotten - was forgotten because of this. Any good argument that was ignored and ridiculed - was ignored because of this. It all got lost in the gaps.

Metrics

I think one method to resolve misunderstanding is to add some metrics for comparing ideas. Then talk about something akin to probability distributions over those metrics. A could say:

""Instruments have parts with different functions. Those functions are not the same, even though they may intersect and be formulated in terms of each other:

1. Some parts create the effect of the instrument. E.g. the head of a hammer when it smashes a nail.
2. Some parts control the effect of the instrument. E.g. the handle of a hammer when a human aims it at a nail.

In practice, some parts of the instrument realize both functions. E.g. the handle of a hammer actually allows you not only to control the hammer, but also to speed up the hammer more effectively.

When we blow up the oven, we use 99% of the first function of the oven. But I believe we can use 80% of the second function and 20% of the first.""

Complicated Ideas

Let's explore some ideas to learn to attach ideas to "levels" of a problem and seek "gaps". "(gap)" means that the author didn't consider/didn't write about that idea.

Two of those ideas are from math. Maybe I shouldn't have used them as examples, but I wanted to give diverse examples.

(1) "Expected Creative Surprises" by Eliezer Yudkowsky. There are two types of predictability:

1. Predictability of a process.
2. Predictability of its final outcomes.

Sometimes they're the same thing. But sometimes you have:

• An unpredictable process with predictable final outcomes. E.g. when you play chess against a computer: you don't know what the computer will do to you, but you know that you will lose.
• (gap) A predictable process with unpredictable final outcomes. E.g. if you don't have enough memory to remember all past actions of the predictable process. But the final outcome is created by those past actions.

(2) "Belief in Belief" by Eliezer Yudkowsky. Beliefs exist on three levels:

1. Verbal level.
2. First "muscle memory" level. Your anticipations of direct experiences.
3. Second "muscle memory" level. Your reactions to your own beliefs.

Sometimes a belief exists on all those levels and contents of the belief are the same on all levels. But sometimes you get more interesting types of beliefs, for example:

• A person says that "the sky is green". But the person behaves as if the sky is blue. But the person instinctively defends the belief "the sky is green".
• Not verbally formulated "muscle memory" belief. Some intuition you didn't think to describe or can't describe.
• (gap) Slowly forming "muscle memory" belief created by your muscle reactions to other beliefs. Some intuition/preference that only started to form, but for now exists mainly as a reaction to other intuitions and preferences.

(3) "The Real Butterfly Effect", explained by Sabine Hossenfelder. There're two ways in which consequences of an event spread:

1. A small event affects more and more things with time.
2. Event on a small scale affects larger and larger scale events.

In a way it's kind of the same thing. But in a way it's not:

• One Butterfly Effect means sensitivity to small events (butterflies).
• Another Butterfly Effect says that there's an infinity of smaller and smaller events (butterflies). And even if you account for them all you have a time limit for prediction.

(4) "P=NP, relativisation, and multiple choice exams", Baker-Gill-Solovay theorem explained by Terence Tao. There are two dodgy things:

1. Cheating.
2. Simulation of cheating.

Sometimes they are "the same thing", sometimes they are not.

(5) "Free Will and consciousness experience are a special type of illusion." An idea of Daniel Dennett. There are 2 types of illusions:

1. Illusions which are complete lies that don't correspond to anything real. E.g. a mirage in a desert.
2. Illusions that simplify complicated reality. E.g. when you close a program by clicking on it with the arrow: the arrow didn't really stop the program (even though it kind of did), it's a drastic simplification of what actually happened (rapid execution of thousands lines of code).

Conscious experience is an illusion of the second type, Dennett says. I don't agree, but I like the idea and think it's very important.

Somewhat similar to Fictionalism: there are lies and there are "truths of fiction", "useful lies". Mathematical facts may be the same type of facts as "Macbeth is insane/Macbeth dies".

(6) "Tlön, Uqbar, Orbis Tertius" by Jorge Luis Borges. A language has two functions:

1. First function focuses on describing objects.
2. Second function focuses on describing properties of objects.

Different languages can have different focus on those functions:

• Many human languages focus on both functions equally (fifty-fifty).
• Fictional languages of Borges focus 100% on properties. Objects don't exist/there's way too much particular objects.
• (gap) Synesthesia-like "languages". They focus 80% on properties and 20% on objects.

I think there's an important gap in Borges's ideas: Borges doesn't consider a language with extremely strong, but not absolute emphasis on the second function. Borges criticizes his languages, but doesn't steelman them.

(7) "Pierre Menard, Author of the Quixote" by Jorge Luis Borges. There are 3 ways to copy a text:

1. You can copy the text.
2. You can copy the action of writing the text.
3. You can copy the thoughts behind the text.
4. You can change the text. ("anti-option")

Pierre Menard wants to copy 1% of the 1 and 98% of the 2 and 1% of the 3: Pierre Menard wants to imagine exactly the same text but with completely different thoughts behind the words.

("gap") Pierre Menard also could try to go for 100% of 3 and for "anti 99%" of 4: try to write a completely new text by experiencing the same thoughts and urges that created the old one.