Last night I read the post Working With Monsters. My response? "Holy shit. This post is amazing and I needed to hear it."

The idea in the post is that you have to be able to work with people who you find morally abhorent. You can't just start a war every time someone believes in a god you don't like. Oh wait... I guess you can.

But you shouldn't! And even before reading the post, I had that stance. If you asked me, I'd have said "of course". So why is it that I found Working With Monsters to be influential?

Well, even though I believed it before, I still felt something "snap into place" after reading the post.

Think of it like this. My mind is composed of subagents. And so is yours.

It's like there are different people living inside your head. In the above image from Wait But Why, the author identifieds the Higher Mind and the Primitive Mind. In reality there are many more "people"/subagents than that living in your head, but let's go with this for now.

When I felt it "snap into place", I think what happened was that my Primitive Mind finally "got it". Higher Mind already understood it, but Primitive Mind did not.

What do I mean by "got it"? I have memory reconsolidation in mind. Here's a hand wavvy description of how that works.

1. When you initially form a memory/belief, like "you're a terrible person if you work with monsters", it "solidifies". It becomes "locked in". It consolidates.
2. If you want to update it, you have to first "unlock it". You do this by really zeroing in on it. Making it extremely salient. "Re-living it", if you will.
3. Once the memory is unlocked, it is available to be updated. In order to update it, you have to basically convince the subagent, eg. Primitive Mind, that the initial memory is false and instead something else is true. Picture a lawyer making an argument to a judge.
4. After the memory is updated, it becomes "locked in" again. It reconsolidates. Now if you wanted to update it a second time, you'd have unlock it again.

Let's get back to the blog post Working With Monsters. I think that what happened to me last night is that the blog post successfully performed steps two and three on my Primitive Mind subagent. For step two, it painted a picture that made things salient enough for me. For step three, despite being fictional, I felt like it still implicitly was making an argument, and I found that argument to be highly convincing.

More generally, I suspect that a lot of good writing has to do the same thing: perform steps two and three successfully.

There are of course exceptions. Sometimes good writing triggers you to form a new belief instead of updating an existing one, for example. And even if you successfully trigger reconsolidation, you still want it to be about something important. If a blog post made me realize that cheerios taste more nutty than carb-y, that isn't an important enough thing. Robin Hanson has a post Write To Say Stuff Worth Knowing, and I don't think the point about cheerios meets that bar.

But don't let these caveats detract from the main point. I suspect that a lot of good writing is good in large part because it triggers memory reconsolidation. And if so, writers can take advantage of this.

How? Probably by spending more time on step two. I'm not sure what techniques/patterns would work well for step two though. Would it be enough to just say, "let's review the arguments for this belief that a lot of you probably believe right now"? Or maybe you need to take things further and come up with examples? Would they need to be compelling, real world examples? Or maybe examples aren't enough, and you'd need to tell stories in order to make things salient enough? I'm not sure. It probably depends a lot on the situation.

Here's another thought. Scott Alexander has a post called Non-Expert Explanation where he argues that it is helpful to have lots of different people take a stab at explaining something rather than everyone reading the same post by some expert. Because sometimes that post by the expert doesn't do it for you. You need to hear it from a different perspective in order for it to "click". I'm not sure, but I weakly suspect that the reason for this is related to the stuff I'm talking about here with memory reconsolidation. In particular, I expect that there's a decent amount more individual differences in step two than in step three. Activation vs updating. I think a convincing argument is going to be convincing to a wide variety of people, but for the part about unlocking the old memory, I expect that to depend more on the person.