Trust-Building: The New Rationality Project

by AllAmericanBreakfast2 min read28th May 202015 comments


CommunityRationalityWorld Optimization
“I did not write half of what I saw, for I knew I would not be believed"
-Marco Polo
It's enlighteningly disturbing to see specifically how the "distrust those who disagree" heuristic descends into the madness of factions.
-Zack M Davis

Old LessWrong: we fail to reach the truth because of our cognitive biases.

New LessWrong: we fail to reach the truth because of reasonable mistrust.

What causes mistrust? It's a combination of things: miscommunication and mistakes; the presence of exploitative actors; lack of repeated trust-building interactions with the same people; and high payoffs for epistemically unvirtuous behavior. Enough mistrust can destroy our ability to converge on the truth.

Once mistrust has led us into factionalism, can we escape it?

Factionalism is not a bad thing. It means that we are willing to accept some evidence from other people, as long as it's not too divergent from our priors. Factions are worse than unity, but better than isolation.

What we want isn't a lack of factionalism, it's unity.

This suggests an activism strategy. Let's form three categories of factions:

  • Your Community: You have high trust in this network, and believe the evidence you receive from it by default. Although you don't trust everyone who calls themselves a part of this network or community, you know who belongs and who doesn't, who's reliable and who's not, and how to tell the difference when you meet someone new in the network.
  • Strangers: This network is untested, and your community doesn't have a strong opinion on it. It would take substantial work to learn how to navigate this foreign network. But because they are relatively isolated from your own community, they have evolved a different constellation of evidence. The existence of factionalism is itself evidence that you'd have something new to gain by trading information with them.
  • Enemies: This network has been examined, either by you or by your community, and labeled a toxic breeding ground of misinformation. Treating your enemies as though they were merely strangers would only alienate you from your own community. They might be exploitative or stupid, but either way, engaging with them can only make things worse. All there is to do is fight, deprogram, or ignore this lot.

To increase unity and pursue the truth, your goal is to find foreign communities, and determine whether they are friendly or dangerous.

Note that the goal is not to make more friends and get exposed to new ideas. That's a recipe for naivete. The real goal is to accurately distinguish strangers from enemies, and make introductions and facilitate sharing with only the stranger, but not the enemy. We might respect, disparage, or ignore our enemies, but we know how to tell them apart from our allies and our own:

“Here people was once used to be honourable: now they are all bad; they have kept one goodness: that they are greatest boozers.”

One of the many difficulties is the work it takes to discover, evaluate, and bring back information about new strangers, the truly foreign. Their names and ideas are most likely unknown to anybody in your community, and they speak a different language or use different conceptual frameworks.

Worse, your community is doing a steady business in its own conceptual framework. You don't need to just explain why the new people you've discovered are trustworthy; you need to explain why their way of thinking is valuable enough to justify the work of translating it into your own language and conceptual framework, or learning their language.

Luckily, you do have one thing on your side. Foreign communities usually love it when strangers express a genuine interest in absorbing their ideas and spreading them far and wide.

You might think that there is a time to explore, and a time to move toward a definite end. But this isn't so.

When there's a definite end in mind, moving toward it is the easy part.

But meaning and value mostly come from novelty.

When it feels like there's no need to explore, and all you need to do is practice your routine and enjoy what you have, the right assumption is that you are missing an opportunity. This is when exploration is most urgent. "What am I missing?" is a good question.

“You will hear it for yourselves, and it will surely fill you with wonder.”

What is our community reliably missing?