Speaking Truth to Power Is a Schelling Point

by Zack_M_Davis2 min read30th Dec 201917 comments


PoliticsHonestyGame TheoryCensorship

Consider a coalition that wants to build accurate shared world-models (maps that reflect the territory), and then use those models to inform decisions that achieve the coalition's goals.

However, suppose that some ways of improving models are punished by the surrounding Society. For example, if the Emperor's new clothes turn out to be "vaporwear", agents who notice this might not want to make it common knowledge within their coalition by adding it to the coalition's shared map, because if that knowledge "leaks" during the onerous process of applying for a grant from the Imperial Endowment for the Arts and Sciences, then the grant application will be more likely to be rejected: the Emperor's men don't want to fund coalitions who they can detect believe "negative" things about the Emperor, because those coalitions are more likely to be disloyal to the regime.

(Because while everyone has an interest in true beliefs, disloyal subjects have a unusually large interest in selectively seeking out information that could be used against the regime during a revolution. ("The corrupt false Emperor is wasting your tax money on finery that doesn't even exist! Will you join in our crusade?") That makes even true negative beliefs about the Emperor become a signal of disloyalty, which in turn gives loyal subjects an incentive to avoid learning anything negative about the Emperor in order to credibly signal their loyalty.)

Coalitions need to model the world in order to achieve their goals, but grant money is useful, too. This scenario suggests coalition members working on their shared maps might follow a strategy schema that could be summarized in slogan form as—

Speak the truth, even if your voice tremblesunless adding that truth to our map would make it x% harder for our coalition to compete for Imperial grant money, in which case, obfuscate, play dumb, stonewall, rationalize, report dishonestly, filter evidence, violate Gricean maxims, lie by omission, gerrymander the relevant category boundaries, &c.

(But outright lying is out of the question, because that would be contrary to the moral law.)

Then the coalition faces a choice of the exact value of x. Smaller values of x correspond to a more intellectually dishonest strategy, requiring only a small inconvenience before resorting to obfuscatory tactics. Larger values of x correspond to more intellectual honesty: in the limit as x → ∞, we just get, "Speak the truth, even if your voice trembles (full stop)."

Which choice of x looks best is going to depend on the coalition's current beliefs: coalition members can only deliberate on the optimal trade-off between map accuracy and money using their current map, rather than something else.

But as the immortal Scott Alexander explains, situations in which choices about the current value of a parameter, alter the process that makes future choices about that same parameter, are prone to a "slippery slope" effect: Gandhi isn't a murderer, but may quickly become one if he's willing to accept a bribe to take a pill that makes him both more violent and less averse to taking more such pills.

The slide down a slippery slope tends to stop at "sticky" Schelling points: choices that, for whatever reason, are unusually salient in a way that makes them a natural focal point for mutual expectations, an answer different agents (or the same agent at different times) might give to the infinitely recursive question, "What would I do if I were her, wondering what she would do if she were me, wondering what ...?"

In the absence of distinguished salient intermediate points along the uniformly continuous trade-off between maximally accurate world-models and sucking up to the Emperor, the only Schelling points are x = ∞ (tell the truth, the whole truth, and nothing but the truth) and x = 0 (do everything short of outright lying to win grants). In this model, the tension between these two "attractors" for coordination may tend to promote coalitional schisms.