Nightmare of the Perfectly Principled

by Benquo7 min read2nd Nov 201714 comments


DefensibilityPoliticsSimulacrum Levels
Personal Blog

My actual literal nightmares about civilizational collapse somehow manage to be insanely optimistic about human nature.

I dreamt that in response to the news of the Trumps’ probable successful intimidation or bribery of their New York prosecutors, the US devolved into a lawless hellscape, since the last shreds of pretense of “we’re punishing you because it’s what the law says” were gone. In my dream, I successively wished I’d transferred more of my assets to paper, then money, then gold, then firearms, as I realized how far things had gone.

If I’d been thinking sanely, the thing I should have wished I’d accumulated is the only real source of safety in a state of war: a bigger, better gang. But fundamentally, I should have known better than to imagine that things would collapse quickly.

What was I getting wrong? I was tacitly assuming that the majority of people were perfectly principled.

The rule of law and the structure of power

The crazy thing about my nightmare was that it assumed that we had rule of law in the first place, that most of the sorts of people who vote for “law and order” politicians meant the same sort of thing I do by law. In practice, at least for decades, they’ve meant whatever “we” can get the police to enforce, to preserve a civil order that keeps the right sort of people on top.

The normal attitude towards police officers, soldiers, and other authority figures entitled to use coercive force, is to feel safe if they seem like they’re on your side, and unsafe if they seem like they’re the enemy. Police are presumed to follow the laws insofar as there is a custom to do so, and they feel that the power structure they’re embedded in wants them to; exceptions are judged on the basis of whether they locally cause harm.

But my intuitions about whether I am protected by police are quite different. The motive I imagine for following legal procedure is a transcendent commitment to following the law in full formality, because it is the law laid down in the civil code.

This explains how other people - especially other people in groups comparatively unlikely to be the targets of state violence - can read stories about cops and courts colluding to screw over some group or other and maybe feel vaguely guilty, but not personally unsafe. My intuitions are basically always that if we have a good shared set of protocols / laws, then it doesn’t matter whether we’re friends, we’ll do the right thing anyway, and if we don’t, then it’s hopeless. But these intuitions are not terribly common, and they’re not a good match for how rules are actually enforced in our society.

In particular, I’ve systematically underestimated how OK things can look to a privileged person without proper rule of law, bc predatory structures still want to allow production that can be taxed.

Law as command structure

If you’re a predatory institution or other extractive coalition, you will often want to be able to coordinate local action. To do so, you’ll need some sort of standard data structure and corresponding behavior schema. If you’re not fastidious about making statements that are literally universally true, you’ll often end up making universal statements that are locally true. In addition, if you’re trying to do this efficiently, you’ll often pick up schemas someone else left lying around, such as legal frameworks invented by systematic thinkers who were trying to be consistent. But since you’re trying to make local decisions, not report on your structural model of reality, you won’t be overly distressed by the fact that sometimes you just know that the rules don’t apply, without a formal articulation. All that matters is that you’re consistent where that is important to the performance of your coalition.

This resembles Wittgenstein’s idea of a command language, which can easily superficially resemble pure structural language. If you can get the people who believe in true rule of law to join your coalition under the misapprehension that that’s what you’re doing, so much the better.

Verbal consistency as submission to official narratives

You could draw a quadrant diagram with 2 axes. One describes the public account people give of their actions - the extent to which they try to fit into an intelligible system (with corresponding defensibility), or explicitly endorse acting based on feeling (with corresponding unaccountability). Call these Ordered and Subjective, respectively. The other describes how formal reasoning and models fit into their internal decisionmaking process - whether they have a variety of unintegrated ad hoc models, or a single, integrated model. Call these Opportunistic and Autonomous.

When I say that I was tacitly assuming everyone was perfectly principled, I mean that I was assuming that everyone was either Ordered Autonomous or Subjective Opportunistic. Basically I was assuming that all the people talking about law and order, police/courts ever not abusing their power Because Of The Law, etc are in Structured Autonomous, following the law as written because they are personally committed to following the law as written. That they have internalized the law so that they follow it autonomously. This outlook is basically Kantian.

But in fact they are Ordered Opportunistic, trying to work within the context of power structures, and taking refuge in their ordered system when challenged, but not acting based on an internalization of the full formal structure. They engage with the words and procedures because those are part of the apparatus of power dynamics, part of how things are done here, which means that they're locally consistent, but they don’t report their true underlying model of what's going on, and maybe don’t have one. If the power dynamics shift, their sense of what is lawful shifts accordingly, to adapt to the new ruling coalition. Law is just a way coalitions work together to hold onto power. Call these Collaborators.

How does this relate to the desire to pass tests?

Most people in developed countries spent much of their formative years in contexts where they were explicitly evaluated, by authority figures whom they could not physically escape, on the basis of their ability to verbalize officially endorsed narratives. For people with an autonomous disposition, this can lead to fully internalized domination; in order to recite the desired phrases, they need to alter their sole internal model of how things are, to match. But for people with an opportunistic disposition, they can simply spin up a sandboxed narrative to recite the teacher’s password, and drop the narrative when the test is over.

This helps explain part of why hedgehogs so often make poor predictors. One commonly discussed result of Philip Tetlock’s research on forecasting skill is that “hedgehogs” - people who reason based on a single integrated model - underperform “foxes”, who mix together multiple considerations.

Some people who verbally report a single explicit model have an autonomous disposition. But they’ve spent a long time in situations in which some narratives are favored for reasons other than truth-tracking. Since these narratives are not sandboxed, they have to be reconciled with the rest of their narratives, achieving consistency at the expense of predictive accuracy and explanatory power.

For opportunists reporting a single model, the situation may be even worse. Their single explicit narrative is not constrained by their need to function in the world, since they make pragmatic decisions using a different mental module entirely. So their reported beliefs track a convenient consistent worldview, but they don’t use the vast majority of their practical knowledge and life experience, and can’t change their mind when it’s not socially convenient to do so.

Defensive irony

So, what is an autonome to do?

One solution is to speak the truth despite social incentives to do otherwise. The problem with that approach is that social incentives are powerful. Most likely you end up deluded anyway - but also socially unsuccessful. However, this approach might be able to work, if you take into account the way in which you’re handicapping yourself, and take active steps to overcome this handicap. For instance, you might form a truth-seeking community with explicit understanding that you’re not seeking the approval of outsiders, and willing to forfeit the goods that would come with such approval.

Another is to simply lie. To keep two sets of narrative books - a public one and a personal one. This is cognitively expensive, but it makes it easier to see the pressures your external narrative is under. Another problem with this strategy, of course, is that you’re collaborating with the pressure on other people to conform to the official narrative, and you lose out on the chance to locate potential intellectual partners.

A third strategy is irony. The Socrates of the Platonic dialogues clearly had a specific, structured critique of Athenian discourse - but often fell back on the claim that he knew nothing. That his only wisdom was knowing how little he knew. This is the famous "Socratic irony." Nietzsche proposes a related sort of irony, in which he says that the truly creative person will often want to hide, dissemble, wear masks, that innovators will claim to merely be old-fashioned. More recently, I talked with someone who claimed that her moral philosophy was “fuzzy utilitarian” - but when I probed for details, it turned out that her problem with most attempted implementations of utilitarianism was that they themselves were too “fuzzy”. Her self-deprecating description as “fuzzy” secretly meant that she thought her thinking was more precise.

Irony occupies the Subjective Autonomous quadrant. Internally, you keep accounts. Externally, you disavow claims to lawfulness, so as to avoid attaching yourself to structures of predation.


Then there's the last quadrant, Subjective Opportunistic. This is the other honest quadrant. People following this strategy distrust external systems of accounting sufficiently to simply avoid any accounts, as they (correctly) see most lawful systems as power grabs. Unlike the Ordered Opportunistic, they don't try to claim credit for being part of a lawful system - the strategy is defensive. Shapeshifters are hard to grab a hold of.

Principles and education

Going back to my nightmare, it seems like it’s not a coincidence that Ayn Rand has the same unrealistic assumption in her fiction that people are perfectly principled, and also tacitly expected a rapid civilizational collapse, as shown by her fiction.

(Her idea of perfectly selfish people make herculean efforts to hold up their end of contracts regardless of incentives. Her perfectly selfish business owners are happy to be competed out of business by someone with a better product, and freely share their trade secrets. Her perfectly selfish philosopher accepts a humble life as a burger-flipper instead of accepting prestige from a corrupt system and thereby legitimating it. This is … not most people’s idea of perfectly selfish behavior.)

It’s also probably not a coincidence that I grew up Jewish. And that Ayn Rand also had a Jewish background. (Or that the US Supreme Court is entirely staffed by Jews and Catholics). I hadn’t noticed just how weird it is that I know lots of people - liberal atheists in liberal atheist communities - who observe some ritual laws specified thousands of years ago, despite no social pressure to do so, that don’t even reinforce current power imbalances, just because it feels right to connect with tradition.

Reverence for the rules is not totally unique, of course - Germans are famous for “but the rules......,” and Scandinavians seem like they have a similar thing going on. But German-Americans and Scandinavian-Americans don’t seem to have the thing.

But I suspect that something about growing up in a context where every Saturday we brought out a holy scroll with a story containing a law code, paraded it around the room, read from it, and sang songs about how it’s a special gift given to us by god, king of the world, it is the tree of life and all its paths are pleasant, and everyone who holds onto it is made happy, might have had something to do with my attitude towards law. And that not everyone had this experience.

And my exposure to a mixing of ritual law (including things like food taboos) and moral law (including things like charity and fair dealing) may have something to do with how I intuitively expect judges to uphold the law, not because they have a social incentive to do so, not because it accords with their self-image, not because it’s their custom and training, but because it would be unclean to do otherwise.

More precisely, the wrong assumption I made was the assumption that it is common knowledge that everybody is perfectly principled. If I aware of the existence of contingent civil order as a byproduct of power structures, I might expect it to continue for a while, and behave accordingly, even if the judges are unclean.

Engineering and the transcendent

Engineering used to seem impossibly hard to me. Permaculture did not feel impossibly hard. Neither did the tinker trade.

What’s going on?

Basically I took engineer superhero / Robinson Crusoe style stories at face value as honest accounts of what engineers can do.

In practice, engineers do what they to by relating to systems with finite bounded scope & affordances, and copy known techniques. I was assuming they had a true art (techne) in the Platonic sense. It seems impossibly hard to build a modern car or computer from first principles, since that implies knowing how to build a bronze age level civilization with many types of mining and trade.

But of course engineers use the tools they find, including know-how copied from books.

Reading The Martian recently helped me recalibrate, because it did not seem impossible, just hard. But, obviously, the engineer hero in The Martian was not creating a full-stack civilization on Mars. He was completely dependent on a static endowment of tools and materials, the technical skills he already had, and the things he could look up.

Engineers are pragmatists, like tinkers except that they can trade with an industrial base, and have access to more books. They basically don't do anything from first principles, and don't actually do whole-system maintenance. So, when I it seemed impossibly hard to do really know how to do e.g. computer programming or bridge building from scratch, the answer is that they just don’t. No one does.

And that’s OK.

But in some domains, I think we can do better. There’s a reason why the culture that produced an outsized number of science Nobelists is not an engineering culture, but a rule of law one.

Related: Geometers, Scribes, and the structure of intelligence, Be secretly wrong


13 comments, sorted by Highlighting new comments since Today at 6:52 AM
New Comment

I think you're misusing the notion of "rule of law", possibly because of the Jewish-upbringing factors you mention. Economist Don Boudreaux would argue, after Hayek, that legislation and law are different things. My version of this, heavily influenced by David D. Friedman, is that law is really a collection of Schelling points, and legislation (the statute book) is one way (of many) in which new Schelling points can be created. This is far more obvious to someone who was brought up under the common-law system — and the US's legal tradition is, after all, derived mainly from English common law, which may help to explain why the US is not now a lawless hellscape.

Thus, your "Collaborator" is not really some kind of inconsistent or irrational person, just someone who follows the law even when it conflicts with legislation. And while that can lead to 'collaboration' with an unjust regime, it can also lead to civil disobedience and other forms of non-collaboration.

So while it may be fruitful to consider where people fall on your graph, I don't think your "Collaborators" fall in the upper left quadrant; I think you place them there because you misunderstand what ordered system such people are referring to when they talk about "the rule of law".

What would it look like not to have the rule of law, on this model?

I suppose a world without law at all would be one in which people habitually defect on the Prisoner's Dilemma. Even when it's the least 'true' a PD can be and still be a PD (so, iterated, with reputational incentives, all that stuff). There are no Schelling points, and thus no coördination: Nash equilibria and Moloch for all.

The "rule of law", perhaps, is a list of particular properties of the law (collection of Schelling points): that they contain no proper nouns, for example (the same law binds the King), that they do not discriminate on irrelevant/prejudicial grounds, etc. That is to say, that the law is neither arbitrary nor capricious (I will hereonin refer to this synecdochically as: the law is just). Of course, a society's beliefs about justice will be reflected in its informal law; thus a society will typically only see itself as lacking the rule of law inasmuch as its legislation, as enforced, is contrary to the principles of its informal law.

This is in contrast with your definition, under which (if I understand correctly) a society without the rule of law is one where enforcement doesn't follow the written legislation. Of course, the patterns in this mismatch are themselves part of (my-model) law; so if everyone agrees that a piece of legislation (say, the Fugitive Slave Law) is unjust and the refusal to enforce it is just, then the society has (or believes it has) my-rule-of-law but does not have your-rule-of-law (unjust legislation is not enforced).

Conversely, if the legislation is enforced even though most people believe it's unjust, then the society has your-rule-of-law but not (it believes) my-rule-of-law (unjust legislation is enforced).

If just legislation is enforced, then the society has both kinds of rule of law (since successful enforcement of legislation makes it law). If the legislation is just but selectively enforced in a way that is unjust, then the society has neither (because the selective enforcement creates law that is unjust).

Appeals to justice, and hence arguments based on my-rule-of-law, are subjective. But I would argue (had I not spent long enough on this response already) that justice itself is objective, even if we have not yet discovered all of its principles — and thus the same is true of my-rule-of-law.

This is in contrast with your definition, under which (if I understand correctly) a society without the rule of law is one where enforcement doesn't follow the written legislation. Of course, the patterns in this mismatch are themselves part of (my-model) law; so if everyone agrees that a piece of legislation (say, the Fugitive Slave Law) is unjust and the refusal to enforce it is just, then the society has (or believes it has) my-rule-of-law but does not have your-rule-of-law (unjust legislation is not enforced).

In this hypothetical world where there is a shared understanding that something like the Fugitive Slave Law is unjust and ought not to be enforced, but for some reason the people sharing this understanding, can't be bothered to rewrite the law, what's going on? Why are there written laws at all in that world? What's their role in coordination?

I'm not saying correspondence between written and enforced law is identical with justice; I'm saying that if there are written laws, and they do not correspond to the law as enforced, then this is symptomatic of a lack of rule of law. In particular, it implies that the written law is meant to misinform, most likely in order to prevent the true norms from becoming common knowledge, which only makes sense if the enforcers intend to apply official standards unevenly in order to benefit from the information asymmetries they set up.

So based on your contrast between engineering culture and rule of law (ROL) culture, it seems to me that you're claiming a connection between ROL culture and a tendency to emphasize first principles. While I don't doubt your account of how this played out for you (i.e. that growing up in an ROL culture caused you to place lots of value on first principles), it seems to me that these two things are at least partially in tension. As a child, I always hated arbitrary rules. My general response to teachers'/parents' appeals to authority was some variant of, "I'll do what you tell me to do, if you can convince me." I mention this not to say that I was in the right, but to point out that following the rules laid out by outside authorities often means ignoring one's own internal rules, and following one's internal rules means breaking/ignoring the rules laid out by outside authorities. That is, deriving results for oneself, from first principles, often means breaking the (external) rules, which is anathema to an ROL culture as I'm understanding it (if I'm misunderstanding ROL culture, or what you mean by first principles, please let me know).

I think you gesture at this at the beginning of your description of "Defensive Irony":

So, what is an autonome to do?
One solution is to speak the truth despite social incentives to do otherwise. The problem with that approach is that social incentives are powerful.

However, I think this is much more common than you seem to be giving it credit for. I don't think most people who are on the "Autonomous" end of the spectrum end up like Socrates, I think they end up like (my mental caricature of) Marx, trying to replace what they see as a hopelessly misguided system with one actually derived from first principles. I think your description of a Kantian who adapts their internal rule system to match the external rules is insightful, and probably correct in a lot of cases, but I don't think that most autonomes who don't end up like that try to hide their internal rule systems, but rather end up in a clash with the external culture.

The alternate (very likely) interpretation of my intuition is that I'm massively falling victim to availability heuristic and generalizing-from-one-example. But my main point isn't necessarily about frequency, but rather this: if one is inclined to first-principles style thinking, that can lead to a clash with an ROL culture if one's own first principles don't match the first principles of the ROL culture itself. (Though I can see how an ROL culture could inculcate first-principles style thinking, and I can see how, if one's first-principles style thinking is due to said ROL culture, the likelihood of a clash is greatly reduced because in all likelihood one's own first principles simply are the principles of the ROL culture itself).

I recently had the opportunity to compare notes with someone from a strong Calvinist background. The Calvinists also have a strong culture of "obey all the rules", but one with an ethos of "lawfulness is submission to rightful authority" rather than "law is the tree of life". Their parents and teachers had an ethos of "don't talk back," very different from my Jewish upbringing where, even among the Orthodox (which we were not), while you were expected to eventually come to the endorsed answer, and were expected to follow the object-level rules in the meantime, it would be considered wrong not to ask questions until you understand.

The Jewish way is of course still highly imperfect, because the system as a whole lacks the appropriate error-correction mechanism for cases where the objection really is valid. Lacking a proper line of retreat short of abandoning the community altogether, there's strong social pressure not to ask some of the most important questions. But it's different in an important way from systems that treat argument per se as disobedience.

I agree that in the absence of this error-correction mechanism autonomes still end up clashing with the culture. Consequently, I think that a full ROL culture would have to have social structures for error-correction. Taking another Jewish example mainly because it's been on my mind lately, the protected role of prophets during and before the era of the First Temple seems like it served something like this purpose - prophets' main role was to criticize, not to predict. And they criticized kings for things that are only violations of orthodoxy in the hindsight of a tradition that made some of their criticisms canonical. Quaker meetings where everyone has full standing to say anything the spirit moves them to say seem like a different and promising sort of error-correction mechanism. Courts of law or other formal venues for petition are another.

There’s a reason why the culture that produced an outsized number of science Nobelists is not an engineering culture, but a rule of law one.

What do you mean by an engineering culture, and how is it distinct from the "rule of law" culture you described earlier?

Based on this quote:

Engineers are pragmatists, like tinkers except that they can trade with an industrial base, and have access to more books. They basically don't do anything from first principles, and don't actually do whole-system maintenance.

it seems to me that "engineering culture" is something like "don't worry about the rules, just do whatever works." Think "bodging". This contrasts with the rule of law culture, which is something like "in conflicts between doing what works and following the rules, always follow the rules."

Maybe more precisely, don't worry about elegant, rationalized rules.

Another type of feedback loop that can change the rules is the retrospective. So you follow the rules until a fixed point and then look back at the rules and see what had a good or bad impact and change the rules.

I have a hypothesis I would like to test around ordered thinking.

On topic, I think it worth remembering: Whatever order we have found in the world, it has only happened via a protean process (evolution)

So the graph is effectively: Internally Ordered vs Internally Unordered and Externally Ordered vs Externally Unordered, where ordered means single system. Are there any issues if I remember it though way?

That doesn't seem wrong to me - one diagonal represents accurate correspondence, the other represents dissembling. I picked different words on the different axes mainly because I thought it would be harder to talk compactly if I used the same words on both axes.