Nobody likes rules. Rules are costly: first, they constrict the space of available actions or force you to expend resources to do something. Second, rules are costly to follow: you need to pay attention and remember all relevant rules and calculate all ways they interact. Third, in real life, rules aren't simple! After you left area of "don't kill", every rule has ambiguities and grey areas and strict dependency on judgement of enforcing authority.
If everybody was good and smart, we wouldn't need rules. We would just publish "hey, lead is toxic, don't put it into dishes" and everybody would just stop using lead. After that, even if somebody continued using lead, everybody would just ask and conduct analysis and stop buying lead-tainted commodities and everybody still using lead would go bankrupt.
Conversely, if everybody was good and smart, we wouldn't need authorities! Everybody would just do what's best.
You don't need to be utility minimizer to do the damage through rules. You need just to be the sort of person who likes to argue over rules to paralyze functioning of almost every group. Like, 95% of invocations of authority outside of legal sphere can be described as "I decided to stop this argument about rules, so I stop it". Heck, even Supreme Court functions mostly in this way.
There are different societies. In broad society, whose goal is just "live and let live", sure, you can go for simple univerally-enforceable rules. In inclusive parts of society, like public libraries and parks and streets - same. it doesn't work for everything else. Like, there can't be comprehensive rules why you can('t) be fired from startup. There are CEOs and HRs and they make judgements about how productive you are et cetera and if their judgement is unfavorable, you get fired. Sure, there is a labor law, but your expences (including reputational) on trying to stay are probably going to be much higher than whatever you can hope to get. There are some countries where it's very hard to be fired, but such countries also don't have rich startup culture.
Nobody likes rules that are excessive or poorly chosen, or bad application of rules. I like rules that do things like[1]:
not a complete list ↩︎
For each of these, the answer about whether the rule is good depends on what margins we're talking about.
For example, lets take your first point, rules that "prohibit others from doing things that would harm me". One way in which you could be harmed is by someone selling a drug to you which has negative side-effects which out-weigh the positive side-effects. Therefore should we ban the selling of those drugs?
I think we shouldn't. Not only do people have a right to put whatever they want in their body which this infringes upon, but the cost of actually following this rule is much higher than the benefit of being able to not (as the seller, selling to many) worry about whether there's a 1% chance this particular customer regrets their purchase.
You may not agree with the sign of that particular example, however, generally speaking, there are costs to following rules, outside of poorly chosen rules or bad application of rules. If the benefit to your rule is less than the cost of following that rule, then no matter how well the rule is chosen or how benevolent the application, its a bad rule!
And note that as the number of rules grows, the cost of following all of them does too (sometimes super-linearly, as rules can interact), while the benefit of the marginal rule decreases. Therefore there's an optimal number of rules, and we should expect that on average adding a new rule is just bad[1].
I will also note that this point was made in the comment you were responding to
Second, rules are costly to follow: you need to pay attention and remember all relevant rules and calculate all ways they interact.
Assuming we in a rule-optimal society, I think few would argue we make too few rules in general. ↩︎
Nobody likes rules that are excessive or poorly chosen, or bad application of rules.
Yes, but for most people, in practice a "bad application" of rules means "the rules being applied to me."[1]
That's the primary sense in which people don't like rules.
When I feel like I have an excuse for why I should get special treatment, even though I wouldn't say this excuse was reasonable if someone else tried to use it
This seems a bit tautological… since roughly half the population is below average in virtue, and will engage in all sorts of bad behavior if they think they can get away with it. Partly because we define good and bad relative to the population average.
And for most of the rest, strong enough incentives can induce them to behave the same, it happens even on this very forum, so when combined that’s most people already.
I disagree that nobody likes rules.
Your comment itself gives a counter-example, when it refers to "the sort of person who likes to argue over rules". Generally this sort of person likes rules. The post you are responding to is another counter-example, Zach writes that "Rules are a critical social technology for helping people live and work together in peace", which sounds like the words of someone who likes rules. Another counter-example is that I like rules. I am not a rule-maximizer, I don't like all rules in all contexts, and I have other values, but in general I like rules more than my peers.
We see in various aspects of leisure, where people are less constrained by instrumental concerns, that some people like rules. There are "rules-heavy" and "rules-light" RPGs, and there are people who prefer each. Some forms of music and dance and other art have more rules than others. Some prefer poems that follow rules of rhyme and meter, others prefer free verse poetry. Some prefer complex sports, others prefer to play tag.
I disagree that if everybody was good and smart, we wouldn't need rules. When good and smart people play chess with each other, they still follow the rules of chess. When good and smart people trade with each other, they still follow property rules. In a world where only good and smart people drive, they would be faster and safer if there were still rules for what side of the road to drive on, who has right of way, how to overtake, etc. In the recently viral red/blue pill hypothetical, where blue pill takers die if less than 50% of people take the blue pill, the risk of death is lower if there is a rule about what pill to take.
One reason not to have rules in the real world is the enforcement cost. In a world where everyone is good, that cost is lower, so we might have more rules. On the other hand, one reason to have rules in the real world is that people make bad decisions. In a world where everyone is smart, there are fewer bad decisions, so we might have fewer rules. And in a world where everyone is good and smart, everyone would agree with me that $SYSTEM is the best way to run the world, haha, which has $IMPLICATIONS for the number of rules. But I don't think that means no rules.
There are “rules-heavy” and “rules-light” RPGs, and there are people who prefer each.
Hmm, I think that this is not a good example. I like “rules-heavy” RPGs (indeed, I think that designers and fans of “rules-light” RPGs are very often thoroughly confused about what the consequences of “rules-lightness” actually are). But the “rules” of an RPG and the kinds of “rules” that I think @quetzal_rainbow’s comment is talking about are not really the same thing.
The “rules” of an RPG are mostly not constraints that prevent players from doing things, but rather mechanisms that enable players to do things. (For example, the 3rd edition of D&D has rules for crafting magic items; the 5th edition of D&D has no such rules. The effect of this is not that players in 5e are more free and less constrained than players in 3e. The effect, rather, is that players in 3e can do something that players in 5e generally cannot.)
Of course there are some rules in an RPG that are mostly constraints… but, well, those are also the rules that people are least likely to like. (Although they may be necessary in order for the game as a whole to be enjoyable! In this, they mirror the sorts of rules we’re most likely to encounter in real life.)
I'm happy with the example of rules-heavy/crunchy RPGs as evidence that some people like rules, but I agree it is not a good evidence for whether people like constraints. I don't know what @quetzal_rainbow might have meant by "nobody likes rules", other than the surface meaning. Maybe they will clarify. I wouldn't have replied to a statement like "nobody likes to be constrained by complex rules that are costly to follow and have no benefits", for example, but that's not what they said.
I completely agree that rules can be mechanisms that enable people to do things. For example, many social partner dances have rules about how the dancers move and dance together. At first glance these look like pure constraints. However, they are also enabling mechanisms for joint movements that otherwise would not work (at all, as well, as easily, etc.) in partnered freestyle. This is one of the things I like about rules.
Continually making new rules has downsides, both in administration attention and also in negative impacts for bystanders. I think those downsides outweigh the benefits of just making new rules in many cases.
I wrote a bit about this in Pick Two: Clear, Concise, or Comprehensive. It's often the case that trying to specifically rule out undesired behavior takes a lot of exasperating, detailed work or is going to involve reserving the ability to make broad judgement calls. For instance, imagine you don't want people calling each other morons, so you make a rule saying don't do that. Ten minutes later Bella calls Carl an idiot, so you make a rule not to call people idiots either. Five minutes after that, Carl calls Bella a numbskull, so you make a rule not to call people numbskulls. One minute later, Bella calls Carl a dumbass. They can obviously keep coming up with new rude things to call each other, and eventually the number of banned insults will impede normal communication.
Imagine Carl calls Bella a child, you ban calling people children because this has been going on for weeks and you're a little bit on autopilot, and then the next day Dean posts a picture of his family where he says "Here's my husband Evan and our children Frankie and George!" You, in your bountiful wisdom, let this slide, and then later Carl starts arguing that moderation is totally inconsistent, he got in trouble for calling someone a child but for some reason it's fine when Dean does it.
Okay, so maybe the rule is "no insulting people."
Well, you've given up having a comprehensive list of insults in favour of having more concise rules. You're now going to have to make moderation judgments about what counts as an insult.
The work of coming up with rules to ensure socially beneficial outcomes can be frustrating, because you won't always get the rules exactly right the first time. You might need to iterate. But it's a finite and achievable amount of work, not an unwinnable unending battle against the formidable intelligence of an adversary who hates everything your Society stands for, because those mostly aren't a real thing either.
Okay, so I want to point out that sometimes "a battle against an adversary who hates everything your Society stands for" is a passable description of what's actually going on. Like, take a look at the relationship between Hamas and the IDF for a second. I'm not saying one side of that is an evil mutant who hates all that is good in the world, but if I was in charge of writing rules (for a ceasefire, for rules of warfare, etc) I would actually be thinking about how someone might argue loopholes in whatever I came up with. That's an extreme example, sure, but if I imagine spending a year in a conflict like that, trying to write down specifically what I did not want people to do, I can imagine eventually winding up with rules that amounted to "stay out, don't send anything to the other side, don't even think about it." Either that or I'd wind up with a book of rules too big to physically print in one book.
And there's things that would break a rule ("send normal water to the other side via this river") which I might still want to happen. I can try and make carveouts and exceptions and more detailed rules, but the thing I really want is for whatever inner generator of ideas and actions they're using to be a bit more in tune with my preferences for the space. There's violations that are fine if they happen once or twice and not if they happen all the time[1], there's violations that are fine because it was done well but are bad if they're done badly[2], there's violations that might just get a warning if it's a small amount but not if it's a big amount[3] and trying to write that down as a rule in advance rather than leaving it up to moderator judgement sounds . . . maybe still finite and achievable, but it's finite and achievable for the U.S. government, not random forum administrators.
(Remember, if you want it in advance then you have to write it down for every situation that might come up. If you don't need to have it in advance, then someone can still get punished for breaking a rule they didn't know about.)
For spaces I don't need to be in (e.g. my local YMCA, or my favourite internet forum) I prefer the administration have a more high level strategy than just making a new rule whenever someone does something undesirable.
Don't constantly ask for money. If once or twice a year when hanging out you go 'shoot, I forgot my wallet, do you mind covering me?' this is probably fine. If you never cover your bar tab then people will likely stop going out drinking with you.
Don't hit on people badly. If you walk up to someone for the first time and say 'I'm Ted, I want to have sex with you.' then this will not go well. If after a couple weeks of hanging out and enjoying each other's company you say 'hey, I'd like to go on a date with you sometime. Would you like to see a movie next week?' this is probably fine.
Compare "Man, I feel like you're being a little dense here" vs "you fucking imbecilic waste of air." Both insults! Or compare a light punch to the shoulder vs a full power haymaker to the temple. Both are punches! But the former might well correctly get off with an informal warning, while the later would be a pretty strong argument for a ban in many venues.
Who are these "people who think rules are unworkable and want to empower an Authority to make judgement calls controlling the minute details of everyone's behaviour"?
I don't think I've knowingly ever encountered such a person, and I'm trying to work out whether I just move in different circles from Zack, or whether I'm failing to see something real, or whether Zack is seeing something that isn't real.
(The first hypothesis seems the most likely, but whichever is correct a few concrete examples seem like our best bet for making one of us less wrong.)
(Those aren't literally all the hypotheses; e.g., it could be that Zack and I are seeing compatible things but Zack's describing them in a way I'm understanding differently. E.g., perhaps Zack's talking about people whose position I'd express as "sometimes the rules need to allow for an element of human judgement" -- cf. Nick Tarleton's comment elsewhere in the thread -- and arguing for a position I'd express as "a society's laws need to be 100% explicitly nailed down with no room for manoeuvre". Again, concrete examples will help clarify.)
Who are these “people who think rules are unworkable and want to empower an Authority to make judgement calls controlling the minute details of everyone’s behaviour”?
That would be the mods of Less Wrong.
(I'm not sure whether that's a serious answer or snark, but I'll treat it as the former since I don't think I have anything much to say to the latter.)
If that's what Zack means, then it seems odd that the OP puts so much emphasis on how rules can be an effective way to keep people safer from literal death, brain damage, and the like.
The rules are just there to stop ourselves from trying to kill each other when your freedom and dignity is getting in the way of my freedom and dignity, so that we can focus on creating Value instead of wasting effort trying to kill each other. [...] Traffic laws make it clear to everyone when it's safe to enter the road. If everyone just entered the road whenever they felt like it, that would be dangerous [...] Lead paint is an environmental hazard, so it was banned in 1978. Because of the ban, paint manufacturers stopped making lead paint. [...] So paint manufacturers still ended up using mercury in some paints until 1991 when that was banned, too. But once it was banned, they stopped.
Zack did mention one example of a rule that was about something less life-and-death-y, namely income tax -- but that was an example of where nice clear rules aren't so effective.
So I think either Zack wasn't primarily taking aim at the LW moderators or his choices of examples are systematically ill-fitted to what they're meant to be illustrating.
The post is about the philosophy of rulemaking. In order to ground my thinking about the topic, I need to mention some concrete examples: I think it would be a much worse post if I just talked about rules in general without considering any specific examples of rules. But I don't think the philosophical substance is particularly affected by how "life-or-deathy" the examples were, and I don't understand why you think it would be. (If for some reason I wrote a blog post about how , the claim would hold whether or .) Can you explain why you think the life-or-deathness is relevant? (As it happens, I had considered using a "noise pollution" example about people who like to throw parties before ultimately going with the lead paint example. I'm having trouble reconstructing from introspection and memory why I went with the paint rather than the parties, but I don't think it matters much.)
Now, it's also true that the reason I was thinking about these aspects of the philosophy of rulemaking recently is because it had been relevant to a moderation dispute on this website. To spell it out, I think that harms from commenters who hurt other users' feelings can be successfully mitigated by rules, because such commenters are just trying to make what they see as good comments rather than being sadistic emotional damage maximizers, similarly to how paint manufacturers are just trying to make high-quality paint rather than being environmental lead maximizers.
The fact that that's why I was thinking about the philosophy of rulemaking is not a secret. You're allowed to notice that, and I'm happy to spell it out in the comments. But I dispute that that's what the post is "really" about (at least for some relevant notion of "really").
(Apologies for the very slow response.)
Well, the post is claiming something along the lines of "our rules shouldn't try to do too much, and if they don't then it isn't impractical to make them all explicit and leave little to individuals' judgement", but ...
... if it's claiming that this is universally true, which seems like a strong and counterintuitive claim, then I don't think it's made much of an attempt to make its case ...
... but if it's claiming only that it's true in some particular set of contexts then your choice of examples is highly relevant, because the examples give some indication of what that particular set of contexts is, and because the plausibility of your argument leans on the plausibility of the examples.
So I think I could buy "this post is a very general and philosophical one so what examples I use doesn't matter much", or I could buy "this post gives some reason to accept the claims it makes", but I don't think I buy them together.
I'd been assuming we were in the second case, but from what you now say it seems like maybe we're in the first, so let me explain why I find the arguments unconvincing if they are meant to be so general.
Perhaps your point isn't that you were making an argument so broad that the details of the examples don't matter, but only that the examples were so appropriate to the particular cases you had in mind. And, in particular, that the life-and-death-ness of the examples isn't particularly relevant. -- But it seems quite relevant to me, because a key part of your argument is that rules should be narrowly scoped: in your words, they are "just there to stop ourselves from trying to kill each other when your freedom and dignity is getting in the way of my freedom and dignity". Now, to be sure, "kill" might be understood in some metaphorical way. But when you say "the rules are just there to stop ourselves from trying to kill each other" and present a bunch of examples in which the rules are literally there to stop people dying, along with one example in which they aren't and which you explicitly say isn't an example of what you're pointing at ... well, I don't think you can be too surprised if someone draws the conclusion that your arguments were aimed at life-and-death cases.
And that life-or-death-ness is, it seems to me, relevant. Because, as I said above, it becomes more plausible that rules-with-sanctions should be narrow in scope and precisely defined when the consequences of breaking them are more severe. Laws whose violation can get you locked up or executed? Yup, it sure seems important that we not have more of those than we need and that you should be able to tell what will and won't make those things happen to you. The moderation rules of an internet forum, where the absolute worst thing that can happen is that you can no longer post or comment there? Maybe not so much.
Dropping down a level or two of meta, what about the object-level claim that
harms from commenters who hurt other users' feelings can be successfully mitigated by rules, because such commenters are just trying to make what they see as good comments rather than being sadistic emotional damage maximizers
? I'm not convinced by the argument, just yet; internet forums are fairly well known for sometimes harbouring commenters who are something like sadistic emotional damage maximizers, and even someone who can't fairly be described that way can get a bee in their bonnet about some other person or community or set of ideas and start behaving rather like a damage-maximizer. Less Wrong seems to me to have fewer such people than most online communities (which may be partly because the moderators work hard to make it so; I don't know) but it also has more ingenious people than most online communities, and when someone goes wholly or partly damage-maximizer on LW I think we have more of a "nearest unblocked strategy" problem than most places do.
I am not claiming that your conclusion is wrong. It might be right. But I am not convinced by your argument for it.
(I am also not convinced that someone has to be a damage-maximizer in order to behave unhelpfully in the presence of inevitably-incomplete rules.)
If that’s what Zack means, then it seems odd that the OP puts so much emphasis on how rules can be an effective way to keep people safer from literal death, brain damage, and the like.
Oh, I didn’t mean that the LW mods are the only examples of this sort of thing. But you did mention that you’d never encountered such people, and my response was to say that yes, you have indeed.
As for Zack’s examples, I think that they illustrate fairly well the general principle that he’s describing. I’ll leave it to him (or others) to answer the criticism beyond that.
Noted. For what it's worth, I haven't as yet seen much reason to believe that the LW moderators think they are empowered "to make judgement calls controlling the minute details of everyone's behaviour".
If you delete the word "the", or better yet make things more explicit along the lines of "to make judgement calls about small details of what behaviour is acceptable when posting/commenting on Less Wrong", then I dare say the moderators consider themselves empowered to do that. But that's very much not the sort of thing that I understand when I read "make judgement calls controlling the minute details of everyone's behaviour", especially not at the end of an article full of examples about regulation of neurotoxic chemicals and road safety.
So perhaps this is yet another case where Zack says something that can with substantial stretching be interpreted as something true, but that (at least as it seems to me; others' intuitions may differ) is an extremely unnatural way to say what he claims was all he was saying, and that strongly suggests something much worse but that is not actually true. I am beginning to find it rather tiring.
Why did the early computer vision scientists not write succeed in writing a formal ruleset for recognizing birds, and ultimately it took a messy cludge of inscrutable learned heuristics to solve that task?
I disapprove of Justice Potter in many respects, but "I know it when I see it" is indeed sometimes the only practical[1] way to carve reality.
(This is not meant to be a robust argument, just a couple of pointers at countervailing considerations.)
For humans.
I think this point and Zack's argument are pretty compatible (and both right).
Rules don't have to be formally specified, just clear to humans and consistent and predictable in their interpretation. Common law demonstrates social tech, like judicial precedent and the reasonable-person standard, for making interpretation consistent and predictable when interpretation is necessary (discussed in Free to Optimize).
I don't think this is relevant and endorse Nick Tarleton's sister comment.
The thing about Jacobellis v. Ohio is that, even if Justice Stewart had to make a judgement call on whether Les Amants constituted an obscene film, at least Ohio did have a law banning obscene films. That law was written down, such that people considering distributing pornography in Ohio could reason about what the law said was allowed, rather than having to proactively infer and conform to Ohio law-enforcement officials' personal preferences about how they want citizens to behave.
Quetzal_rainbow's comment points out that how much you want to rely on authority vs. rules depends on what social system you're talking about: for example, public libraries should be more rule-based than a private company.
In the case of, say, a public web forum, I think you clearly want it to be more like a public library, because all sorts of people might have something interesting to say about the forum's topic. There's no good reason to expect loyalty or mission-orientedness the way you would at a company.
So if, say, the administrators of a web forum want to make sure that authors feel comfortable using their moderation tools, and they're worried that authors won't feel comfortable if banned users retaliate for being banned by complaining on their shortform, they should make a rule that says, "Don't complain about being banned," so that users can reason about what the rules say are allowed, rather than having to proactively infer and conform to moderators' personal preferences about how users should behave.
It's true that we don't have a hand-coded Python program that classifies whether a comment constitutes a "complaint", much as we don't have a hand-coded Python program that classifies whether an image contains a bird, but that level of rigor just isn't what's being asked for. It's okay if the rules have to be written in English instead of Python, but for clarity's sake just make a rule!
The thing about Jacobellis v. Ohio is that, even if Justice Stewart had to make a judgement call on whether Les Amants constituted an obscene film, at least Ohio did have a law banning obscene films. That law was written down, such that people considering distributing pornography in Ohio could reason about what the law said was allowed, rather than having to proactively infer and conform to Ohio law-enforcement officials' personal preferences about how they want citizens to behave.
This observation is insufficient to prove the conclusion you're using it for. This is because there's yet another layer of abstraction/reasoning involved, namely the constitutional one. Justice Stewart was not making a local comment about the scope of a specific state law.[1] He was interpreting and judging state law in light of the Constitution.
It's worth it to be precise about all the layers involved:
So the question the Court was facing wasn't "is Les Amants obscene according to the Ohio law?"
The question they were actually facing was "is Les Amants obscene according to an judicially-created re-interpretation of what Ohio law means by the word 'obscene' that makes it the most constitutional, so that it can fit into a poorly defined category that we deem an exception to the literal text in the Constitution because of our unwritten understanding of history and tradition?"
Undergoing this endeavor is generally the province of state court judges/justices, except for constitutional avoidance purposes... more on that below
This is all interesting, but let’s recall that Jacobellis v. Ohio was not the last Supreme Court ruling on this! The important subsequent case was Miller v. California:
the Court acknowledged "the inherent dangers of undertaking to regulate any form of expression", and said that "State statutes designed to regulate obscene materials must be carefully limited."[1] The Court, in an attempt to set such limits, devised a set of three criteria which must be met for a media item to be legitimately subjected to state regulatory bans:
- whether the average person, applying contemporary community standards, would find that the work, taken as a whole, appeals to the prurient interest;
- whether the work depicts or describes, in an offensive way, sexual conduct or excretory functions, as specifically defined by applicable state law; and
- whether the work, taken as a whole, lacks serious literary, artistic, political, or scientific value.[1]
This test clarified the definition of obscenity originally set out in the Memoirs precedent.[6] This three-part analysis became known as the Miller test.[2]
The Miller ruling, and particularly the resulting Miller test, was the Supreme Court's first comprehensive explication of obscene material that does not qualify for First Amendment protection and thus can be banned by governmental authorities with criminal charges for those who distribute it. Furthermore, due to the three-part test's stringent requirements, very few types of content can now be completely banned, and material that is appropriate for consenting adults can only be partially restricted per delivery method.[13]
I recommend reading the linked page in its entirety. In short, the conclusion of later Supreme Courts was that Justice Stewart’s standard does not suffice, and that there need to be actual rules. Those rules have been created, and have greatly improved the state of free speech protections in the United States, precisely by both limiting the scope of what sorts of actions are out of bounds, and by creating greater clarity about what sorts of actions are out of bounds.
And while we gain little by spelling out the “actual rule” in these “complex and/or illegible exception-judging criteria” situations, nevertheless we do gain something—namely, making explicit (and therefore salient) the fact that unexpected exceptions (driven by irreducible judgment) are a possibility. What is explicit, can be better prepared for, and can be discussed, and problems addressed; so this is a benefit, if not a very great one.
(Of course, to the degree that “unexpected exceptions (driven by irreducible judgment)” are expected—or known!—to be frequent occurrences, the above is a proportionally greater benefit.)
I think the normal people have it basically right, and the people who aren't normal are being scared of ghosts
I think both sides are missing a VERY important element. No society or decision process is purely driven by rules. There are all sorts of incentives and societal consequences to legal-but-unpleasant behaviors. Human judgement fills in most of the gaps.
You only make more rules when there's some reason that natural coordination (aka: bullying) isn't doing the job. And the non-normal people are right to point out that this is underspecified and probably fragile in many edge-cases. And the normals are right that it's mostly worked so far.
"Rules" are a critical social technology for helping people live and work together in peace. From the laws passed by legislatures to govern a whole nation, to the bylaws of a neighborhood homeowner association, to the informal household rules of a single family, explicit rules make it clear to everyone what behavior is required and what behavior is forbidden, without otherwise controling every minute detail of everyone's behavior.
When there are clear rules, people don't have to drive themselves crazy contorting themselves into unnatural shapes to satisfy the whims of some distant Authority. All you have to do is make sure to obey the rules. With that taken care of, you can go about living your life the way you see fit, in freedom and dignity. As can be attested in the annals of human experience from the time of Hammurabi into the present day, it mostly works pretty great—at least compared to the alternatives. In summary, rules are good. It's good to have clear rules, and for people to obey the rules.
Normal people understand this pretty well and probably don't need to read a blog post about it, but some people who aren't normal have a theoretical objection. The space of all possible behaviors is unthinkably vast. What if the formidable intelligence of an adversary who hates everything our Society stands for, comes up with a behavior that's really bad but isn't forbidden by any of Society's rules?
The normal person is unfazed by the theoretical objection. If that happens, you could just make a new rule forbidding that behavior, right? How hard could that be?
The people who aren't normal are unimpressed with this reply. They can tell that the normal person doesn't understand the vastness of the space of possible behaviors at all. If you just make a new rule, surely the formidable intelligence of the adversary will contrive some other eldritch behavior that minimizes Society's utility function while complying to the letter of all of Society's rules. The theory of nearest unblocked strategies in the lore of AGI alignment, and the specter of specification gaming in the practice of ML engineering, make it clear that this is so. Thus, rules won't suffice; we need to empower leaders with the Authority to make judgement calls—even to control the minute details of anyone's behavior, if that's what it takes to safeguard Society's Values.
Now me, I'm normal on my mother's side, which puts me in a good position to understand what both parties to the disagreement are saying. And while my full belief-state about related topics in the theory of decision and optimization is nuanced and complex, on the narrow question of what to do about rules in human Society, I think the normal people have it basically right, and the people who aren't normal are being scared of ghosts. Let me explain.
I do not dispute the lore of AGI alignment, nor the practice of ML engineering. But crucially, the purpose of rules in human Society is highly disanalogous to the purpose of a utility or reward function in AI. Rules aren't supposed to express Society's true Values, let alone be a perfect specification robust to nearest unblocked strategies. The Values live in the hearts of Society's individual women and men, to be expressed in the way they go about living their lives the way they see fit, in freedom and dignity. The rules are just there to stop ourselves from trying to kill each other when your freedom and dignity is getting in the way of my freedom and dignity, so that we can focus on creating Value instead of wasting effort trying to kill each other.
Rules are written to ensure conditions conducive to people living their lives in freedom and dignity when those conditions wouldn't obtain in the absence of a rule. Traffic laws make it clear to everyone when it's safe to enter the road. If everyone just entered the road whenever they felt like it, that would be dangerous, and the danger would interfere with people living their lives in freedom and dignity.
The theory of nearest unblocked strategies can be relevant to rules in human Society to the extent that the conditions that a rule is intended to ensure are something that some people oppose either terminally or due to strong instrumental convergence. Income tax laws are passed so that the government will have money to fund police to enforce all the other laws, but that money has to come from somewhere and people really don't like having less money, so they put the full force of their effort and ingenuity into side-stepping the law with clever nearest unblocked strategies: underreporting cash transactions, hiding money in offshore accounts, recategorizing consumption as business expenses, &c.
But more often, the conditions that a rule is intended to ensure aren't something that people terminally or convergently-instrumentally oppose. The rule merely restricts behavior that people would otherwise engage in instrumentally, but not convergently instrumentally: if the rule is in place, they can and will avoid the behavior in order to comply with the rule.
Lead paint is an environmental hazard, so it was banned in 1978. Because of the ban, paint manufacturers stopped making lead paint. The paint manufacturers did not put the full force of their effort and ingenuity into clever nearest unblocked strategies for increasing the amount of lead in the environment, because they're not environmental lead maximizers, which aren't a real thing. The paint manufacturers just wanted to make paint. When there wasn't a rule against it, they used lead carbonate in their paint because it was convenient, but when there was a rule against it, they stopped. The rule worked—without the need for empowering an Authority to make judgement calls controlling the minute details of everyone's behavior. Why wouldn't it?
In some situations, there might be weak instrumental convergence pressures such that the first attempt at making a rule doesn't quite succeed at ensuring the conditions that the rule was meant to ensure. It turns out that, on further consideration, Society doesn't just want to avoid environmental contamination with lead in particular, but all other toxic heavy metals, too, some of which also happen to be convenient for making paint. So paint manufacturers still ended up using mercury in some paints until 1991 when that was banned, too. But once it was banned, they stopped. Why wouldn't they? They're not environmental mercury maximizers, either, which also aren't a real thing.
The work of coming up with rules to ensure socially beneficial outcomes can be frustrating, because you won't always get the rules exactly right the first time. You might need to iterate. But it's a finite and achievable amount of work, not an unwinnable unending battle against the formidable intelligence of an adversary who hates everything your Society stands for, because those mostly aren't a real thing either.
In conclusion, I think that people who think rules are unworkable and instead want to empower an Authority to make judgement calls controlling the minute details of everyone's behavior need to read less science fiction and spend more time relating to other people in their Society as people. Notwithstanding that terrifying alien superintelligences couldn't be constrained by rules because a merely human intellect lacks the capabilities to enumerate all the nearest unblocked strategies, other people in your Society are not terrifying alien superintelligences. We're just people who don't have exactly the same preferences as you. We won't always agree, but it shouldn't be this hard to live in peace with each other. If there are problems, you can just make a new rule!
(Thanks to Robert Mushkatblat and Ben Pace.)