In real (American gridiron) football, this problem is solved by the NFL tweaking the rules every year. They understand they are producing an entertainment product, and a typical pattern is something like this:
As a fan of both the New England Patriots and David Sirlin, I find this cycle itself very entertaining. I am glad when the Patriots do this because it helps them win and it is, IMO, in the true spirit of competition to make all legal moves in service of victory.
[1] I recognize that the Patriots are widely believed to have cheated on several occasions. I don't condone cheating, and don't think that cheating makes one a "lovable scamp" or whatever. I personally find the actual evidence that the Patriots cheated on some of those occasions to be less than credible (and people's views on it colored by their negative reaction to the Patriots Cycle), but I also recognize that I am biased and likely not able to fairly evaluate that evidence.
This pattern is even more pronounced in motorsport. The history of Formula 1 is the story of teams finding ways to tweak their cars to gain an advantage, other teams whining about unfairness, and the FIA then tweaking the rules to outlaw the "innovation".
Examples include:
In fact, one of the criticisms that many fans have of the FIA is that it goes too far with this. It seems like the moment any team gains an advantage by exploiting a loophole in the rules, the FIA takes action to close the loophole, without necessarily waiting to see if other teams can respond with innovations of their own.
There's a similar, albeit slower moving, pattern in baseball. This season saw some of the largest rule changes in years with massive limitations on shifts (to wipe out some of the Moneyball-type strats) and the addition of a pitch clock.
The more common strategy taken by baseball, though, is to codify the spirit of the game. For example, the game codifies the concept of player error and limits how much the other team can benefit from errors, such as only allowing a double when overthrowing first in an out attempt. The best example is likely the infield fly rule which prevents fielders intentionally missing the catch in order to get more outs on the play.
I'm not sure if this is better or worse than cricket's approach, but it does suggest an alternative strategy to achieve similar ends.
Isn't it worse, for the reason mentioned in the OP (it's Goodhart-able)?
This is why it is important that the 'spirit of cricket' is never properly codified into laws. If it was, then players would simply game the rules and find the most successful strategy that operates within the laws of the game and the process would be Goodharted.
Maybe in theory, but in practice humans have actual limits on performance and sometimes adequate rules actually succeed in limiting our ability to continue to game the systems to our advantage. That or the cost of further gaming is too high to be worth it (for example, in baseball egregious bad sportsmanship is often rewarded with the pitcher intentionally hitting the batter with the pitch).
This year, the big NFL controversy is over a play that the Philadelphia Eagles were repeatedly successful with last season. When the Eagles offense only needed a short gain, they would have their quarterback run the ball with 1 or 2 of his teammates pushing him from behind like in a rugby scrum. Despite the arguments that this play is a loophole that is "not football", the NFL decided not to change the rules for the upcoming season, so we'll probably see a lot more of this play by other teams for at least 1 more year.
Interesting, thanks for sharing! Its cool to see how different games manage the conflict between coming up with innovative tactics (which for me is all part of the fun of sports) and exploiting the rules in a way that makes the game boring.
Also thanks for the link to David Sirlin. I haven't heard of him and the website looks interesting!
Due to it's nebulous and poorly-defined nature, the 'spirit of cricket' cannot be easily Goodharted.
I think this conclusion is premature. In many sports, a foul is ultimately defined by whether the referee calls it a foul. Isn't this 'spirit of cricket' similar except with the cricket-loving-public-and-chattering-class as the referee?
Moreover, why would something being poorly defined make it harder to optimize? Can we solve AGI alignment by just being super vague about all the instructions we give to our AIs and data-labellers?
why would something being poorly defined make it harder to optimize
Well, you need to know what you're optimizing! In a two-player game, if the second player gets to redefine the rules after the first player has moved, then they get a huge advantage. That's essentially what happens by defining "the spirit of cricket" vaguely.
How is this different from games with a referee? A foul is what the referee says it is; the spirit of cricket is what the cricket-lovers say it is. In both cases a savvy optimizer would start modelling the relevant humans and predicting what they would and wouldn't judge illegal.
I agree that different rules or optimization targets have different complexity levels, and the spirit of cricket seems more complicated than ordinary fouls which are more complicated than "did the ball hit the pegs."
I think the two-player-game-but-player2-gets-to-modify-the-rules is not a fair analogy here. Like I said it's the cricket-loving public that decides, not player 2.
Ah, sorry for unclarity. The game I'm referring to is the one between the player who's trying to game the rules, and the referee/rule-judging body that's trying to avoid being Goodharted. The judging body can either "move first" by specifying the rules precisely, or "move second" by judging whether or not an action broke the rules according to illegible criteria. The latter is straighforwardly much harder to Goodhart. Or they can do a combination: I think of referees as doing a combination of these things, because they're meant to interpret fixed, well-defined rules, but there's still some room for judgment calls.
I think the two-player-game-but-player2-gets-to-modify-the-rules is not a fair analogy here. Like I said it's the cricket-loving public that decides, not player 2.
Broadly, I agree with Richard Ngo's characterisation. You are right that the 'cricket loving public' plays some part in determining what counts as 'within the spirit' but it is the decision of the players themselves that often is most important.
How is this different from games with a referee? A foul is what the referee says it is; the spirit of cricket is what the cricket-lovers say it is. In both cases a savvy optimizer would start modelling the relevant humans and predicting what they would and wouldn't judge illegal.
I agree that different rules or optimization targets have different complexity levels, and the spirit of cricket seems more complicated than ordinary fouls which are more complicated than "did the ball hit the pegs."
I agree with you that the complexity is an important factor. I think you are correct that in principle this can still be Goodharted, but in practice it doesn't seem to happen as it is much harder than Goodharting the written rules of the game, due to the increased complexity. There is nothing to prevent a superintelligent player from brainwashing the opposing team and general public to agreeing that their actions are legitimate. It's just that doing this is a lot harder than normal ways of 'gaming the system'. This is why I used the term 'resists Goodharts law' as opposed to 'defeats Goodharts law' or something similar.
It may be that there isn't big enough money in cricket for it to be attractive to hypercompetitive athletes and coaches who are most likely to apply that optimization pressure?
Second player judging is in some ways an advantage, and in other ways a disadvantage (and either way not really what's happening in cricket, because the players are actually cooperating rather than trying to exploit the rules but being foiled).
The disadvantage is that, lacking rules, it's hard to communicate what you want the first player to do at all! You don't get children to play soccer by not telling them any rules or giving any demonstrations, only judging their actions as legal or illegal.
If you do manage to communicate your preferences about what kind of game is even being played to player 1, then there's no qualitative barrier to communicating enough information about your standards that you can get goodharted.
An inverse example is the role of fights in hockey.
Fighting is explicitly disallowed by the rules of hockey. If players get into a fight, one or both players will be penalized. Nonetheless, it is widely held by coaches, players, and fans that fighting is part of the "spirit of hockey", and so fights still occur with some regularity. This is sometimes for strategic reasons (baiting an important player into a fight in order to get them into the penalty box), and sometimes for personal reasons, to settle grudges, or to punish certain kinds of technically-legal player behavior. Thus, even though the rules don't allow fighting, fighting is an accepted part of the strategic metagame.
Unfortunately, in the past several years the owners and the NHL have tried to stamp out this practice, as a means to make the sport more "respectable" and (I assume) the avoid something like the concussion controversy that has followed the NFL. All of the long-term fans of the game that I've talked to agree that this is a bad idea and they should bring the fights back.
I'm reminded of speedrunning where games often have a spectrum of categories from all glitches allowed to "beating the game as intended."
To clarify Mankading... it sounds to me like: sometimes the batter near the bowler starts to run before the bowler has actually thrown. In the rules of cricket, that gives the bowler the chance to get them out instead of throwing the ball like they normally would. It's in the spirit of cricket for the bowler to say "hey, if you do that I'm gonna try to get you out". It's not in the spirit of cricket for the bowler to simply get them out without warning.
Is that right? If so, some other questions: How common is this run-before-throw thing? Is it deliberate, or careless? Does the batter ever try to run even after being warned, and if so, how often do they survive? Would it be in the spirit of cricket for the bowler to warn before every throw, or are they expected to tolerate a certain amount of run-before-throw? If exactly one team took every possible opportunity to Mankad, would that give them an advantage, or would people simply stop giving them the opportunity?
sometimes the batter near the bowler starts to run before the bowler has actually thrown.
Yes!
In the rules of cricket, that gives the bowler the chance to get them out instead of throwing the ball like they normally would.
There is a line drawn on the floor known as the 'crease', about a metre pasts the stumps. If the batter has run past this line while the bowler still has the ball, the bowler can tap the stumps with the ball and get the batter out.
It's in the spirit of cricket for the bowler to say "hey, if you do that I'm gonna try to get you out". It's not in the spirit of cricket for the bowler to simply get them out without warning.
Yes, the bowler would say something like 'you ran too far and were out of the crease. I could have got you out then.'
How common is this run-before-throw thing? Is it deliberate, or careless?
Running or walking as the bowler is about to bowl is deliberate and common (happens pretty much every ball) and is known as 'backing up'. Batters do it both to get a head start and to be 'on their toes' and ready to run. However, if you run too far and step out of the crease, this is careless.
Does the batter ever try to run even after being warned, and if so, how often do they survive?
Yes, sometimes, and often they do not survive. I don't know the numbers, but this non-comprehensive list gives two examples where they got out after being warned.
Would it be in the spirit of cricket for the bowler to warn before every throw, or are they expected to tolerate a certain amount of run-before-throw?
As a bowler, you tolerate it provided that the batter is not running out of his crease. As a bowler, you provide a warning by stopping your run up, and pointing out to the batsman that he has run out of his crease. If you cared and were interested in Mankading (for example, because you felt that the batter was getting an unfair advantage by running too far) you could warn them the first (or second or third...) time you noticed them running out of the crease.
If exactly one team took every possible opportunity to Mankad, would that give them an advantage, or would people simply stop giving them the opportunity?
In the short term, they would have an advantage. Then (I guess) other teams would adapt their playstyles (for example, they would be more cautious when backing up, and probably as a result running between the wickets less) and the advantage would be negated. On the other hand, if a team knew that their opponents would *not* Mankad, regardless of how egregious their backing up was, they would be able to exploit this, by running far down the wicket before the bowler had bowled.
Isn’t this stupid? To have an extra set of ‘rules’ which aren’t really rules and everyone disagrees on what they actually are and you can choose to ignore them and still win the game?
Yes, it is stupid.
Games aren't real life. The purpose of participating in a game is to maximize performance, think laterally, exploit mistakes, and do everything you can, within the explicit rules, to win. Doing that is what makes games fun to play. Watching other people do that, at a level that you could never hope to reach is what makes spectator sports fun to watch.
Imagine if this principle were applied to other sports. Should tennis umpires suddenly start excusing double faults, because the sun was in the eyes of the serving player? Should soccer referees start disallowing own-goals, because no player could possibly mean to shoot into their own net? If a football player trips and fumbles the ball for no particular reason, should the referee stop the play, and not allow the other team to recover? If Magnus Carlsen blunders and puts his queen in a position where it can be captured, should Ian Nepomniachtchi feel any obligation to offer a takeback?
Fundamentally, what happened was that Bairstow made a mistake. He made a damned silly mistake, forgetting that overs are six balls, not five. Carey took advantage of the error, as was his right, and, I would argue, his obligation. Everything else is sour grapes on the part of the English side. If the Australian batsman had made a similarly silly mistake, and the English bowler had not taken advantage, I would be willing to bet that very few would be talking about the sportsmanship of the English bowler. Instead, the narrative would have been one of missed opportunities. How could the bowler have let such an obvious opportunity slip through his fingers?!
The purpose of participating in a game is to maximize performance, think laterally, exploit mistakes, and do everything you can, within the explicit rules, to win. Doing that is what makes games fun to play. Watching other people do that, at a level that you could never hope to reach is what makes spectator sports fun to watch.
I don't know if you read the rest of the piece, but the point I was trying to make is that sometimes this isn't true! Sometimes if each team does everything within the rules to win then the game becomes less fun to watch and play (you may disagree, but many sports fans feel this way). I already gave some examples where this happens in other sports, so I don't see the need for your list of hypotheticals (and I feel like they are strawmen anyway).
For what its worth, I agree with you on Bairstow/Carey but which side you take on it is irrelevant (though I can see you are quite passionate about it!). The piece was about the 'meta' aspects of games which try to address these kind of issues.
Sometimes if each team does everything within the rules to win then the game becomes less fun to watch and play
Then the solution is to change the rules. Basketball did this. After an infamous game where a team took the lead and then just passed the ball around to deny it to their opponents, basketball added a shot clock, to force teams to try to score (or else give the ball to the other team). (American) Football has all sorts of rules and penalties ("illegal formation", "ineligible receiver downfield", "pass interference", etc) whose sole purpose is to ensure that games aren't dominated by tactics that aren't fun to watch. Soccer has the off-sides rule, which prevents teams from parking all their players right next to the other team's goal. Tennis forces crosscourt serves. And, as I alluded to above, motorsport regularly changes its rules, to try to ensure greater competitive balance and more entertaining races.
With regards to chess, specifically, Magnus Carlsen agrees (archive) that classical chess is boring and too reliant on pre-memorized opening lines. He argues for shorter games with simpler time controls, which would lead to more entertaining games which would be easier to explain to new viewers.
None of these other sports feel the need to appeal to a wooly-headed "spirit of the game" in order to achieve entertaining play. What makes cricket so special?
EDIT: I would add that cricket is also undergoing an evolution of its own, with the rise of twenty-20 cricket and the Indian Premier League.
American baseball went thru something like this in 1908, the Merkle Game. Fred Merkle of the Giants was on 1st when a teammate singled to apparently win the game-but Merkle never touched second. Cubs got the ball out of the madding crowd and touched 2nd for the force, nullifying the run.
Merkle was roundly reviled for his boner, but just like in the cricket game the common practice at the time was to not run to second. But this time the defense broke the unwritten rule in question and the umpire likewise enforced it.
As an idealist I appreciate the sentiment of the entry here. But we I'd say are delving a bit into game theory. The danger of these unwritten rules is that they can be broken at untimely junctures-all it takes is one bad actor and the unwritten rule is destined for the scrap heap. In a more enlightened realm we'd likely not need any rules at all (Calvinball über alles), but not here at this time in history.
This is why it is important that the 'spirit of cricket' is never properly codified into laws. If it was, then players would simply game the rules and find the most successful strategy that operates within the laws of the game and the process would be Goodharted.
This is a fascinating take! Ambiguity and things different people see differently as a defense against Moloch and Goodhart. I think there's a lot of people in this community, myself very included, that don't like ambiguity and would prefer if everything had a solid, clear, objective answer.
The point of games is the games themselves, and "games" are wider than "winning". That which restricts goodhart's law is a holistic perspective (because it's not a single factor, at least in a way).
If you play for personal enjoyment, then the optimal strategy for yourself is also subjective, which adds a sort of diversity (playstyles and tendencies) rather than converging towards a handful of strategies (meta-gaming)
To generalize, the problem with Mollock is how society degenerates, losing its humanity as everything tends towards mathematical rules. If you think logically enough, these rules will bind you, you will have no choice but making the optimal choice. If you're unluckly, you possess enough self-awareness to see that the summation of locally optimal choices results in (or tends towards) a terrible global state.
Funnilly enough, while this problem is unsolvable, there's still a way to avoid it. All it takes is for one to not be aware of the problem in the first place! There's a correspondence between problems and solutions, following newton's third law. The real solution is to throw out the problem, not solving it. A sort of Buddhist "Letting go" of a conflict (what you resist, persists)
Edit: Perhaps I should explain: If most people present are trying to have a good time rather than just optimizing, then there will be no pressure for you to optimize either. These issues were less common in the past because we were more ignorant. An example is how we could enjoy vanilla World of Warcraft because the strategies weren't well-known to most users. Information is too redibly available nowadays, so I'm not sure that these old gaming communities can come to exist again.
Recently, there has been some controversy in the world of cricket. The full details are described well in this piece but the gist of it is as follows. During a game between Australia and England, after the ball had been bowled, the English batsman Jonny Bairstow stepped forward thinking that the game was no longer in play. This allowed the Australian wicketkeeper Alex Carey to throw the ball at the stumps and get Bairstow out. As a result, Bairstow had to leave the field and later that day, Australia won the game. Would England have won were it not for this piece of quick thinking from Carey? It's hard to say, but the game certainly would have been a lot closer.
So why is this controversial? After all, Carey's actions were completely within the rules of the game and all of the umpires and players agreed that Bairstow indeed out. The controversy stems from a nebulous concept known as 'the spirit of cricket'. The 'spirit of cricket' is mentioned in the preamble to the official Laws of Cricket and broadly encompasses good sportsmanship and fair play. But the fact that it is not properly defined means that different people will have different views on what kind of actions are within the spirit of the game. Pat Cummins, the Australian captain believes that Carey's actions were within the spirit of the game, and the Ben Stokes, the English captain believes that they were not. Various media outlets have taken sides on this issue.
To be clear, the 'spirit of cricket' is not just 'good sportsmanship applied when convenient'. Players often make choices that are within the spirit of the game which put their team at a severe disadvantage. Indian player MS Dhoni famously withdrew his appeal against Ian Bell. Within the rules of the game Bell had been run out in a moment of confusion but Dhoni believed that it was not within the spirit of the game, and let Bell come back to bat. The spirit of the game is something that spectators and players genuinely care about. Certain historical cricketing episodes, such as the underarm incident, and the bodyline tour were within the rules but still evoke strong emotions to this day[1]. Other tactics, like Mankading (where a bowler hits the stumps during his run-up to 'run out' an over-eager batsman who has started running) are allowed by the rules but rarely occur, though they still happen occasionally, generating controversy when they do and exist in a grey area with respect to 'the spirit'.
Isn't this stupid? To have an extra set of 'rules' which aren't really rules and everyone disagrees on what they actually are and you can choose to ignore them and still win the game? I would argue that, in fact, it is this strange, poorly-defined concept of 'the spirit of cricket' which makes cricket entertaining to watch and helps it avoid pitfalls present in many other sports.
Take football (soccer[2]) which is arguably the most popular spectator sport worldwide. Presently, one of the key ways that players can get an advantage for their team is to be 'fouled' by a member of the opposing team. Examples of 'fouls' include tripping or pushing. In order to convince the referee that they have been fouled, football players will often dramatically fall to the floor, screaming and clutching some body part, only to get up moments later, seemingly unhurt. While 'diving' (attempting to deceive the referee by pretending to be fouled) is against the rules of football, it is not against the rules to draw attention to a potential foul by exaggerating your injury. As a result, many high-level games become competitions as to who can get the referee to award them a penalty or free kick and players end up spending significant amounts of time writhing on the floor. Ask any football fan what their least favourite part of the modern game is and they will probably point to this. If football had something like 'the spirit of cricket' maybe this behaviour would be less common.
The way that players play football, (and many other sports) is optimized purely for winning. As a result, a few strategies come to dominate and everyone who doesn't follow them loses. I am told by friends who follow baseball that the 'Moneyball' approach to baseball is now ubiquitous, leading all teams to play in very similar, optimized styles. This blog on 'Refinement Culture' makes the case that, across many sports, strategies are becoming homogenised.
A similar criticism has been levelled against chess. At high levels players will memorise long lines of opening theory. Players who do not do this lose. But as a result, it will often be over twenty moves into a game before a 'novelty' is played. Memorising long opening lines is a good strategy for winning, but the advantage is negated if everyone does it and the game becomes less exciting. Bobby Fischer tried to remedy this by creating a chess variant with random the positions of the pieces on the board, hoping that this would engender a game with greater emphasis on creativity [3]. Fischer was partially successful . While Fischer Random chess is popular, it is still nowhere near as popular as regular chess.
These are all examples of Goodhart's law. Having a game where both sides are trying to win makes for an exciting game. But if both teams are only optimizing for winning, they will hit upon strategies (such as diving in football and memorising lines in chess) which are effective at winning but boring to watch.
This is why it is important that the 'spirit of cricket' is never properly codified into laws. If it was, then players would simply game the rules and find the most successful strategy that operates within the laws of the game and the process would be Goodharted. I suspect that this is the reason that cricket seems to have resisted the tendency towards 'refinement' that is present in other sports. If anything, cricket is becoming more baroque and varied. The last ten years of cricket have seen several new types of shots such as the reverse sweep, the switch hit , and the scoop all played commonly at high levels[4].
Due to it's nebulous and poorly-defined nature, the 'spirit of cricket' cannot be easily Goodharted. It also helps solve the coordination problem inherent in many sports. There exist many strategies (such as diving in football, or Mankading in cricket, or memorizing opening theory in chess) which give you an advantage if you are the only one using them but if everyone uses them, the advantage is negated and the quality of the game becomes worse. Everyone ends up back where they started, and playing a slightly different, slightly less fun version of the game. Then the process repeats, the tactical arms race continues, and the quality of the game decreases. Every individual and team behaves rationally, yet the endgame is result that nobody wants. This is an example of a bad game-theoretic equilibrium, resulting in Moloch-like dynamics. But in cricket when someone tries a new tactic, it is followed by discussions of 'the spirit' which allow players to decide if this is the kind of thing they want to allow all other teams to do. If it is, then the tactic is declared 'within the spirit' (an example of this is the switch shot). If it is not, either the rules of the game are changed (as in the case of underarm bowling or bodyline) or the tactic continues existing in a grey area, within the rules, but outside of the spirit (like Mankading).
I don't know if there is a general lesson to be drawn from this, other than 'it is hard to Goodhart when the thing you are optimizing is poorly-defined'. It strikes me that the 'spirit' presents a novel way of avoiding race-to-the-bottom Moloch dynamics which could be useful. I also don't know much about other sports, so would be interested to hear if something equivalent exists in other games.
The rules surrounding underarm bowling and bodyline theory have since been revised, but they were both within the rules at the time they were played.
I am English, so I have to call it 'football' not 'soccer', otherwise the King will rescind my crumpet license.
It should be taken for granted that these criticisms are not detracting from the talent and skill of professional chess and football players. It is the incentives for them to play in this way that I am criticizing.
Technically, most of these shots have been around for a long time, but were not often played. The point is that they are becoming more common, not less.