Overconfidence is easy to maintain if, and only if, it goes untested. This is the great trap found in many “soft sciences.” If feedback loops are short enough, with high enough fidelity, authority is of no great independent significance because it is derived from accuracy and its validity is updated regularly. But when feedback loops are long and noisy, authority bubbles can grow to enormous size. 

You will be hard-pressed to find such authority bubbles in the world of chess. If a charismatic leader becomes a proponent of 1. a4 as a new paradigm of opening theory, he will be hard-pressed to find followers because the local IM will blow him off the board. 

Much easier in macroeconomics, for example, where theories must largely go untested — because there's no way to test them. Easiest of all in politics, where truth is adjudicated by popular vote. 

There are a couple of interesting thoughts to chase down by comparing macroeconomics to chess. I think of it this way: what if, instead of playing through the game, we were forced to evaluate the position at move 15 and declare a winner? 

First, it would be difficult to know who the best chess player in the world actually is, because there would be lots of disagreement about who won a given game. This disagreement would be grounded in reasonable argument on both sides, but because it couldn’t be tested it also couldn’t be settled properly. 

We might also collectively arrive at incorrect conclusions. Suppose some opening sequence gives white a decisive material advantage but would lead to forced checkmate through a sequence of accurate moves by black. This could be nearly impossible to detect without actually playing out the moves, and players might reasonably conclude the position is a win for white. 

With that piece of incorrect knowledge lodged into the collective consciousness, chess theory would develop under the assumption that black must avoid this line, and white should favor it. When, one day in the far future, move-15 restriction is lifted, the chess world would be shocked to learn that all of the opening theory resting on this assumption was wrong.

Thus, the move-15 restriction would have two large, negative effects on chess theory. First it makes pairwise comparisons of skill much more difficult. But in the long run, it hurts everyone’s skill level by perpetuating incorrect assumptions. 

With chess it is an easy fix – lift the move-15 restriction! Play the game out to its conclusion to test out disagreements. In other fields it is often not so easy. Historically it hasn’t been so easy to test out macroeconomic theories – though I think crypto may be changing that, at least when it comes to some aspects of monetary policy. When testing theories really isn’t possible, we must acknowledge the kind of world we are in, one of great uncertainty, where we cannot reliably tell who is the more skilled practitioner, and core assumptions might turn out to be wrong.