Generalized efficient markets (GEM) says that low-hanging fruit has already been picked. If there were some easy way to reliably make a fortune on the stock market, it would have already been done, and the opportunity would be gone. If some obvious theory explained all the known data on some phenomenon better than the current mainstream theory, that theory would already have been published and widely adopted. If there were some simple way to build a better, cheaper mousetrap, it would already be on the market.
Look back - one of those three is not like the others.
One nice property of markets is that it only takes a small number of people to remove an inefficiency. Once the better, cheaper mousetrap hits the market, the old mousetraps become obsolete; the new mousetrap is adopted even if there are initially many more old-mousetrap-makers than new-mousetrap-makers. Same with the stock market: even if most investors don’t notice a pattern, it only takes a handful to notice the opportunity and remove it. Competition drives market-beating profits to zero with only a handful of competitors, even if everyone else in the market misses the opportunity.
But academia is a different story.
In order for a theory to be adopted within an academic field, most people in the field must recognize the theory’s advantages. If a theory has fatal flaw which most people in the field won’t understand (e.g. because of insufficient mathematical fluency), it doesn’t matter how comprehensively fatal the flaw is, the theory won’t be dropped until some easier-to-understand evidence comes along (e.g. direct experimental refutation of the theory). A handful of people who understand the problem might sound the alarm, but that won’t be enough social signal to stand out from the constant social noise of people arguing for/against the theory in other ways.
Some examples of ways this might happen:
- Most people in the field don’t understand statistical significance
- Most people in the field don’t understand confounding
- Most people in the field can’t distinguish a predictive theory from phlogiston
- Most people in the field don’t have a notion of gears/causality, or don’t see why it matters
- Most people in the field don’t understand equilibrium reasoning
Now, as long as people in the field recognize the importance of correctly predicting experimental outcomes, the right theory will probably win out eventually. But we should still expect to see low-hanging fruit in the meantime, among theories not yet fully nailed down by experiment.
People who do understand these sorts of things should expect to see obvious problems in the theories of fields where most people don’t understand these things. Of course, that doesn’t mean we can win fame in the field just by pointing out these errors - quite the opposite. The whole point is that the structure of academia means that one person can’t correct the errors of a field, when that field does not have the background needed to understand the error. But it does mean that, if we want to understand the topic of such a field for reasons other than making a career as an academic, then we should not be surprised if we can theorize better than the supposed experts. We don’t need to outperform the best in order to outperform the field; we just need to outperform the average.