A Genius for Destruction

by Eliezer Yudkowsky1 min read1st Aug 200819 comments


Personal Blog

This is a question from a workshop after the Global Catastrophic Risks conference.  The rule of the workshop was that people could be quoted, but not attributed, so I won't say who observed:

"The problem is that it's often our smartest people leading us into the disasters.  Look at Long-Term Capital Management."

To which someone else replied:

"Maybe smart people are just able to work themselves up into positions of power, so that if damage gets caused, the responsibility will often lie with someone smart."

Since we'd recently been discussing complexity, interdependence and breakdowns, the first observation that came to my own mind was the old programmers' saying:

"It takes more intelligence to debug code than to write it.  Therefore, if you write the most difficult code you can create, you are not smart enough to debug it."

(This in the context of how increased system complexity is a global risk and commons problem; but individuals have an incentive to create the "smartest" systems they can devise locally.)

There is also the standard suite of observations as to how smart people can become stupid:

  • You become overly skilled at defending beliefs you arrived at for unskilled reasons;
  • Success on comparatively easy problems in the past, leads to overconfidence on more difficult problems in the future;
  • Because most of the advice offered you comes from people (apparently) not as smart as you, you become intellectually isolated;
  • You spend too much time as the smartest person in the room and can't handle it emotionally;
  • Because you enjoy looking smart, you avoid trying new and difficult things where you might not perform as well, and so become very narrow;
  • Because you enjoy looking smart, you don't like confessing your mistakes or realizing your losses;
  • Because people praise you for being good at math or something, you assume you are already wise, and you don't apply your intelligence to becoming wiser in other areas.

But I also think we should strongly consider that perhaps the "highly intelligent" sponsors of major catastrophes are not so formidable as they appear - that they are not the truly best and brightest gone wrong; only the somewhat-competent with luck or good PR.  As I earlier observed:  Calling the saga of the fall of Enron, "The Smartest Guys in the Room", deserves an award for Least Appropriate Book Title.  If you want to learn what genius really is, you probably should be learning from Einstein or Leo Szilard, not from history's flashy failures...

Still, it would be foolish to discard potential warnings by saying, "They were not really smart - not as smart as me."  That's the road to ending up as another sponsor of catastrophe.

Personal Blog