So we know that many smart people make stupid (at least in retrospect) decisions. What these people seem to be lacking, at least at the moment they make a poor decision, is wisdom ("judicious application of knowledge"). More from Wikipedia:
It is a deep understanding and realization of people, things, events or situations, resulting in the ability to apply perceptions, judgements and actions in keeping with this understanding. It often requires control of one's emotional reactions (the "passions") so that universal principles, reason and knowledge prevail to determine one's actions.
From Psychology Today:
It can be difficult to define Wisdom, but people generally recognize it when they encounter it. Psychologists pretty much agree it involves an integration of knowledge, experience, and deep understanding that incorporates tolerance for the uncertainties of life as well as its ups and downs. There's an awareness of how things play out over time, and it confers a sense of balance.
Wise people generally share an optimism that life's problems can be solved and experience a certain amount of calm in facing difficult decisions. Intelligence—if only anyone could figure out exactly what it is—may be necessary for wisdom, but it definitely isn't sufficient; an ability to see the big picture, a sense of proportion, and considerable introspection also contribute to its development.
(1) wisdom as epistemic humility, (2) wisdom as epistemic accuracy, (3) wisdom as knowledge, and (4) wisdom as knowledge and action.
Clearly, if one created a human-level AI, one would want it to "choose wisely". However, as human examples show, wisdom does not come for free with intelligence. Actually, we usually don't trust intelligent people nearly as much as we trust wise ones (or appearing to be wise, at any rate). We don't trust them to make good decisions, because they might be too smart for their own good. Speaking of artificial intelligence, one (informal) quality we'd expect an FAI to have is that of wisdom.
So, how would one measure wisdom? Converting the above description ("ability to apply perceptions, judgements and actions in keeping with this understanding") into a more technical form, one can interpret wisdom, in part, as understanding one's own limitations ("running on corrupt hardware", in the local parlance) and calibrating one's actions accordingly. For example, of two people of the same knowledge and intelligence level (as determined by your favorite intelligence test), how do you tell which one is wiser? You look at how the outcomes of their actions measure up against what they predicted. The good news is that you can practice and test your calibration (and, by extension, your wisdom), by playing with the PredictionBook.
For example, Aaron Swartz was clearly very smart, but was it wise of him to act they way he did, gambling on one big thing after another, without a clear sense of what is likely to happen and at what odds? On the other end of the spectrum, you can often see wise people of average intelligence (or lower) recognizing their limitations and sticking with "what works".
Now, this quantification is clearly not exhaustive. Even when perfectly calibrated, how do you quantify being appropriately cautious when making drastic choices and appropriately bold when making minor ones? What algorithms/decision theories make someone wiser? Bayesianism can surely help, but it relies on decent priors and does not compel one to act. Would someone implementing TDT or UDT to the best of their ability maximize their wisdom for a given intelligence/knowledge level? Is this even a meaningful question to ask?
EDIT: fixed fonts (hopefully).