Which weighs more: a pound of feathers, or a pound of gold?
Close consideration of this riddle - and the conditions under which people tend to get it wrong - is helpful in understanding the limits of human rationality. It is a specific example which leads us to general principles of rationality failure.
These sorts of riddles and similar interpersonal language tricks (such as "Stupid says what?") are especially popular among children but not among adults. Why is this the case? Partly because adults are more likely to have previously encountered and become familiar with their patterns, but there are other factors - including one very relevant one. Children tend to have less-developed capacities of impulse control.
It takes very little analysis to discover the 'trick' in the question; the concepts involved are relatively simple. But we're confronted with the fact that people do answer it incorrectly, and that by manipulating aspects of the context in which the question is delivered, we can significantly increase the chance people will fall for it. What does this imply? That analysis is not being conducted in the erroneous cases, and that context is a contributing factor to whether people successfully engage in conceptual analysis. Specifically, that context determines whether people will counter their impulses long enough for analysis to be completed.
The key to these sorts of riddles is time pressure. If people feel free to take as much time as they like thinking over the question, they rarely fall for the trick. But if they're trying to answer rapidly, they'll screw up. Examples of situations that often result in such behavior include: competing against others to see who can be correct first, trying to demonstrate competence by investing little effort in answering, or encountering the question as part of a limited-duration examination. If several superficially-similar questions whose answer depends on retrieving facts from memory rather than performing logical analysis of the question are asked before the riddle is presented, that also tends to result in a wrong response.
The error occurs because of our weight-related associations with the concepts of 'feathers' and 'gold', our conditioned assumptions about the sorts of questions people are likely to ask, and a failure to inhibit the first impulses towards response. Feathers are far less dense than gold; any given volume of feathers will weigh far less than the same volume of the metal. Questions about a property rarely contain their own answers in a trivial way - we do not expect the defined quantities in the question to be equivalent relative to the property being asked about. And - this is the most vital aspect - it takes longer for our brains to process the question at a conceptual level than it does to activate our associations.
In the state of nature, organisms are often under intense pressure to produce results quickly. If they take too long, the resource they're trying to exploit may be taken by a competitor - or worse, they may become exploited resources by a predator. So stimulus-response methods which produce generally-useful reactions tend to be favored over extremely accurate and precisely analysis that takes longer. As a consequence, natural modes of though available to humans favor rapid responses more than rigorous correctness - and in much the same way that the limits of our visual processing systems lead to optical illusions, which can be understood and thus constructed, the limits of our conceptual processing lead to inherent tendencies towards fallacies of reason, which can be exploited to produce riddles and language gags.
Just as other aspects of our behavioral response involve the repression of rudimentary reflexes, our thinking involves the inhibition of associational activation and reflexive reactions. The "more advanced" cognitive functions can take place only because the simpler, less resource-intensive, and faster functions are prevented from initiating responses before them.
In the wrestling match between the modern functions and the ancient ones they try to control, the more subtle and advanced features are at a distinct disadvantage. Which brings us to the next post.