I'm indulging in the simple pleasure of drawing large conclusions from a single study.... Why exams are nothing out of context:
the story about the maths ability of Brazilian street kids living in the in the favelas of Recife. This story helped both of us realise the importance of carrying out usability tests in context. Three researchers (see: Carraher, Carraher, and Schliemann 1985) carried out research with children aged 9 to 15. These kids had dropped out of school, and were selling sun screen, and chewing gum on the streets. The researchers worked out that they could set the kids questions by purchasing goods off them. For example, 1,000 minus 300 is the same as giving the kid a 1,000 Cruzeiros note for a product that costs 300 Cruzeiros. Multiplication can be done by asking the kids how much 3 of a product would cost. In these tests the Brazilian street kids scored 98%. But when they were put into a formalised test setting, and asked instead of how much would 3 apples cost or what 3×9 is, the kids performance dropped to just 37%.
What is scary is that the researchers later tested middle class children in a private school. These kids did very well in the formal exam. But when they had to do transactions with real money in the street, using the same maths, they failed in being able to do the transactions.
Is it possible that the correlation between g and success isn't about raw intelligence, it's about being able to access one's intelligence in situations (like classrooms) which involve thresholds for easily improving one's status?