Testing Intelligence

by paulfchristiano 2 min read2nd Mar 201119 comments


In my model of human behavior, there is an unobservable parameter I associate with intelligence. I observe people's behavior when playing a game, when solving a problem, when defending their beliefs, or when learning something new, and I infer something about their intelligence. This in turn informs my predictions about their success in a wide variety of pursuits. In practice, I often make strong predictions about someone's intelligence after observing their behavior on a single occasion.

An accurate conception of intelligence seems to be generally important. Understanding what easily leveraged factors affect someone's intelligence---during childhood, later education, and after formal education is complete---is important if your goal is improving intelligence generally. If you are considering relatively expensive personal engagement to develop rationality, you may want to direct efforts at individuals who have the potential to have a significant impact as researchers or entrepreneurs. And so on.

Before thinking about how to understand determiners of intelligence, how to measure intelligence effectively, or the effects of intelligence on behavior, I would first like to get a feel for what my intuitive understanding of intelligence really corresponds to, if anything. It is possible that my intuitive assessments of intelligence are largely unrelated to reality, and that my beliefs about the world could be improved by discarding them. It is also possible that some of my intuitions about intelligence are quite accurate, and I could make better decisions by giving them more credence or by changing the way I use those intuitive judgments.

Intuitively, I expect the results of many types of otherwise apparently unrelated tests to be very tightly correlated with intelligence. To understand the extent to which this intuition is correct, I am considering conducting a slightly systematic study of the relationship between different metrics.  I would appreciate pointers to reliable scholarship surrounding this question, but a brief search turned up mostly very muddled thinking and a general lack of people doing good experiments.

Here is a range of metrics which I suspect correlate well with my conception of intelligence, at least in certain regimes (some of these metrics may only correlate meaningfully when applied to very bright subjects, or may not correlate meaningfully when applied to very bright subjects):

1. General intelligence factor as estimated by standardized cognitive tests, e.g. Raven's Progressive Matrices.

2. Ability to quickly learn an unfamiliar formalism. For example, to quickly learn a new game and to understand simple strategic consequences of its rules.

3. Ability to infer an underlying model. For example, to learn how to achieve a goal when allowed some constrained interaction with / observation of an unknown environment.

4. My assessment of intelligence during collaboration or discussion of a complex but rigorously defined topic; or, the assessment of anyone who I consider to be intelligent.

5. Ability to solve hard problems in a well-understood environment, potentially given hours or days. For example, performance in high school olympiads.

My hope is that by gaining a better understanding of the relationship between these metrics I may learn to what extent my current rather monolithic conception of intelligence is valid and, to whatever extent it is, how to effectively measure it. Ultimately I would like to understand what easy measurements are the best indicators of success at various particular pursuits, but is even more extraordinarily difficult to acquire data about how good someone is at, say, choosing good research problems.

What do readers expect the results of inquiry to look like? Is my choice of metrics influenced unduly by my own experience? What are other metrics I should be considering but am not? Is improving a student's ability to perform any of these tasks likely to have a positive influence on other tasks?