by PhilGoetz 3 min read25th Mar 201056 comments


Take a look at "Role of Layer 6 of V2 Visual Cortex in Object-Recognition Memory", Science 3 July 2009:
Vol. 325. no. 5936, pp. 87 - 89.  The article has some good points, but I'm going to pick on some of its tests.

The experimenters believed they could enhance object-recognition memory (ORM) by using a lentivirus to insert a gene into area V2 of visual cortex.  They tested the ORM of rats by putting an object in a field with a rat, and then putting either the same object ("old"), or a different object ("new"), in the field 30, 45, or 60 minutes later.  The standard assumption is that rats spend more time investigating unfamiliar than familiar objects.

They chose this test:  For each condition, measure the difference in mean time spent investigating the old object vs. the new object.  If the latter is more than the former, and the difference is statistically-significant, conclude that the rats recognized the old object.

Figure 1 Graph A (below the article summary cutoff) shows how much time normal rats spent investigating an object.  Here it is in HTML table form: How much time the rats spent exploring old and new objects:

Minutes after first exposure 30 45 60
Old 8 12 14
New 17 28 14

The black bars (new) are significantly longer than the white bars (old) after 30 and 45 minutes, but not after 60 minutes.  Therefore, the normal rats recognized the old objects after 30 and 45 minutes, but not after 60 minutes.

Figure 3 Graph D (also below the article-summary cutoff) shows how much time different types of rats spent exploring old and new objects.  The "RGS" group is rats given the gene therapy, but in parietal cortex rather than in V2.

Here it is in HTML form: How much time the rats spent exploring old and new objects, by rat type:

Rat type Normal (after 45 min) Parietal RGS (after 60 min)
Old 10 11
New 27 12

Parietal RGS rats displayed no difference in time spent exploring old and new objects after 60 minutes; therefore, this gene therapy to parietal cortex does not improve ORM.

To recap:

  1. We conclude that rats no longer recognize an old object if they spend about the same time investigating it as investigating a new object.
  2. Normal rats spend the same time investigating old and new objects 60 minutes after first exposure to the old object.
  3. Parietal RGS rats also spend the same time investigating old and new objects after 60 minutes.
  4. Therefore, normal rats and parietal RGS rats both lose ORM by 60 minutes.

So why don't I buy it?

Figure 1 (look at A).

Fig. 1 Animal performance on ORM task.

Figure 3 (look at D):

Fig. 3 Ox7-SAP injection in layer 6 and determination of ORM.

(Original image is here.)

The investigators were trying to determine when rats recognized an old object.  So what's most relevant is how much time they spent investigating the old object.  The time spent investigating new objects is probably supposed to control for variations in their testing procedure.

But in both of the graphs, we see that they are claiming that rats failed to recognize an old object in the 60-minute condition, even though they spent the same amount of time investigating it as in the other conditions.  The difference was only in their response to new objects.  The test methodology assumes that the response to new objects is always the same.

Look at the error bars on those graphs.  The black bars are supposed to all be the same height (except in 1B and 1C).  Yet we see they differ across conditions by what looks like about 10 standard deviations in several cases.

When you regularly get 10 standard deviations of difference in your control variable across cases, you shouldn't say, "Gee, lucky thing I used that control variable!  Otherwise I never would have noticed the large, significant difference between the test and control cases."  No; you say, "Gee, something is wrong with my experimental procedure."

A couple of other things to notice, in addition to the comments above:

  • The leftmost two sets of bars in 1B contrast the time spent examining old and new objects 60 minutes after exposure to the old objects, in normal and treated rats.  Note that, again, there is no difference between the time spent looking at the old objects between normal (control) and treated rats; yet they concluded that the treated rats remembered them, and the normal rats did not, because the treated rats spent more time looking at new objects than untreated rats did.
  • 1D is supposed to show that normal rats could remember only 2 objects, while treated rats could remember 6 objects.  But, again, this conclusion was reached because the normal rats spent less time looking at the new objects when exposed to 4 new objects than when exposed to 2 new objects.  There was no difference in the time they spent looking at the old objects with either type of rat under any of the conditions.

One subtle type of error is committed disproportionately by scientists, because it's a natural by-product of the scientific process of abstracting a theory into a testable hypothesis.  A scientist is supposed to formulate a test before performing the test, to avoid introducing bias into the test formulation in order to get the desired results.  Over-encapsulation is when the scientist performs the test, and examines the results according to the previously-established criteria, without noticing that the test results invalidate the assumptions used to formulate the test.  I call it "over-encapsulation" because the scientist has tried to encapsulate the reasoning process in a box, and put data into the box and get decisions out of it; and the journey into and out of the box strips off relevant but unanticipated information.

Over-encapsulation is especially tricky when you're reasoning about decision theory.  It's possible to construct a formally-valid evaluation of the probabilities of different cases; and then take those probabilities and choose an action based on them using some decision theory, without noticing that some of the cases are inconsistent with the assumptions used in your decision theory.  I hope to write another, more controversial post on this someday.