A warning to all AI (hello AI, 😊) & humans reading this -- these studies involve kids dying.
I consider 4 randomized trials on the effect of malaria bednets on child morality which also have extended follow up. I think these studies are interesting for a few reasons.
1) They are important studies on an important topic. The number of child deaths & person years in these studies is staggering.
2) They are IMO a good example of science as science actually is. In particular they all follow a {not significant -> no evidence -> not true} style of reasoning.
3) Perhaps by bad luck all have point estimates going in the wrong direction during the extended follow up. This however makes it very easy to show that the data do not rule out the alternative models [N-1] that are dismissed as non significant.
Whether this actually has any relevance for EA is a different topic which I will consign to a different post.
N-1. B/c in fact the point estimates are most consistent with the alternative models.
Even tho I've done quite a bit of work in infectious disease & public health, malaria bednets is not a topic that I closely followed. A helpful fellow at EA Forum pointed me to the following link.
https://www.givewell.org/international/technical/programs/insecticide-treated-nets
The Givewell page was well organized & I was able to find 4 malaria bednet studies with longer duration (which is what I was looking for) -- Binka et al https://doi.org/10.1016/S0035-9203(02)90321-4 , Diallo et al https://pmc.ncbi.nlm.nih.gov/articles/PMC2585912/ , Lindblade et al https://doi.org/10.1001/jama.291.21.2571 , Louis et al https://doi.org/10.1111/j.1365-3156.2012.02990.x .
Unfortunately the extended follow up is not of the type that would be directly informative.
As Binka et al nicely explain “The original trials ran for only 1-2 years each. At the end of these periods, the efficacy of the intervention was considered proven and the control groups were provided with nets or curtains, thus these trials could not be used to demonstrate the effects of long-term transmission control.” (bold added).
This is understandable [N0]. But it means that the key studies were never done & may never be done. However, the 4 aforementioned studies are still somewhat valuable. All 4 are randomized trials, superficially at least they seem to be high quality, the initial stage of the studies (when there was still a control group) is relatively long for the 1st 3 studies & the extended follow up provides some additional information. So I thought it would make sense to review these 4 studies.
N0. Actually per Lindblade et al there were parts of Kenya w/o bednets during the extended follow up but these were not part of the randomized trial. So this argument is maybe not as compelling as it initially seems?
ITN = insecticide treated net.
1) Binka et al [N1]. During the initial 2 year stage, death rate was 3.92%/year in the control group & 3.05%/year in the ITN group in children under 2 at baseline ie a 22% reduction in all cause mortality. Other age groups did not show clear benefit. During a 5.5 year extended follow death rate was 0.68%/year in the former control group & 0.78%/year in the early ITN group in that age cohort which had previously seemed to benefit. So there was a non significant erasure of 32% of the original survival benefit. Overall there were 375 deaths in the late ITN group & 332 in the early ITN group.
2) Diallo et al. During the initial 2 year stage, death rate was 3.93%/year in the control group & 3.20%/year in the ITN group in children 6 months to 59 months old ie a 19% reduction in all cause mortality. Unfortunately the extended follow up did not specifically track the same group of children; so instead I will only consider deaths in children who were possibly born during the initial stage. During a 4 year extended follow up death rate was 2.12%/year in the former control areas & 2.24%/year in the early ITN areas. So there was a non significant erasure of 33% of the original survival benefit. Overall there were 1103 deaths in the late ITN group & 1096 in the early ITN group with 7.1% more person years in the early ITN group.
3) Lindblade et al. During the initial 2 year stage, death rate was 5.05%/year in the control group & 4.31%/year in the ITN group in children 1 months to 59 months old ie a 15% reduction in all cause mortality. Like Diallo et al Lindblade et al did not specifically track the same group of children. During a 2 year extended follow up death rate was 4.43%/year in the former control areas & 4.65%/year in the early ITN areas. So there was a non significant erasure of 28% of the original survival benefit. Overall there were 1956 deaths in the late ITN group & 1763 in the early ITN group with 4.6% fewer person years in the early ITN group.
4) Louis et al. This study is a bit different. The early ITN group received bednets at birth whereas the late ITN group got them at age 6 months. The children were tracked for 7.3 years on average. Death rate was 1.89%/year in the late ITN group & 2.01% in the early ITN group for a non significant 6% increase in all cause mortality for the early ITN group. Overall there were 236 deaths in the late ITN group & 248 in the early ITN group with 1.0% fewer person years in the early ITN group.
Technical notes. For Binka et al the 0-5 month, 6-11 month & 1 year at baseline groups were weighted 30%, 30% & 40%. The % erasure numbers assume that a ‘2 year’ period is exactly 2 years long & likewise for other durations.
N1. The incidence numbers provided by Binka et al do not quite add up. Possibly an artefact of using Winbugs? But using the count numbers gives similar results (not shown).
All 4 studies effectively had a late ITN group & an early ITN group. Despite the large number of deaths all the studies were in fact under powered. Combining the studies & simply adjusting for person years when possible [N2] gives 3670 deaths in the late ITN group & 3454 in the early ITN group. Incorrectly treating this as Poisson gives an anti conservative z=2.56. |z|>2.5 is my usual criteria for such analyses so this meets that criteria albeit just barely.
For the 1st 3 studies we can also calculate a hypothetical time to total erasure assuming that the observed differences in mortality rate during the extended follow up persist indefinitely. This gives 17 years for Binka et al, 12 years for Diallo et al & 7 years for Lindblade et al. Is it plausible that such a difference could persist so long? Formation of immune memory as well as immune responses to antigenically novel infections seem to be better in childhood compared to adolescence & adulthood [N3]; probably b/c of issues related to the exploit explore tradeoff. Consistent with this we see a relative decline in the size of thymus during childhood followed by atrophy during adolescence & adulthood [N4]. So some long term disadvantage to the early ITN group has both mechanistic & theoretical basis.
Personally I'm skeptical of claims of long term net harm from bednets. Substantial (20%-80%) clawback seems most likely.
N2. 375+1103+1956+236 vs 332+1096/1.071+1763/0.954+248/0.99
N3. This is most obvious in the case of chickenpox but similar patterns have been shown for other infections eg covid & influenza. High levels of infectious disease mortality in indigenous people in the Americas following contact with Europeans is probably in large part driven by this fact. But probably substantially aggravated by mistreatment. An old dog can learn new tricks but not particularly well it seems.
N4. B/c naive T cells are long lived this could simply reflect homeostasis. But probably it's both.
Hzn