Scoring My AstraZeneca Predictions

After a relatively good month or so, as many Western countries have begun to relax their restrictions, COVID-19 cases and deaths are on the rise again, increasing especially in Africa, the Levant and South-East Asia.^[1] Over fifty thousand people die each week, and to date over four million people have died.^[2] The increase is largely due to the Delta variant, which now has a foothold on every continent that isn't Antarctica.^[3]

At the same time, just over a fourth of the world population has received at least one vaccination dose; about half of that number are fully vaccinated.^[4] I now see that the global vaccination rate has been declining since it peaked at the end of June.^[5] This makes me somewhat concerned. I am certain that the most likely way out of this purgatory phase of COVID-19 is through mass vaccination campaigns; in Texas, for example, 8,787 people have died from the virus since early February, and of those only 43 were fully vaccinated^[6]. That is why I argued against the suspension of the AstraZeneca (AZ) vaccine that many European countries announced last March, calling it among other things "stupid" and "a frustrating mess".

I think the arguments I made in that post are sound and that ultimately I was right. But I also made four predictions for the future, predictions that were less impressive. I wrote:

By July 1st 2021,

the expert consensus will be that continuing with the AstraZeneca vaccinations would have saved more lives than pausing it ⇒ 95%

a majority of the listed countries will have resumed administering the AstraZeneca vaccine ⇒ 90%.

the FDA will have approved the AstraZeneca vaccine for use in the U.S. ⇒ 95%

at least one elected official from one of the listed countries will have admitted that it was a mistake to suspend vaccinations ⇒ 70%

Let us review these one by one.

Prediction #1: Continuing Vaccinations Would Have Saved Lives, 95%

Obviously we will never know the true counterfactual here, but everything I have read indicates that this is true. The European Medicines Agency's website states:

[The AZ vaccine] offers a good level of protection against COVID-19 which is a critical need in the current pandemic. The main trials showed that the vaccine has around 60% efficacy. Most side effects are mild to moderate in severity and are gone within a few days. The European Medicines Agency decided that [the vaccine's] benefits are greater than its risks and it can be authorised for use in the EU.

A preprint from the end of March makes some simple Fermi estimates and concludes that pausing vaccination causes at least an order of magnitude more deaths than it prevents in France and Italy.^[7] According to the estimates, the longer the interruption in vaccinations, the more people die.^[8] Also according to the estimates, the relation is stronger where the virus is spreading more quickly.^[9] This image, where red is no vaccination and blue is a vaccination campaign with 59% efficacy, illustrates it clearly:

Plot showing estimated deaths in Italy and France from suspending the AZ vaccine compared to continued vaccination.

It does assume a fixed rate of vaccination; in reality, the bottleneck in most European countries was in vaccine doses, so it is possible that these countries could have caught up if their distribution capacity had exceeded their production capacity. But it also does not take into account other likely negative impacts of interruption, chiefly that it can damage public trust in the vaccine. It concludes that the estimates are likely to be conservative, that excess deaths caused by vaccine interruption are likely greater than estimated.

Another estimation comes from Christian Gollier:

The doubts that emerged in March about possible side effects [...] of the AstraZeneca vaccine resulted in a suspension of this vaccine for half a week in France and Germany, and for much longer in other countries. The benefit of this suspension remains unknown as I write this column, as the causal link between the inoculation and the symptoms, and the frequency of these side effects, are unclear. The cost of the delay can easily be estimated from the SIR model. A one-week suspension of the entire vaccination campaign in France at the end of March would have increased the dead toll at the end of the pandemic by around 1,800 compared to the benchmark. The economic loss would be increased by €8 billion.

There is also some evidence that the suspension eroded trust in the AZ vaccine. One study estimates that the suspension in Italy led to vaccine hesitancy and 662,760 lost doses in just three weeks.^[10] Another study (well, somebody's Bachelor's Thesis) found that Germans had lower confidence in AZ than in the other major Western vaccines.^[11] We have similar anecdotal reports from Australia, with one doctor saying: "The horse has already bolted [...] People talk about hesitancy or reluctance. It is well beyond that: it is a refusal."

Even more serious may be the fact that these events have damaged confidence in AZ in the developing world, which is sorely in need of effective vaccines:

In Malawi, people are asking doctors how to flush the AstraZeneca vaccine from their bodies. In South Africa, health officials have stopped giving the Johnson & Johnson shot, two months after dropping the AstraZeneca vaccine. And in the Democratic Republic of Congo, 1.7 million AstraZeneca doses have gone unused. [...] "We fought so hard with vaccine messaging, but what has happened this past week has brought us back to square zero." [...] Health officials fear that any setbacks in vaccinations could sow the seeds of the next calamitous outbreak, one that deluges hospitals and exports new mutations around the world. In those places, doctors said, the math is obvious: Many more people will die without the AstraZeneca and Johnson & Johnson vaccines than with them.

Verdict: True.

Discussion: I was quite certain of this due to the extreme rarity of the AZ side-effects. We knew that the AZ vaccine was effective, and that many more people were at risk of getting COVID. So I think this was a good prediction.

Prediction #2: a Majority of Countries Will Resume AZ Vaccinations, 90%

As far as I can tell, out of the 16 countries that I mentioned as having suspended AZ vaccinations in my original post, at a minimum these ten have resumed AZ vaccinations:

Bulgaria
Cyprus
France
Germany
Ireland
Italy
Latvia
Netherlands
Portugal
Spain

Verdict: True.

Discussion: The fact that several countries chose not to suspend the AZ vaccine meant that resuming its use was a viable option for the suspending nations. Still, I might have been too confident here, because (1) some nations, like Denmark, had very low transmission rates, which would have changed the cost-benefit calculation, (2) countries may have been able to get ahold of more mRNA vaccines more quickly than I thought and (3) hesitancy towards the AZ vaccine may have reduced the benefits of resuming its use. Still, I think this was a solid prediction.

Prediction #3: the FDA Will Approve the AZ Vaccine, 95%

This did not happen. An article by Sarah Boseley provides some of the story. It seems that the initial trials were partly flawed, and partly tailored for scientific research, not for regulatory approval:

By the time AstraZeneca got involved, Oxford's scientists had already set up the early trials. That meant [...] that the studies were not tailored to the needs of regulators in the way that big drug companies would have done it.

[...]

Two things happened that would cause serious problems with regulators later on. Oxford had an extremely cautious approach to older people, and chose to recruit mostly under-60s for the earliest trials in the UK.

Second, there was a glitch in the production of vaccines for the studies. A contractor accidentally supplied half-doses, according to AstraZeneca's Sir Mene Pangalos [...] When they found out, the academic researchers told the Medicines and Healthcare products Regulatory Authority (MHRA) in the UK, and got the go-ahead to continue with two dosing strengths to see what happened.

When the trials reported, it turned out that volunteers given a half-dose followed by a full dose got more protection – up to 90%, compared with 62%.

Pangalos described it as serendipity. Regulatory bodies such as the Food and Drug Administration in the US don't like serendipity. They like predictability and no surprises. The oddity of the information sowed doubt at the FDA.

As it turned out, the increased efficacy was due to the larger gap between the two shots in that group, not the lower dosage. In September of last year, the FDA was unhappy about what they perceived as lack of transparency with regard to two cases of possible side-effects (these later proved to be unrelated to the vaccine); this slowed down trials significantly in the U.S. The article continues:

When the US trials eventually reported in March, AstraZeneca's top executives thought they were home and dry. They reported good results: 79% efficacy against symptomatic illness and 100% against deaths. They had hardly had time to pop a champagne cork before the world turned upside down again. They were accused by experts in the US of massaging the data to give a more favourable result.

The data safety monitoring board issued a statement accusing them of putting out "potentially misleading" figures.

[..]

[Anthony] Fauci appeared on Good Morning America, describing the data issue as "an unforced error [...] this kind of thing does nothing but really cast some doubt".

Oxford and AstraZeneca's scientists were astounded by the onslaught. They worked day and night to update the figures. Adding the very latest data pushed the overall efficacy down from 79% to 76%, which was barely a drop, and actually pushed up efficacy in the older age group from 80% to 85%. AstraZeneca's vaccine is still not licensed in the US.

Verdict: False.

Discussion: If I recall correctly, I mostly deferred to Metaculus here. On March 16th, the day that I published my predictions, the median prediction on Metaculus was April 16th, two and a half months ahead of July 1st. But I have only myself to blame; I was far too confident. For one, I was not really aware of the bad history between AZ and the FDA. I also did not consider the fact that another vaccine, also based on an adenovirus vector, was being ramped up in the U.S. around that time, reducing the need for AZ. Bad and lazy prediction.

Prediction #4: an Elected Official Will Admit that Suspension Was a Mistake, 70%

I did not really find any record of an official having stated this, though I am limited somewhat by my knowing only a few languages. To take one example, this was the closest that Jens Spahn, the Health Minister of Germany, came to admitting his mistake:

In announcing the government's latest policy change on AstraZeneca, Mr. Spahn did not make a scientific argument, though anyone under 60 who takes the shot will have to discuss the risks with a doctor. Instead, he emphasized the need for flexibility and getting more people vaccinated.

[...]

"We will make a lot more flexibility possible", Mr. Spahn told the public television station WDR on Wednesday. "Many people want to have their second shot earlier, with an eye on summer, and that is possible with Astra."

To the question of whether a suspension causes more deaths than it prevents, the website of the Bundesministerium für Gesundheit (the German Federal Ministry of Health) gave the following answer back in March (original German in the footnote):

This is a statistical (and legitimate) aspect. But the state is legally obliged in particular to the individual citizen who is vaccinated in a state vaccination campaign! The state provides the vaccine and thus has special duties of care. Officials of the [Federal Ministory of Health] and [Paul-Ehrlich-Institut] are obliged to monitor the safety of the vaccine and to react in case of relevant signals. If these duties are neglected and the vaccination campaign continues without properly informing the population and the persons to be vaccinated, there could also be legal consequences.^[12]

But that was an artless dodge, and not anything close to an admission of oversight.