I owe tremendous acknowledgments to Kelsey Piper, Oliver Habryka, Greg Lewis, and Ben Shaya. This post is built on their arguments and feedback (though I may have misunderstood them).

Update, May 13
I first wrote this post before investigating the impact of covid variants on vaccine effectiveness, listing the topic as a major caveat to my conclusions. I have now spent enough time (not that much, honestly)  looking into variants that I have a tentative position I'm acting on for now.

My moderately confident conclusion is that the current spread of variants in the US barely impacts vaccine effectiveness. The Pfizer vaccine is reported to be 85% as effective against the feared B.1.351 variant (South African) as it is against B.1.1.7 (UK). Assuming that other variants are no more resistant than B.1.351 on average (a reasonable assumption) and that presently variants are no more than 25% of Covid cases (in Alameda and San Francisco). The net effect is 0.25*0.85 + 0.75*1.0 = 0.9625. In other words, vaccines still have 96% of the effect they would if B.1.1.7 were the only variant.

Plus, that tiny reduction of vaccine effectiveness is dwarfed by the falling background prevalence of Covid. When I first wrote this post, Alameda and San Francisco were at 0.1-0.15%; now they're at ~0.05%. The same for New York and the United Kingdom. 

Although relaxing of restrictions might reverse this, right now, Covid-risk is very, very low in the Bay Area and many parts of the US.

All updates/changelog can be viewed here.

I plead before the Master of Cost-Benefit Ratios. “All year and longer I have followed your dictates. Please, Master, can I burn my microCovid spreadsheets? Can I bury my masks? Pour out my hand sanitizer as a libation to you? Please, I beseech thee.”

“Well, how good is your vaccine?” responds the Master. 

“Quite good!” I beg. “We’ve all heard the numbers, 90-95%. Even MicroCOVID.org has made it official: a 10x reduction for Pfizer and Moderna!” 

The Master of Cost-Benefit Ratio shakes his head. “It helps, it definitely helps, but don’t throw out that spreadsheet just yet. One meal at a crowded restaurant is enough to give even a vaccinated person hundreds of microCovids. Not to mention that your local prevalence could change by a factor of 5 in the next month or two, and that’d be half the gains from this vaccine of yours!”

I whimper. “But what if . . . what if vaccines were way better than 10x? What about a 100x reduction in the risks from COVID-19?” 

He smiles. “Then we could go back to talking about how fast you like to drive.”

In its most extreme form, I have heard it claimed that the vaccines provide 10x reduction against regular Covid, 100x against severe Covid, and 1000x against death. That is, for each rough increase in severity, you get 10x more protection.

This makes sense if we think of Covid as some kind of "state transition" model where there's a certain chance of moving from lesser to more severe states, and vaccines reduce the likelihood at each stage.

I think 10x at multiple stages is too much. By the time you're at 1000x reduction, model uncertainty is probably dominating. I feel more comfortable positing up to 100x, maybe 500x reduction. I dunno.

There is a more limited claim of extreme vaccine effectiveness that I will defend today:

1. In the case of the Pfizer vaccine (and likely Moderna too), the effectiveness in young healthy people is 99% against baseline symptomatic infection, or close to it.
2. We can reasonably expect the effectiveness of the vaccine against more severe cases of Covid to be greater than effectiveness against milder cases of Covid.

(Maybe it's 2x more effective against severe-Covid and 3x more effective against death compared to just getting it at all. Something like that, it doesn't have to be 10x– it'd still be a big deal because more severe outcomes are where most of the disutility lies.)

It's a very simple argument, really. First, the data very clearly suggests effectiveness of ~99% for young people, with nice tight confidence intervals. Second, across all the data we see trends of increasing effectiveness against increasing severity, granted that the confidence intervals are wide in some cases. Third, a very reasonable (imo) mechanistic model supports this interpretation of the data.

The 1.2 Million-Person Pfizer Israeli Observational Study

This observational study matched ~600k vaccinated people 1:1 with ~600k demographically similar controls. It covered the period December 20 to February 1. As far as I know, it is by far the largest Covid-19 vaccine study published to date. The other studies are clinical trials with sample sizes on the order of 20k-40k, and some other observational studies, typically with healthcare workers, in the single-digit thousands.

Why it's not as big as it sounds

Before we get to looking at data, I think it's important to note why uncertainty remains despite the huge N. To start with, the outcomes are all quite rare. Eyeballing it, Israel had a Covid-19 prevalence of ~0.5% during the study period. Out of a million people, a few thousand might be expected to actually catch Covid. Of that few thousand, only dozens or hundreds will progress to more severe forms of Covid. When sample sizes are in the dozens, confidence intervals are wide.

The authors report:

During a mean follow-up of 15 days (interquartile range, 5 to 25), 10,561 infections [6,101 control vs 4,460 vaccinated]  were documented . . . of which 5996 [2494 vs 2,071](57%) were symptomatic Covid-19 illness, 369 required hospitalization [259 vs 110], 229 were severe cases of Covid-19 [174 vs 55], and 41 resulted in death [32 vs 9].

See Appendix B for complete breakdown of outcomes by period and vaccination status. 

What's more, those numbers are for the entire study period (44 days). Only in a subset of days had participants been vaccinated long enough for it to be a real test. Nominally it is a 1.2 million person study, but practically, when you're looking at results 2 weeks after the first dose (~day 14) or one week after the second dose (~day 28), the numbers are much lower. ~80% lower.

All that to say, sample sizes aren't as big as they sound. Well, let's look at the results. This is the main outcome table. Definitions in Appendix A.

It's a bit hard to track trends formatted like that, so here's an equivalent graph:

To me, the headline result is that  2nd-dose + 7-days, vaccine effectiveness against Symptomatic Illness is 94% (87-98). Pretty good! Also, efficacy clearly rises from the earlier to later periods after vaccine administration.

Unfortunately, we don't see efficacy improvements moving to the right on the rightmost graph (more severe outcomes), counter to claims of extreme vaccine effectiveness. We do see upwards-right trends in the earlier periods (left and middle graphs). 

Well, 2 out of 3 ain't bad! Too bad the last one is the one we care about most.

It's okay. I've got more. The astute reader will have noted that the above graphs have "subgroup = Full" in their title. The study made available endpoints (outcomes) for multiple subgroups. Below are some of them. (Here are ALL OF THEM.)

Since there was data for it, I also added in the "2nd dose and 6 days after" period.

Many of the points are missing. The authors did not compute vaccine effectiveness if the control and vaccine group combined did not have 10 or more instances of an outcome. For example, the Age 16-39 subgroup did not have 10 instances of hospitalization, severe-Covid, or death for almost all of the time periods.

In cases where there are 10 instances of an outcome in the control group but 0 in the vaccine group, a value is reported without confidence intervals, e.g. the dots in the Females Subgroup, 2nd-2nd+6 period.

The up-and-to-right shape of the graphs persists across subgroups, except for Males and when the values are already maxing out near 100% (the right-most graphs). Overall, I think this is suggestive of the general trend that vaccines are more effective against progressively more severe outcomes. I also suspect that an uncertainty modeling that took account of the not uncorrelated neighboring values would shrink the error bars beyond the naive bootstrap method.

I'll comment more on why the flat/missing trend in the rightmost graph doesn't bother me much, beyond the fact that sample size means it's hard for it to show much at all.

Also! If you prefer Tables, here's the top half from the paper itself corresponding to the above graphs:

As someone falling in the Age 16-39 subpopulation, I'm quite pleased to see 99% effectiveness against Symptomatic Infection, with a nice tight 96-100 confidence interval. This is higher than I'd had anyone cite to me, and is approximately a 5x increase in how effective I believe my Pfizer vaccine to be. That's even before we get to more severe outcomes.

So why is the absence of data showing increasing efficacy not evidence of absence?

Because we expect to see data that looks like this even in worlds where the vaccine is 100% effective (at least for all vaguely healthy people). To be evidence against something, it has to be less common in worlds where that thing is true, and that's not the case here.

Why would we see these numbers with a 100% effective vaccine?

"Saturation" and "noise" 

When you have 100 true positives and 3 false positives, the false positives aren't such a big deal. When you have 0 true positives and 3 false positives, the false positives can change the entire picture.

I argue this is very likely what is going on with Covid-vaccine effectiveness, above and beyond sample sizes. 

Consider that PCR Covid-19 tests have both a false negative and false positive rate (FPR). According to this random site I found by Googling that looks legit enough, the FPR for Covid-19 tests is between 0.2% and 0.9%. Let’s choose a point estimate of 1% for the FPR to be safe, but run it twice for every case to compensate. So now our FPR is 0.01% 

Let’s now imagine using this test on a 99% effective vaccine (the same argument holds for 99.9% and 99.99% even more so). We run an RCT with 100,000 people receiving the vaccine and 100,000 receiving placebo. Covid-19 prevalence is 0.1% in the region our hypothetical test is running. 

99 people from the control group catch actual Covid and receive positive test results (we lose one to a realistic false-negative rate of 10%, run twice to become 1%) plus 0.01% false positives for a total of 109. From the treatment group with 99% effective vaccine, we get 1 true-positive and 10 false-positive test results. Our final effectiveness estimate is 1 - 11/109 = 90% 

90%! And that’s from what is in truth a 99% effective vaccine. The control is mostly unaffected by the noise (109 vs actual 100) but the treatment is enormously changed. Instead of 1, it’s 11. 

The greatest noise of all is selection effects

Never mind fluke false positive tests, on priors we have reason to suspect that the people who are still getting severe Covid despite being vaccinated or even hospitalized are very likely not like you. Why isn’t the number of vaccinated people who ended up in critical condition zero?

Because within the Israel Observation study, the vaccinated group contains some very sick people. (In a half-million person group without exclusion criteria against it there simply will be, and that’s before the fact that we know many people are elderly and many explicitly meet risk criteria such as cancer, obesity, pulmonary disease, Type 2 diabetes, etc. See Table 1. Demographic and Clinical Characteristics of Vaccinated Persons and Unvaccinated Controls at Baseline.)

Vaccines work by stimulating an immune response. If your immune system is in tatters, unable to manufacture healthy antibodies or something, your vaccine might not do much. You might fare little better than the unvaccinated.

There’s actually a term for immune system failure in the elderly: immunosenescence. A couple of papers on that topic: Immunosenescence and vaccine failure in the elderly (2009) and Immunosenescence: A systems-level overview of immune cell biology and strategies for improving vaccine responses (2019)

Quoting the abstract from the first one:

An age-related decline in immune responses in the elderly results in greater susceptibility to infection and reduced responses to vaccination. This decline in immune function affects both innate and adaptive immune systems . . . Essential features of immunosenescence include: reduced natural killer cell cytotoxicity on a per cell basis; reduced number and function of dendritic cells in blood; decreased pools of naive T and B cells; and increases in the number of memory and effector T and B cells . . . Consequently, vaccine responsiveness is compromised in the elderly, especially frail patients . . . In the future, the development and use of markers of immunosenescence to identify patients who may have impaired responses to vaccination, as well as the use of end-points other than antibody titers to assess vaccine efficacy, may help to reduce morbidity and mortality due to infections in the elderly.

The last line there suggested antibodies can be present even when a vaccine overall is not protective against disease.

Don’t get old kids, it’s bad for you.

When I see that the treatment group has lower but not zero severe cases than the control group, I assume it’s coming from the very ill, the immunocompromised and immunosenescent.

I am pretty confident that I am not in those groups. I’m pretty sure my vaccine elicited some real response (I had some side effects, not terrible, but some). When I see the effectiveness numbers showing globally that there’s still some chance of really bad outcomes, I adjust them downwards because they were very likely not happening to people with remotely my level of health. 

To invoke the math of the previous section regarding false positives, the immunocompromised might be only 10% to the number of severe Covid patients in the control group (because their prevalence is low), but after the filter/selection effect of vaccination, they make up 95% of severe cases in treatment group.

Here’s a diagram for good measure:

This is a causal diagram. Arrows indicate the direction of causality but not the specific causal relationship. (Having a poor immune system, e.g. because you’re older, might make you more likely to be vaccinated, but of course, in our study we’re conditioning on vaccination status, so it doesn’t matter.)

Since we didn’t condition on immune system health, we can’t expect that a naive interpretation of the table of efficacies (above) tells us the true relationship between vaccines and outcomes.

Well, we approximately can condition. The vaccine efficacy is higher for the Age 16-39yr subgroup (99%, 96-100) vs the entire population (94%, 87-98). It's not just that younger people get less Covid, but that the vaccine worked better on them.

The Age 16-39yr subgroup didn't have zero cases of symptomatic disease (it might have zero of hospitalization, etc), but as above, I'm guessing those were almost all people who knowable had weak immune systems within the overall healthier group too.

Priors & Trends

In the above section I argued two things:

• that we see something of a trend towards increasing effectiveness with increasing severity in the Pfizer Israeli mass study
• Reasons why even if the vaccine was 99.9% effective, we would expect to observe effectiveness data that is lower than this.

In this section, I want to offer 1) a plausible mechanistic model for why this should be true, 2) further indications of increasing effectiveness from other trials and studies.

Shifting the Distribution of “Infection”/Viral Load

Epistemic status: I'm not a biology/virology/immunology person and this feels kinda hand-wavy to me.

This post started with a table that lists a progression of discrete outcomes: Documented Infection, Symptomatic, Hospitalization, Severe Disease, Death . . . and I’ve been referring to them since. Obviously, the underlying biological reality isn’t quite as discrete as that.

It’s probably more something like there’s a continuous value of how infected you are, and the higher that value gets, the worse your condition will be.

How infected you are is probably a fuzzy thing in reality, but viral load might be an adequate proxy. It’s been documented that viral load varies together with Covid-19 severity. See SARS-CoV-2 viral load is associated with increased disease severity and mortality and Saliva viral load is a dynamic unifying correlate of COVID-19 severity and mortality

The graphs in the second paper particularly make this point. 

Presumably, your viral load is the result of competition between the virus replicating and your immune system fighting it. Vaccination gives a significant boost to your immune system. (Cf the oft-cited claim about “4x lower” viral load in vaccinated people)

!! To engage in some inexpert armchair speculation, I’d guess that in the virus-immune system race, about x% of the time the virus gets the upper hand with an infected person and makes them symptomatic (~Level 1), and then in x% of cases where it got to Level 1, the virus wins out again to progress to Level 2 before the immune system can stop it. The virus progresses two levels x^2% of the time. If x is 20%, then overall 4% to get to two levels worse.

At each level, the virus only has an x% of winning out and progressing to the next level. Alternatively, in each time period, there’s some chance, y%, that the immune system will catch up and win. 

In cases where someone has a weak immune system (high x%, small y%), increasing levels of case severity aren’t much less likely than earlier ones. You might get an approx flat effectiveness curve.

But suppose that someone is vaccinated and has a real boost to their immune system. Intuitively, I’d reckon they’re now at x/5% for each stage. For the virus to progress two levels, it’s (x/5)^2, or 0.16% when x=20%.

Maybe it’s a factor 2 or 3 instead of 5, but either way, it’d be a compounding effect. More severity means the virus has to replicate more times, which is more time for the immune system to catch up and beat it, ergo less chance for it to get that bad.

It's tough being a dude

Incidentally, Silva et al (2021) who provided the graphs of viral load immediately above, also had this to say re male vs female:
 

The difference in viral loads lines up with the Male subgroup having worse vaccine efficacy than the Female subgroup: 88% (71-98) vs 96% (90-100) against symptomatic infections, 2nd dose+7 . It is also the case that across the world, women live longer and die less often from cardiovascular disease, cancer, diabetes, and chronic respiratory disease (Our World in Data).

Johnson & Johnson & Friends

Although not an mRNA vaccine like today's star, Pfizer's BNT162b2, the J&J clinical trial tracked multiple endpoints that show our hoped-for trend.

• ~40,000 participants (treatment and control)
• 65% of cohort is 18-59yr

The design/reporting of the J&J clinical trial differs from others, particularly the large Pfizer observational study. The time periods and outcomes are defined differently. 

To avoid copying multiple tables, I’ve extracted the numbers as I’ve understood them.

• Moderate Covid-19: Positive test, any 1 really bad symptom or 2 of regular symptoms like fever, sore throat, cough
• Severe/Critical: 3+ regular symptoms or  things like respiratory rate ≥30 breaths/minute, heart rate ≥125 beats/minute, oxygen saturation (SpO2) ≤93%, shock, admission to ICU, death
• Requiring Medical Intervention: hospitalization, ICU admission, mechanical ventilation, and/or ECMO. 

There were deaths in the J&J study, all within the placebo group:

Using a somewhat different formula, the authors also report on interim asymptomatic results. They present four different operationalizations of which I choose two, the ones with the highest and lowest efficacy after 29 days. See Table 20 for further detail.

Despite the J&J trial using overlapping criteria, both explicitly lumping things and in their criteria, we see the progression we’d expect to see if vaccines work better to prevent worse outcomes than they do milder ones. Modulo confidence intervals, that is.

While efficacy against moderate to severe Covid-19 is 65% (for 18-59 years old), it jumps to 85% for severe alone, granted the overlap in confidence intervals. 0 cases in the vaccine group fell into the Requiring Medical Intervention endpoint, compared to 7 with placebo. It’s as good as we could hope to see with this data.

However, there isn’t a clear jump between “asymptomatic” and “moderate to severe”. Partly because the operationalization isn’t clear (59.7% to 66% is a jump) but there are still wide error bars. The After Day 29 antibody test was conducted at Day 71 and had only been completed for ~30% of participants at the time of publication.

On net, I think the overall endpoint trend lines up with increasing vaccine efficacy against more severe outcomes, but the asymptomatic vs symptomatic doesn't clearly show it, but part of that is I don't really understand the groups or how to interpret them.

Let's get clinical, clinical

Moderna Clinical Trial

The Moderna Phase III Trial tracked Covid and severe Covid but no other end points. There were no severe Covid or deaths in the vaccine group, but 30 severe Covid cases in control and one death. Doesn’t let you compute a precise value, but is consistent with the vaccines being very good.

• 59% of cohort is aged 18-64 yr and not in any risk categories
• Further breakdown of participant characteristics in Table 1
• Asymptomatic Infections and Hospitalizations were not tracked.
• Table S13 in the appendix provides a very detailed breakdown of symptoms. No severe symptoms at all occured to participants in the treatment group, granted that very few occurred in the control group either.

Pfizer Clinical Trial

• 58% of cohort is aged 16-55yr, median 52yr, range 16-89
• 35% have BMI >= 30
• After the second dose, there 5 cases of severe-Covid in the control group and 1 in treatment, no reported deaths.

It’s nice (for me) to note that the vaccine efficacy, and particularly the confidence intervals, are higher for the 16-55yr group. 95.6% with CI 89.4%-98.6%. Doesn’t prove the main point, but is in line with the immunosenescence model.

No Covid deaths are reported in either placebo or vaccine groups. Table S5 from the appendix least severe-Covid outcomes. A total of 9 after the first dose in control, 1 in the vaccine group for a 89% reduction with confidence interval between 20.1%-99.7%. If we break it down to different time periods (before/after 1st/2nd dose), we end up with confidence intervals (-3800% to 100%). Yes, maybe taking the vaccine will increase your chance of severe-Covid by 39x!

As expected, the data doesn’t show that the reduction in severe-Covid is greater than lesser-Covid, but it also they doesn’t show the opposite either.

As described above, we have to note the greatly reduced sample size in the later periods.

Evidence of Increased Asymptomatic/Symptomatic Ratios

This was already shown in the mass Pfizer study, but several other sources indicate the ratio of asymptomatic-to-symptomatic cases is increased for vaccinated people. In other words, vaccination works better against symptomatic Covid (more severe) than asymptomatic Covid (less severe). This is at earlier side of "severity", compared to hospitalization, severe-Covid, and death, but it suggests the same trend–plus there's more data than when looking at more severe outcomes.

I've copied most of the numbers here form MicroCOVID.org, see their analysis for calculation detail.

Pfizer Asymptomatic percentage goes from 48%->65%, Moderna from 17%->50%, J&J from 34%-57%, AstraZeneca 23%->40%

I expect the very different absolute numbers to come from the widely varying study methodologies as much as differences between the vaccines. (AstraZeneca uses home tests, for example.) 

I didn't exhaustively look through all possible sources of asymptomatic vs symptomatic efficacy. I would be very interested if someone had a credible source not showing this trend.

I also didn't scrutinize these calculations much, so I wouldn't be that surprised if it turned out there were deep flaws that undermine the trend seen here.

It gets better

If we go back to the big Israeli Pfizer observational study, we see increasing vaccine effectiveness as more time passes since first/second dose. Unfortunately, the study didn't have enough time/data to show us things two weeks after the 2nd dose.

Fortunately, there was a follow-up. On March 11, Pfizer/Israel Ministry of Health made the following press release:

Findings from the analysis were derived from de-identified aggregate Israel MoH surveillance data collected between January 17 and March 6, 2021, when the Pfizer-BioNTech COVID-19 Vaccine was the only vaccine available in the country and when the more transmissible B.1.1.7 variant of SARS-CoV-2 (formerly referred to as the U.K. variant) was the dominant strain. Vaccine effectiveness was at least 97% against symptomatic COVID-19 cases, hospitalizations, severe and critical hospitalizations, and deaths. Furthermore, the analysis found a vaccine effectiveness of 94% against asymptomatic SARS-CoV-2 infections. For all outcomes, vaccine effectiveness was measured from two weeks after the second dose.

The lack of actual paper makes this a little harder to interpret, but I don’t find it surprising given that (1) at this later date in their roll-out, an even greater proportion of people will be young and healthy, (2) this data is only counting two weeks after the second dose, whereas the previous large observational study only had a “7 days after 2nd dose until end of follow-up” (maximum of day 28 to 44).

And, of no small significance, the Pfizer vaccine appears fully effective against the UK variant. (Yay!!)

If the vaccine is showing 97+% amongst everyone, I would expect that's at least as true when you filter for younger/healthier people and filter out those with comorbidities. 

What I believe

I believe that what I wrote above supports my initial assertions:

1. In the case of the Pfizer vaccine (and likely Moderna too), the effectiveness in young healthy people is 99% or close to it.
2. That we can reasonably expect the effectiveness of the vaccine against more severe Covid to be greater than effectiveness against milder cases of Covid.

The initial Pfizer mass study has 99% (96-100) for the age 16-39yr group, and the subsequent follow-up gives 97% for everyone. At baseline for symptomatic cases, we're talking 30-100x reductions, which is hella good.

Further, across multiple studies, vaccines, and outcomes we see trends of increasing effectiveness against progressively severe outcomes. In some cases, we don't definitively see it, but that's easily attributable to lack of sample size and inherent limitations in methodology due to noise and selection effects.

If we're talking 99% against symptomatic cases (100x reduction), then I think it's reasonable to expect at least that for hospitalization, 99.5% (200x). Hence the title, extreme vaccine effectiveness.

What about J&J (and AstraZeneca)? Granted, The effectiveness numbers for J&J look lower than for Pfizer and Moderna, but I think they're higher than MicroCOVID.org's numbers imply. First, we get 85% (61.2-95.4) effectiveness against Severe/Critical in the 18-59yr subgroup. That number matters more. Second, that is a very wide age range. I would bet that restricting it to 18-39 would show an improvement relevant to most of those reading this. Lastly, I suspect all the factors mentioned above (selection effects, noise/saturation) to affect it and make the result lower than it would be otherwise.

On net, J&J might not be quite as extremely effective as the mRNA vaccines, but it's no pushover either.

AstraZeneca isn't on offer in the Bay, and was recently abandoned in my home country of Australia too, so I apologize for not examining it.

Tell me where I’m wrong

I want the case I've made today to be true, but even more than that I want to believe true things (and I certainly don't want people to believe false things because of me). If you think any of this is wrong. PLEASE SAY SO. 

Many thanks!

Objections

Maybe the vaccine and control groups are different?

The Mass Israeli observation study is not an RCT. Instead, it's looking at their database of vaccinated people and then trying to match each of them to someone who's mostly the same except not vaccinated.

This might be hard to do. The kinds of people who seek out vaccines ("health-seeking" behavior) are probably a bit different from those who haven't gotten around to it. Maybe they wear better masks (or masks at all) on the hand, and maybe they risk-compensate on the other. 
 

The authors attempt to control for this by 1:1 matching vaccinated people and controls on multiple characteristics. They cite as evidence this worked the fact the Covid cases among vaccinated and non-vaccinated groups are the same before Day 12 (before the vaccine really starts working), and explain the initial gap in the first couple of days as a selection effect, because vaccinated people only choose to get vaccinated when they're feeling well.

I didn't look at the data especially long, but I'm not sure what to make of it. If vaccinated people are more cautious, even after being vaccinated, this means vaccines are less effective than we'd otherwise think. If they risk-compensate, that means vaccines are more effective than what we observe.

I'm not sure how this should net out, but I do think it should widen confidence intervals somewhat.

I also expect the effect to get stronger over time in Israel, as the people who remain unvaccinated and serve as controls (in observational studies) who least cared about Covid and getting vaccinated. So...maybe that explains the later 97% effectiveness result.

 Ben Shaya's Thoughts

Ben Shaya, one of the people responsible for MicroCOVID.org's models, has been kindly taking time to discuss the topic of vaccine effectiveness with me. 

Here wrote a document arguing Why I think vaccines don’t bring the chance of severe COVID to 0. While that's a stronger claim than I would make, his arguments and models are still valuable when thinking about the topic generally. 

Read them all in his doc, but I'll highlight one that I found very relevant to mechanistic models of Covid severity:

There's is a bit of mechanistic nuance - COVID tends to start with an upper respiratory tract infection - that's when nasal swabs work, and then moves to the lower respiratory tract (lungs) - which is where is screws over your blood oxygen. There's research suggesting that the immune response of the lining of your windpipe is what determines whether the virus reaches your lungs: https://www.biorxiv.org/content/10.1101/2021.02.20.431155v1.abstract?%3Fcollection=

That is to say, there isn't a single "immune response" that determines how bad your infection is; immune response in your upper respiratory tract determines if you get COVID while immune response in your lungs and windpipe determine whether you get a severe reaction. Immune response in your blood correlates with these, but it's not the same as either.

(this comes from Riley Drake, who is a virologists and one of the authors on the paper, who strongly cautioned against reading too deeply into antibody concentrations as a proxy for immunity)

When we treat these three immune systems as separate, we see there are at least 3 hidden models - upper respiratory response, lower respiratory response, and blood response. Of the 3, the side effects to the vaccine only reflect the blood response. The overall efficacy of the vaccine only reflects the upper response.

Further, we know that humans have significant heterogeneity in all thee of these responses, since only some people get severe covid, and some people get lung damage from mild covid, and some people get exposed to COVID and don't contract the virus at all.

Note further that, assuming a strong immune response protects you is also not necessarily true; people with very strong response can cause cytokine storms (which we now know how to handle, but will land you in a hospital).

This gets at more gears in immune response and is the kind of thing that can expose where simple state transition models and immune response as a single thing don't hold up.

What if the vaccine boost one part of the immune system but not another? In that case, you might see the vaccine be very effective against symptomatic and asymptomatic Covid, but not more severe disease. If  the virus makes its way deep into your lungs, and the lungs are protected by a different immune response that isn't helped by the vaccine– then, conditional on having gotten to that point, you might not be better off than a non-vaccinated person.

All this to say we should be cautious in putting too much stock in simple mechanistic models.

What if vaccines are all or nothing?

The model behind the claim of extreme vaccine efficacy is that even if your post-vaccine immune response isn’t enough to stop you getting Covid at all, it should be stronger than it would have otherwise been, and you’ll do better at fighting off severe-Covid. This takes vaccine efficacy as a continuous thing.

But maybe the vaccine is 100% effective against all outcomes! So long as it’s correctly transported and administered, that is. Except sometimes vaccines are left at high temperature for too long, the delicate proteins are damaged, and people receiving them are effectively not vaccinated. If this happens 5% of the time, then 95% of people are completely immune to Covid and 5% are identical to not be vaccinated. Whatever chance they had of getting severe Covid before, it’s the same now.

In this world, not knowing whether your vaccine was a dud or not, post-vaccine you should assume you have a 95% reduction across all outcomes equally. 

I originally found this argument very persuasive. How could assume that the “continuous” model of vaccine-immune response was true? But actually, most things are continuous in the real world, particularly in biology. People aren’t old or young, but somewhere on a continuous measure. Immune response isn’t an all or nothing affair, and some people’s body’s will produce more antibodies than others. Some enough to stymy any symptoms at all, but some only enough to prevent them getting hospitalized.

One person told me that the mRNA vaccines induce 60x the antibodies of a recovered Covid patient (closest source I found), such that even if your response was weaker, it should still be more than powerful to deal with any actual Covid. Therefore, we should take people still getting quite sick with Covid as a sign that some people’s vaccines must not be working at all.

I would be surprised if complete vaccine failure didn't occur some of the time, the question is how much. Suppose we have a vaccine that, when it works successfully, confers 100x reduction (99% efficacy). If it fails to work at all 5% of the time, we'd see a
1-(0.01*0.95 + 1*0.05) = 94% efficacy overall. Something like that could be a big part of what we observe.

Also, Kelsey Piper says that she couldn't find anything about any other vaccines working this way (all or nothing).  [Edited: However, as Romeo points out in the comments, Moderna and Pfizer are mRNA vaccines that might not behave like the older types.]

Caveats – read before you act

In my long case for the extreme effectiveness of vaccines, there are some topics of crucial practical importance. These imply that maybe you don't want to throw caution to the wind just yet. I'm not sure, I didn't get to looking into these.

Long-Covid

Long-Covid is not an endpoint tracked by any of the studies I've looked at. I would think that'd be related to increased viral load and behave like an outcome more severe than just a symptomatic case, but there isn't data for that. Anecdotally, I've heard of a couple of cases where someone experienced mild Covid yet was dramatically affected for months afterwards.

My conservative gut estimate is that your odds or getting long-Covid are reduced by a vaccine by as much as your chance of getting symptomatic Covid at all, but not necessarily any more than that.

Variants

The Israeli study showed supreme efficacy even against the much-feared UK variant (B.1.1.7). However, there are fears the vaccines aren't not nearly as good against the South African variant (B.1.351) or Brazilian P.1 variant.

A new study published on MedRxiv last week, Evidence for increased breakthrough rates of SARS-CoV-2 variants of concern in BNT162b2 mRNA vaccinated individuals, state that vaccinated individuals were more likely to contract B.1.1.7 and B.1.351 than unvaccinated controls, suggesting both variants are more resistant to vaccination than other strains. (Caveat: I didn't read the paper in detail.)

. . . we performed a case-control study that examined whether BNT162b2 vaccinees with documented SARS-CoV-2 infection were more likely to become infected with B.1.1.7 or B.1.351 compared with unvaccinated individuals. 

Vaccinees infected at least a week after the second dose were disproportionally infected with B.1.351 (odds ratio of 8:1). Those infected between two weeks after the first dose and one week after the second dose, were disproportionally infected by B.1.1.7 (odds ratio of 26:10), suggesting reduced vaccine effectiveness against both VOCs under different dosage/timing conditions.

Not good.

The CDC's CovidTracker page actually has some nice dashboards for tracking variant proportion, though I haven't looked into the data quality. Their brief is helpful too.

There's a breakdown by US state too. I'm in California and am pleased to see that currently, B.1.351 is only 0.3% of the Covid cases and P.1 is 1.6%

Of course, if B.1.351 and friends are resistant to vaccination, we will see them rise in prevalence.

This needs more investigation. Without looking into it more, the sensible strategy would be something like act according to your local prevalences and beliefs about how vaccine-resistant the different strains are. Right now the suspicions are on B.1.351 and P.1, but they're uncommon in California (0.3% and 1.6% respectively).

If you've got the time and skill to look into this more, please do, I can provide you money and glory.

Spreading to the Unvaccinated

Even if the vaccine protects you 200x against more severe outcomes, that doesn't help the unvaccinated if they catch Covid from you when you had an asymptomatic or mild case. This means that until such time as those you interact with most are vaccinated, you might want to be more conservative in your microCovids.

MicroCOVID.org calculated reductions in contagiousness for vaccinated people (10x reduction for Pfizer/Moderna, 3x for J&J), and that's what I'd stick to myself if interacting with the unvaccinated. (But hey, California is just about at universal eligibility, now is the time!)