A vastly faster vaccine rollout

85

Coronavirus
Frontpage

When a traveler introduced smallpox to New York City in 1947, the city—and in particular its health commissioner, Israel Weinstein—apparently ran an epic vaccination campaign, reaching 5 million people in the first two weeks.1 That is, four hundred thousand vaccinations per day. San Francisco in two days.

For covid, the first New York City vaccine was given on the 14th of December, and if I understand, by the 10th of January, twenty seven days later, 203,181 doses had reportedly been given. That’s around eight thousand doses per day. A factor of fifty fewer.

That’s a pretty incredible difference. Why is New York fifty times slower at delivering covid vaccines in 2021 than it was at delivering smallpox vaccines in 1947?

Part of the answer is presumably ‘regression to the mean’: if thousands of different cities at different times try to roll out vaccinations quickly, with a similar basic ability to do so, and there is some random chance in how well it goes, then the one we tell stories about seventy years later will one that got surprisingly lucky. You shouldn’t expect your own effort to go as well as that best one. But—without having done the statistics—I don’t think you should expect your attempt to be fifty times worse. New York didn’t get that lucky.

Perhaps there are other differences in the problems faced. For instance, the current vaccine needs refrigeration, or we are so late in the disease spread that we can’t all crowd together in around fast-moving vaccinators, or be rounded up in our natural crowded habitats, like classrooms or offices.

Though the 1947 situation looks harder in important ways too. For one thing, there was no time to prepare. The vaccination project began the day that the disease was identified to health commissioner Weinstein. By 2pm, he was apparently holding a news conference publicly asking residents to be vaccinated. With covid, there were about ten months to prepare. For another thing, now people have smartphones with which they can be alerted, and computers and online systems that might be used to coordinate them and tell them what to do.

I heard about this episode from the Rachel Maddow Show, and read about it in the New York Times, both of which bring it up to note the inadequacy in the current vaccine efforts. The New York Times says a rollout like this, “almost certainly couldn’t happen today”, and offers some explanations:

  1. Complicated relationships between city and other governments these days

    “In 1947, the city was able to act alone, as opposed to navigating a complicated relationship with the governor of New York and the federal government,” said Dr. Irwin Redlener, director of the Pandemic Resource and Response Initiative at Columbia University’s Earth Institute. “The city was able to say, ‘We’re going after this,’ and then make it happen.”

  2. A ‘hollowing out’ of the public health infrastructure

    But this time, with the coronavirus pandemic, New York faces a logistical hurdle. Experts in infectious disease point to a hollowing out of the public health infrastructure — not just in the city, but across the country.

  3. Lack of public faith in medical science, government and the media

    “This was the height of polio in the United States,” he said. “People had a much better sense of the impact of infectious disease. They saw it all the time, and they were rightly fearful. But they were also optimistic that medical science could conquer this. In 1947, there was tremendous faith in the medical community, unlike today.”…

    …Yet, [infectious disease experts] believe the biggest obstacle is not distribution but the public’s distrust of government, science and the media.

    “We’re coming out of a train wreck of messaging,” Dr. Redlener said. “We’ve learned that politics is poison to a public health initiative, especially during a crisis. Honesty and straightforward, clear messaging are absolutely critical.”

    In 1947, Dr. Weinstein was the only voice with a megaphone. He spoke and people listened.

    “Back then, there was a much simpler media landscape,” Ms. Sherman said as she laid out the Ad Council’s campaign, which is due to kick off early next year. “In today’s environment, we’re dealing with a highly, highly fragmented media. We’ll be relying on micro-influencers who are the trusted voices.”

They seem to favor #3, noting that it is what ‘experts believe’. But it seems so implausible—am I totally misunderstanding the claim? On the current margin, if lack of public trust was making the vaccine rollout even ten times slower, wouldn’t we see campaigns begging us to go out and get the incredibly accessible vaccine, rather than seeing elderly people camping in outside queues to get vaccinated, and most people being told that they just can’t have a vaccine for months or a year? Perhaps they mean that ultimately the number of doses given out will be limited by public willingness to receive them? (Which seems surely true, but not necessarily the answer to an interesting question).

The NYT’s other suggestions don’t seem immediately wrong, but are too vague for me to understand. I guess I’d like to know how things went differently at an object level. At what point, in 1947, did someone decide that it was their job to gather a volunteer army to administer vaccines, for instance? Do people work in similar roles nowadays? Did they think of this? Did they expect to get into trouble for having a bunch of lay people giving injections? Do they have to fill out a lot of paperwork to recruit them, whereas in 1947 they would have just shouted for them in the street? (I don’t know.) If New York had had these constraints, would their vaccination campaign have looked like ours, or is there more to explain?

I suppose I have several questions:

  1. What is really going on, that can account for a near 50x slowdown?
  2. Why does the New York Times have such an unenlightening, vague, and seemingly wrong discussion so close to where one could have a genuinely interesting one?
  3. (Are these things related?)
  4. My guess after thinking about it for ten seconds is that the gap in speed is to do with more things being regulated and formalized. The difference in time it would take me to physically cause someone to have a cookie versus to legally sell someone a cookie seems huge enough for this kind of thing to account for large slowdowns, and I do expect creating a massive band of volunteer non-nurses to administer a vaccine to require time-consuming paperwork or to be straight up illegal. How does this explanation sound?

This isn’t just an idle inquiry, and shouldn’t be just another interesting story for another payload of political disapproval.

Naively extrapolating, New York City could be fully vaccinated in about seven weeks if we knew how to do what was done in 1947. At the current rate, which will presumably change, vaccinating everyone would take years.2

What if someone figured out how to replicate even part of what New York did before at Weinstein’s direction? In America alone, around three thousand people are dying each day now, as they wait for the vaccine. My boyfriend’s ninety year old grandmother in Vermont was diagnosed with covid last week. Her center was scheduled to begin vaccinating its residents this Wednesday.

(Regardless of what makes things slower these days, good on everyone who is working to forward the vaccination effort, and also those doing their best to make it appropriately safe. Good luck being fast and right.)

P.S. In fact the whole of America vaccinated fewer people in the first two weeks of having a covid vaccine than New York did in 1947:

Also interestingly, others are not doing better.

  1. My information about this is all from the New York Times, Wikipedia, and the Rachel Maddow Show 

  2. Naive extrapolation says six years, but this is especially naive since at that rate covid will speed things up by reaching many people before the vaccine does. Plus we should probably expect some speedup over time, if it was going to take that long. Or something different to happen. 

85

38 comments, sorted by Highlighting new comments since Today at 2:07 PM
New Comment

One key limitation for vaccines is supply, as others have noted. That certainly doesn't explain everything, but it does explain a lot. 

This obstacle was, of course, completely foreseeable, and we proposed a simple way to deal with the problem, which we presented to policymakers and even posted on Lesswrong, by the end of April. 

Thus beings our story.

Unfortunately, we couldn't get UK policymakers on board when we discussed it, and the US was doing "warp speed" and Congress wasn't going to allocate money for a new idea. 

We were told that in general policymakers wanted an idea published / peer reviewed before they'd take the idea more seriously, so we submitted a paper. At this point, as a bonus, Preprints.org refused to put the preprint online. (No, really. And they wouldn't explain.) 

We submitted it as a paper to Vaccine May 20th, and they sent it for review, we got it back mid-june, did revisions and resubmitted early July, then the journal changed its mind and said "your paper does not appear to conduct original research, thus it does not fit the criteria." After emailing to ask what they were doing, they relented and said we could cut the length in half and re-submit as an opinion piece.  

We went elsewhere, to a newer, open access, non-blinded review journal, and it was finally online in October, fully published: https://f1000research.com/articles/9-1154

I think that your experience trying to get this published would be well worth reading in extended form. Do you think that any of the pushback you received was justified, looking at your submissions from an "outside view?" I.e. regardless of the merit of your idea, was there anything about the way you presented it, who you presented it to, or the credentials/evidence/effort backing the paper that might explain why you didn't get it published until October? Is there anything you would have done differently?
 

I think it's amazing that you made this effort, so I'm just hoping we can learn something for the future from this outcome.

From an outside view, this piece was a bit weird, but much more straightforward pieces have gotten similarly frustrating responses from top journals. If I was optimizing for publishability rather than impact, I would do things differently and likely be more successful, but that would avoid the entire goal. (You can always come up with an approach tailored to a journal, and work on finding results that they will like, but that's part of the problem with the spate of poor science in the past decades.)

However, I definitely don't think it is particularly anomalous for the submissions process to take months, since academia isn't meant/set-up for rapid response publications. In general, for academic publications, rejections are slow, acceptances are even slower, and everything is uncertain. But I have learned on several occasions during 2020 that the top-journal publication process is even more dicey and frustrating than I expected, even when co-authoring with people who have done that type of publication repeatedly, and after successfully publishing many things in more niche journals.

Despite all of that, I don't think that my initial selection of a higher tier journal was a mistake, a priori, since I think that it was the most likely way for it to get actually significant policy attention, and the final publication in F1000 was far less impactful, and not only because of the lateness. (But F1000 has a clearly superior publication model.)

A response from @politicalmath, based on Smallpox: The Death of a Disease by  DA Henderson:

1) There were no phases, it was just "show up, get poked"
2) There were plenty of vaccines to go around. Countries typically had millions of smallpox vaccine doses ready to go just in case
3) with no lockdowns, they could go to schools / churches / offices & line people up
4) the smallpox vaccine was incredibly heat-stable. There were batches that were still efficacious after being stored for a year at 113F
5) the public health infrastructure had a lot of practice with mass vaccinations (they did them all the time in other countries)
And it did cause a city-wide panic. Smallpox has like a 20% fatality rate so people were pretty motivated to get the vaccine. All this is from this book
https://www.amazon.com/Smallpox-Disease-Inside-Eradicating-Worldwide-ebook/dp/B002FU5O54/

Do they have to fill out a lot of paperwork to recruit them

A medical worker who tried to sign up for vaccine administration recruitment was confronted with needing 21 pieces of paperwork showing that they had completed various trainings, many of which have nothing to do with vaccination.

My cursory reading has indicated that every part of the process is fractally like this. We are in a liability liquidity trap of some sort as a civilization.

A medical worker who tried to sign up for vaccine administration recruitment was confronted with needing 21 pieces of paperwork showing that they had completed various trainings, many of which have nothing to do with vaccination.

Care to paste a source link?

Originally, former health workers were being asked to provide 21 pieces of evidence before being allowed to take part, which included evidence of anti-terrorism training, fire safety and conflict resolution.

But the [British] Government came under huge pressure to get rid of the excessive bureaucracy, with Boris Johnson promising last Wednesday 'all such obstacles and all such pointless pettifoggery has been removed'.

It was reported last night that the number of requirements had been chopped down from 21 to 15, which critics said did not go far enough. NHS sources told MailOnline today there had been confusion and that medics looking to sign up now actually only need to provide 13 documents.

―Boyd, Connor (2021, January 11). Retired medics are still being asked to fill out THIRTEEN forms to join national Covid vaccine effort despite Boris' promise to 'blast away bureaucracy'. Mail Online. Retrieved from https://www.dailymail.co.uk/news/article-9134613/Retired-medics-asked-THIRTEEN-forms-join-Covid-vaccine-effort.html

[+][comment deleted]10d 2

This NYT article is from the 10th of January as well:

https://www.google.com/amp/s/www.nytimes.com/2021/01/10/nyregion/new-york-vaccine-guidelines.amp.html

It states that the state as a whole has received 1.2 million doses. NYC is about half the population of the state and has administered about half the doses. So estimate that it’s received about 600,000 doses, while NYC had received 4 million doses during the smallpox crisis. That accounts for maybe 80% of the disparity.

I believe Dryvax was the vaccine administered for smallpox in 1947 NYC, and it was made 50 years before that. So there was time to gain trust and produce enough to vaccinate 4 million people without any question of queues.

Despite the righteous anger at this process, what we’re trying to accomplish is a flippin’ scientific and logistical miracle: going from “new disease” to “global vaccine rollout” in less than a year. Could they have done that in 1947?

A few observations:

  1. Compared to 1947, there are many fewer things that you can do without permission, and many more people from whom you must seek permission. This naturally inhibits quick action.
  2. Institutional incentives are all pointed towards caution, conservatism, and proceduralism. Undertaking decisive action to deal with a new situation is discouraged, and therefore these institutions are full of people without the disposition to even try.

I don't have citations for these, but cf. Zvi's entire series on moral mazes.

To these, we add a corona-specific factor, that we do not have enough vaccine doses to vaccinate everyone, so we have to ration and prioritise our doses. This, in turn, opens up vaccine prioritisation to political input, which means that the two factors mentioned above come to dominate, and we wind up with a vaccine plan which is maximally cautious, procedural, and bureaucratic. If we had enough doses to actually vaccinate everyone, it would be much easier to sell a policy of just vaccinating whoever walks through the door.

But why don't we have enough doses to actually vaccinate everyone? The vaccines were designed almost a year ago; what we've done since then is test them to make sure they don't have bad side-effects etc. I find it hard to believe that we can't ramp up to a billion doses over the course of an entire year.

There’s been a dangerous impression communicated by the media over the last few months that these mRNA vaccines were “designed” a year ago, and that everything since then is just testing. That’s not really how drug development works, particularly for a brand new modality like this. It’s a multi-dimensional engineering challenge with many, many decisions downstream of the mRNA sequence that need to be made: how it’s manufactured, how much dose, how the sequence will be delivered in a carrier molecule, how the vaccine will be packaged. None of that can be done too far in advance, because the FDA approval is contingent on those details, and could require changes.

I'd be interested to hear more about this. I'm very skeptical, not of the claim that it's a multi-dimensional engineering challenge etc. but of the claim that it couldn't have been done substantially faster than it was.

Empirically speaking, this was already done substantially faster than any drug development process I can think of. The reason it was done relatively quickly was because as much work was done “at risk” as possible, supported by funding guarantees from the government. But that only gets you so far, and there just are steps in the development that need to be done sequentially, taking time. Happy to answer more specific questions.

Thanks! (Strong-upvoting for going against the conventional wisdom here, being polite, and willing to back it all up with discussion. I really hope you are right.)

OK, here goes: This diagram makes it seem that large-scale production only began after Phase II completed: https://www.ema.europa.eu/en/human-regulatory/overview/public-health-threats/coronavirus-disease-covid-19/treatments-vaccines/covid-19-vaccines-development-evaluation-approval-monitoring

Is that true? If not, when did large-scale production begin? I guess I want to know what sub-process took 10 months. Was it that they needed 10 months to build the new factories to build the vaccine, because old factories couldn't be repurposed? Was it that old factories could be repurposed but need new equipment which couldn't be 3D printed but had to be made in a traditional assembly line which needed to be purpose-built?

Why did it took from Feb 24 when Moderna shipped their vaccines to the NIH till Mar 16 till they vaccines where given to a patient?

Why did the phase 1 trial only start Mar 27?

What do you think about this: https://marginalrevolution.com/marginalrevolution/2021/01/fact-of-the-day-get-to-those-rooftops.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+marginalrevolution%2Ffeed+%28Marginal+Revolution%29

It claims that we could have had billions of doses several months ago, if we had been willing to pay the vaccine producers a few billion dollars more early on.

What is this "we" you are talking about? Vaccines aren't cotton masks, where anyone with $200 of equipment and materials can make a dozen of them. Especially the mRNA version, which requires different equipment than the annual flu vaccine.

Plus, there were X different vaccines being pursued as of March 2020. 6 of them have now been approved (in various countries worldwide), but how many didn't work out. The process of scaling from lab bench to factory is difficult and expensive, and that couldn't even start until it is know which ones are going to work. (Or if someone was willing to spend the $$$ to develop every candidate in parallel. I don't know how much that would have cost, but it didn't happen.)

We = humanity. Or the USA, if you prefer.

and that couldn't even start until it is know which ones are going to work.

First of all, we totally could have started whilst still uncertain about which ones were going to work. The cost of starting and then stopping the ramp-up is probably at least an order of magnitude less than the cost of actually producing a billion doses. Given the stakes involved, the rational thing for humanity to do would be to ramp up multiple vaccines in parallel, so that at least one would be ready to go when it is proven safe.

Secondly, we didn't need 10 months to know which ones were going to work. The FDA approval process is not the fastest way of determining whether a vaccine is sufficiently safe to be worth producing during a pandemic.

There were some government funded and philanthropic (gates I think) backstops, essentially guarantees of paying for losses if manufacturing before knowing if it worked turned out bad. There should have been more and faster, as pointed out by Cowen.

I remember reading around the beginning of the pandemic that Bill Gates was going to do exactly that: subsidize production of many different vaccine candidates with his own money, and accept the sunk cost for any vaccines that ended up not working. I haven't seen anything about this idea recently though, and it seems he has not been (at least publicly) behind any vaccine production efforts. Any idea why? To avoid perceived competition with Operation Warp Speed?

They can't ramp up until they are approved.  You don't make a billion doses only to discover a year later you aren't going to get paid for them.

You can get paid whether or not the vaccine gets approved by making contracts with governments that you provide the vaccine dosis that are not dependent on approval. 

It's just a matter of governments valuing vaccine doses enough to be willing to pay enough money for their production. 

I mean....sure.

But why would the government, or anyone, pay for a vaccine that couldn't be used? An unapproved vaccine does nothing for anyone.

Option value.  They're not paying for a vaccine that can't be used, they're paying for a vaccine that MIGHT be usable in the near- or medium-term future.  Paying for many options before approval so you're ready with volume for the one(s) that get approved is +EV in most scenarios.

Which is why we proposed exactly that, at the end of April: https://www.lesswrong.com/posts/uXb4gcDP2fgBPcMHJ/market-shaping-approaches-to-accelerate-covid-19-response-a

Now published: https://f1000research.com/articles/9-1154

Unfortunately, we couldn't get policymakers on board.

I'm honestly surprised by someone on LessWrong engaging in that kind of Black White thinking. If you are uncertain about whether or not a given vaccine gets approved, there's a probability of it getting approved and you can calculate expected utility given the cost you pay for the vaccines that might or might not be approved.

Is the idea of thinking in terms of probability new to you, or is there some way that using it in a political context is hard?

The question was "why didn't we ramp up sooner?".  I answered the question as best I knew how. Everyone is acting like I'm the one that decided to not use option pricing. 

And yes, government paying for things that might not work is hard. Besides generic mind killing, there are people that don't want the government to spend money and the people who don't believe in vaccines at all.  They all vote.

This is a problem that a more competent society would have solved easily.

How?

If you want to eliminate testing and government approval, I'd be willing to have that conversation. I don't think many would though.

Why would anyone pay to ramp up production on something that might not do anything? Or might even make thins worse?

Why would anyone pay to ramp up production on something that might not do anything?

Because it might do something sufficiently important that the benefit * probability of benefit outweighs the cost.

I think there are significant differences in manufacturing and transportation.  My understanding is that cowpox was common enough that vaccines for smallpox were made somewhat locally, and there was no constraint about how many doses could be available.  For COVID-19 vaccines, the necessity of distant manufacturing and limited doses added a LOT of politics in who gets it first.   And this political truth led to massive restriction in how it's delivered.

There are two parts to the 50x difference - number of doses available to doctors/nurses, and percentage of those doses actually injected (vs still in storage or stuck in the supply chain).  From what I can tell, many big cities have injected ~25% of the doses they have received.  I don't know how this compares to smallpox, as JIT manufacturing could be modeled as delivering 100% of available doses, or less than 1%.

edit: not local or JIT manufacturing, it seems.  But still no supply limits.  I find it easy to believe that the limited supply of vaccine is driving all of the slowness, both directly and indirectly.  Directly because we don't have enough, and indirectly because the fact that some people have to wait adds the politics and perception problems of "fairness" and "optimization" that hugely slows things down.

Am I being naive in thinking that most of the 50x comes from manufacturing the vaccine? In 1947 they had 650k vaccines ready to go, then got 7 pharma companies across the country to work round the clock to get them the rest. They were aiming to vaccinate 6.35 million, we are aiming to vaccinate 328 million just in the US (57x more doses needing to be manufactured). We'd expect to have more total capacity today of course but we have fewer companies doing the manufacturing.

I guess the mRNA vaccines are also more difficult to manufacture. Based on the cost the P/B is 5x more expensive than O/A viral vector. The latter is planning on producing more than twice as many in 2021 (I don't know how the size and number of facilities compare but Pfizer are the bigger company).

Say we were giving every dose procured by the US evenly spread across the country - how many would we be doing a day in NYC?

New York has apparently distributed 35% of the vaccine that it has. Maybe they are focusing on other bottlenecks? Though my naive guess would be that the main problems are that the staff at US agencies are more numerous, less-competent, more regulated, as part of the aging process of any bureaucracy, compounded by the declining prestige of governmental jobs.

My assumption is that the smallpox vaccination was first-come-first-served, and that they weren’t attempting to identify and vaccinate small, targeted subpopulations first. It’s the difference between spraying your lawn with weed killer and selectively hand-pulling weeds. It’s not surprising to me that the former is many times faster than the latter.

After spending months saying that social distancing is very important, the current vaccination efforts happen against the backdrop of social distancing. It's valuable to do vaccination in a way that doesn't result in crowded waiting rooms in which people get infected with COVID-19.

What I'd expect to see to scale up getting needles in arms (off the top of my head):

  • All major population centers with multiple vaccination centers (temporary tents, school gyms, etc)
    • Should go without saying that these places should be very strict about masks, distancing, ventilation, etc.
  • Drop-in vaccinations allowed, forcing appointments shouldn't be a high priority
    • but make it easy to see delays
    • delays mean you probably don't have enough vaccination centers
    • quickly built appointment and tracking systems will themselves fail enough to slow down vaccinations; keep it simple and low-tech
  • Don't worry so much about "fairness"
    • the goal is to get needles in arms, as quickly as possible (it doesn't much matter that the most vulnerable don't strictly come first if you're quickly vaccinating the majority of people)
    • handle vulnerable or non-mobile populations separately, by bringing the vaccine to them
    • cheaters (people jumping the queue) don't matter that much if you're vaccinating many quickly
  • Don't worry so much about tracking who's been vaccinated or not
    • assume people aren't so stupid as to get vaccinated multiple times; likely not a major problem
    • you can't force people to get vaccinated, so don't worry about it; make it easy for those that want it

That we're not seeing these sorts of things (in Canada, where I am) feels like the people making these decisions aren't really prioritizing vaccinating people. Maybe that'll change in a couple of months, but we've had a long time to prepare for actually having vaccines available, so again feels like a failure to really prioritize (as in "does this thing make it easier or harder to get vaccinated now?"). There are risks (of the "unknown unknown" variety) that can happen just by being so slow (e.g., yet-another-variant that requires yet-another-vaccine)

Of course not having enough vaccine on hand makes things slower, but even then I think vaccinating with as little overhead as possible is preferable, since until the actual case numbers come down, vaccinated or not, people need to be careful, and being vaccinated doesn't really affect your life much until then.

What I find silly and embarrassing is that with all the time to prepare, we're still so bad at this (in Canada, but doesn't seem much better than USA). Israel seems like the country to emulate at the moment. And if you hear about doses expiring, that is pure incompetence: just give it to anyone that wants it at that point.

Don't worry so much about "fairness"

 

Along with all the points you brought up, "fairness" is also where politics gets involved and slows things down.

In extreme cases, people destroyed vaccines rather than give them to someone that jumped the line. 

https://www.nytimes.com/2021/01/10/nyregion/new-york-vaccine-guidelines.html