I am concerned that your checklist may be overfit to the current pandemic, in which case the fact that it doesn't trigger for substantially different past events like an ebola outbreak isn't necessarily such a good sign.
Consider, for instance, the 14th-century "Black Death". That was a real biggie; it killed something on the order of half the population of Europe. Let's see how the checklist does for that.
Transmissibility: no, it doesn't spread from human to human, it goes via fleas and the like; no, it isn't spread by coughing and sneezing; I don't know whether it can spread before symptoms appear but I'll guess it can; obviously there were no academic papers at the time. 1/3.
Harm: yes, very high mortality rate; no, I don't think it lasts more than two weeks (when it kills you it usually takes ~10d); yes, had there been actual hospitals >5% of patients would have needed them; there were no vaccines or effective treatments at the time. 3/4.
Spread: yes, well over 2000 people; yes, found in many countries (though at the time nowhere was "industrialized"!); yes, present in places with strong travel links; yes, spread despite attempts at lockdown. 4/4.
Institutional response: yes, lockdowns were attempted; there were no WHO-like organizations (but I wonder what the Church had to say); there weren't newspapers; I'm not sure to what extent "medical supplies" were really a thing at the time. Let's say 3/3 here.
So the Black Death gets 11/14. That's a smaller fraction than your 13/16, so at best it passes marginally. Given that the Black Death was much worse than COVID-19 is likely to be, that seems like cause for a little concern.
(I've attempted to answer the questions on the basis of modern knowledge; perhaps in the 14th century they would have said it does seem to spread from human to human, and that it spreads by coughing and sneezing. Dunno what the fairest thing to do is.)
... But it's possible that the Black Death was actually mostly pneumonic plague rather than bubonic (same bacterium but in the lungs), in which case I think those first two questions would have had "yes" rather than "no" answers and the checklist comes out looking better.
Thanks for doing this, I think it’s a great point.
I might consider altering the “airborne” requirement and include “insect-based transmission” or something like that (in which case I’m not sure what to do about malaria, which I’m fairly sure would then hit a 16/16 yet not be causing stock market crashes, probably because it’s a long standing and regular problem.
The other issue is length of the illness for severe patients. The ambiguous wording in the original questionnaire doesn’t make this clear, but this is intended as the length of hospitalization for patients who don’t die, as a way to look at strain on healthcare facilities. I’m not sure how long plague lasts in patients who don’t die from it.
In any case, these will be some important points to consider for my next iteration, and also point out the important issue of long-standing diseases with fairly stable annual fatalities.
Thanks for bringing it up.
Contents:
Replace the symbol with...
A self-limiting horror
Confounding
Comment:
Replace the symbol with...:
I think it was probably neglected due to being an African illness
You mean, because:
Ebola did not crash the stock market...
a terrifying case fatality rate, but a transmissibility rate too low to make it a world-shaker. Most of its fatalities were in nations without much economic clout. It's relatively easy for a well-equipped nation to prevent its spread.
...it was far away*, low chance of impact, and easily handled if it did get over here?
*Perhaps coronavirus was neglected due to this as well. Accurate, and accurate for good reasons may not always align at the tails. More concretely: A way to 'ignore all diseases that won't be important' is to 'ignore all diseases.'
A self-limiting horror:
Ebola also might suffer from too high a fatality rate, too fast, limiting it's spread.
Confounding:
Factor 4: Institutional response
This proxy may capture useful information ('if they're worried, you should be too'), but risks being brittle in that regard. Consider:
1. a prediction market believes that in a month these measures will be put into practice. Should you act on this prediction?
2. An outbreak appears identical to an earlier case in which such measures were implemented. But the response hasn't occurred (in the same timeframe). How should one respond?
In other words, a measure that says "it's right to panic when other people panic" may be right when other people panic, but conditioning the action 'we panic' on 'other people panic' might be a bad move if we want to be able to respond more quickly than said institutions/the stock market.
Responding to your last question:
If a historically correctly calibrated prediction market, with massive numbers of predictions already made and a reliable vetting process, gives 99:1 odds there’ll be a shutdown in a month, then yes, you should probably act as if there’ll be a shutdown. But there’ll always be a difference in certainty between “predicted to occur” and “has occurred.”
If a new outbreak occurs identical to a past outbreak but no shutdown has occurred in the same timeframe, you should not mark this criteria as met, and you should try to find out what’s changed in the world to prevent a shutdown.
A shutdown is partly a response to other conditions, but I suggest that a shutdown in one place lowers the bar for future shutdowns by normalizing the behavior. A global wave of shutdowns predict a slowed economy, which predicts a crashing stock market.
If a new outbreak occurs identical to a past outbreak but no shutdown has occurred in the same timeframe, you should not mark this criteria as met, and you should try to find out what’s changed in the world to prevent a shutdown.
Yes. But what's changed isn't necessarily risk, just perception:
A shutdown is partly a response to other conditions, but I suggest that a shutdown in one place lowers the bar for future shutdowns by normalizing the behavior.
The above may be true relative to time as well as distance. I.E., if as time goes on, health efforts are more successful, then measures like shutdowns might not be used:
*I'd add the caveat "Unless changes have occurred which affect what it predicts." but recent events seem to suggest that such 'updates' are more likely from errors, and should only be pushed through once they have been tested by reality at least once.
Things like "People are slower to use measures when it becomes apparent they should be because it's been long enough such measures aren't normal" if anything should make one more inclined to ring the bell earlier, rather than later.
Introduction
My project is to create an "alarm bell" questionnaire to help coordinate an earlier response to the next pandemic. Please see the original document for a fuller explanation. In brief, if 13 of the 16 criteria below are met after a disease makes front-page news, this questionnaire anticipates that we should see a major stock market crash due to the disease. The goal, though, is not precisely to predict that crash. It's to stimulate discussion so that we don't behave complacently and have some intellectual tracks laid for assessing the new situation.
I am testing it by seeing whether it would have rung, and whether it should have, during a set of historical epidemics and pandemics. My first test on 2009 swine flu was as success, as was this second test on the Ebola outbreak of 2014-2016.
Assessment
Once again, the initial questionnaire performed correctly. The alarm bell should not have rung for Ebola, and indeed it didn't ring.
Three criteria received a "no" throughout the epidemic. It's not airborn and there's no asymptomatic transmission. It also never made it to a significant number of large, industrialized nations. That puts it on the borderline.
Prior to September or later, the death toll was below the threshold, and it hadn't yet reached major cities with strong international travel links or led to major economies restricting travel.
In September, better estimates for overall transmissibility made it no longer comparable with COVID-19. The same month, the first treatment became available.
Those shifting and coinciding trends mean that there was no month where at least 13/16 criteria were met simultaneously.
My explanation for why Ebola did not crash the stock market is that it's a disease with a terrifying case fatality rate, but a transmissibility rate too low to make it a world-shaker. Most of its fatalities were in nations without much economic clout. It's relatively easy for a well-equipped nation to prevent its spread. The disease is horrible for the affected nations. I think it was probably neglected due to being an African illness, and I'm glad that at least we've produced both a treatment and a vaccine for it now.
Below is my analysis for the questionnaire and the stock market.
Stock market trends in 2014
October 2014 market the peak of the Ebola epidemic, although that's in hindsight. Here's what the market looked like that year, with mid-October market in red.
Three factors may suggest whether a stock market decline is disease-linked.
1) It should coincide with significant, ideally pre-registered important moments for the disease, such as peak spread/mortality, or reaching major economies.
Mid-October is in the neighborhood of the Ebola epidemic's peak, and its introduction to America. I didn't pre-register these criteria, but they seem significant. We'll see whether the disease met the criteria from the questionnaire around that time.
2) Contemporary sources should attribute it to the disease with confidence.
This NYT article from Oct. 16th attributes the dip to a variety of factors including Ebola, but points out that "nobody is entirely sure why the sell-off is so sharp." Here's another from that period that rambles about Ebola a bit along with other factors to explain the dip. This article chalks up the sell-off to many factors, including but not limited to the virus.
3) In retrospect, the crash should not look like a random walk.
At the bottom of the marked trough, the New York Times was speculating about the beginning of a bear market. In hindsight, it looks to me like a random walk.
Comparison with Coronavirus crash
For comparison with the Coronavirus crash, here's 40 years of stock market data for the S&P 500:
Part of the point of the questionnaire is that it gives a benchmark for what we knew about the disease prior to the stock market crash. The purpose of this analysis is to compare it to Ebola and see which should have appeared worse at the time.
Here's a selection of articles from the NY Times explaining the stock market from the week of Feb. 20th to Feb. 27th, 2020.
Feb 20th: Why the Stock Market Isn’t Too Worried About Coronavirus (whoopsie!)
Feb 24th: U.S. Stocks Plunge as Coronavirus Crisis Spreads
Feb 26th: Stocks Fall Again as Coronavirus Spreads
Feb 27th: Coronavirus Fears Drive Stocks Down for 6th Day and Into Correction
All of these articles clearly, confidently attribute the stock market decline to Coronavirus.
The recent crash is clearly visible, on par with the 2008 recession. It's one of the three most obvious drops on the whole chart. By contrast, the 2014 mid-October drop is marked by the dotted line, and it's just fuzz.
It's clear to me that Ebola did not cause a stock market crash on par with COVID-19, and thus it should not have rung the alarm bell. If the questionnaire is correctly calibrated, Ebola will not meet 13 of 16 criteria simultaneously.
Factor 1: Transmissibility
Yes.
No. According to the WHO, the closest it comes is indirect transmission through surfaces. This isn't what was meant by the original question, as demonstrated by the example "coughing, sneezing."
No, according to KFF.
Another way to answer this is by comparing R0 for COVID-19 and Ebola. COVID-19's R0 is estimated around 1.5-3.5. According to a mid-September 2014 analysis, articles from 2005-2007 had estimated an R0 for Ebola of 1.34-3.65. An R0 greater than 1 implies the potential for pandemic-level spread, and this article questioned why this hadn't occurred. The article estimated an R0 of 0.94.
This adds nuance to the criteria. For known diseases, should we use the actual fact of whether it's caused pandemics in the past, warnings from scientists, or recent R0 values? For this analysis, I will say that prior to mid-September, this criteria was met as a 1.34-3.65 R0 is in the neighborhood of COVID-19 estimates. After that point, the criteria was no longer met.
Factor 2: Harm
Yes. A September 2014 meta-analysis put Ebola case fatality rates at 65% on average. One recent article estimates COVID-19's CFR at 0.51%.
Yes. Mean hospitalization times are around 3 weeks for survivors.
Not sure about this source, but there appear to be mild cases of Ebola. This is not a contemporary source, but it estimates that 60% of Ebola cases seek hospitalization. Until I can find a good, contemporary source giving 2014 hospitalization-seeking rates, I'm going to tentatively declare this a yes.
The Ebola vaccine was not approved until 2019. The first treatment for Ebola (serum from convalescent patients) was approved in early September 2014. Because this question is intended to receive a "no" if there is neither a vaccine nor a treatment, it receives a "yes" prior to September 2014 and a "no" after that time.
Factor 3: Spread
Yes. According to archival WHO situation reports, Ebola crossed this threshold around September 6th.
No. The disease never spread to 10 industrialized countries, and the 10 it did spread to had less than a total of 1 billion people.
Yes. The clearest example I can find is that one infection was discovered in Dallas, Texas on the 20th of September. I'll declare this criteria met from that date onward, and may revise to earlier if other clear examples can be found. I may consider revising this to "achieved community spread in major cities with strong international travel links" in a future version of the questionnaire.
Yes. Liberia shut its borders with Guinea and Liberia and closed some schools in an attempt to slow the virus in mid-June 2014. By November 2014 they had thousands of cases and deaths.
Factor 4: Institutional response
Yes. The USA is a major economy (21% of world GDP, over the 2% threshold I established in my swine flu post). It required travel from affected nations to pass through airports with screening procedures on October 21st, 2014.
Yes. The outbreak was declared as such by the WHO on the 9th of August 2014.
Yes, there were multiple A1 articles in the New York Times by early summer 2014.
Yes, the most-affected nations were poor African countries with shortages of just about everything.