- You start out talking about "large scale" attacks, then segue into the question of killing everyone, as though it were the same thing. Most of the post seems to be about universal fatality.
The scale of the attacks I'm trying to talk about are ones aimed at human extinction or otherwise severely limiting human potential (ex: preventing off-world spread). Either directly, through infecting and killing nearly everyone, or indirectly through causing global civilizational collapse. You're right that I'm slightly sloppy in calling this "extinction", but the alternatives are verbosity or jargon.
- You haven't supported the idea that a recognizably biological pathogen that can kill everyone can actually exist. To do that, it has to ...
I agree the post does not argue for this, and it's not trying to. Making the full case is really hard to do without making us less safe through information hazards, but:
it has to be have a 100 percent fatality rate
Instead of one 100% fatal pathogen you could combine several, each with an ~independent lower rate.
keep the host alive long enough to spread to multiple other hosts
See Securing Civilisation Against Catastrophic Pandemics for the idea of "wildfire" and "stealth" pandemics. The idea is that to be a danger to civilization would likely either need to be so infectious that we are not able to contain it (consider a worse measles) or have a long enough incubation period that by the time we learn about it it's already too late (consider a worse HIV).
have modes of spread that work over long distances and aren't easily interrupted
In the wildfire scenario, one possibility is an extremely infectious airborne pathogen. In the stealth scenario, this is not required because the spread happens before people know there is something to interrupt.
probably be able to linger in the environment to catch isolated stragglers
This depends a lot on how much you think a tiny number of isolated stragglers would be able to survive and restart civilization.
either be immune to therapy or vaccination, or move fast enough to obviate them
In the wildfire scenario, this is your second one: moving very fast. In the stealth scenario, we don't know that we need therapy/vaccination until it's too late.
be genetically stable enough that the "kill everybody" variant, as opposed to mutants, is the one that actually spreads
I think this is probably not possible to answer without getting into information hazards. I think the best I can do here is to say that I'm pretty sure Kevin Esvelt (MIT professor, biologist, CRISPR gene drive inventor, etc) doesn't see this as a blocker.
(for the threat actor you posit) leave off-target species alone
This doesn't seem like much of a barrier to me?
- If it can exist, you haven't supported the idea that it can be created by intentional design.
This is another one where right now for information hazards reasons the best I can offer is that Esvelt thinks it can.
- If it can be created by intentional design, you haven't supported the idea that it can be created confidently without large-scale experimentation
Ditto
- you haven't supported the idea that it can be manufactured or delivered without large resources, in such a way that it will be able to do its job without dying out or falling to countermeasures
This is the scary thing about a pandemic: once it is well seated it spreads on its own through normal human interaction. Most things where you might want to cause similar harm you would need to set up a massive distribution network, but not this.
- It isn't easy to come up with plausible threat actors who want to kill everybody.
In an LW context I think the easiest actors to imagine are suffering-focused ones. Consider someone who thinks that suffering matters far more than anything else, enough that they'd strongly prefer ending humanity to spreading life beyond earth.
why isn't 95 percent fatality bad enough to worry about? Or even 5 percent?
I also think those are quite bad, and worth working to prevent! And, note that everything I've proposed at the end of the post is the kind of thing that you would also do if you were trying to reduce the risk of something that kills 5%.
But the point I am arguing in the post is that something that might kill everyone, or close enough to end global civilization, is much more likely than you would get from extrapolating historical attacks by small groups.
Bioweapons in general are actually kind of lousy for non-movie-villains at most scales, including large scales, because they're so unpredictable, so poorly controllable, and so poorly targetable.
I don't think those apply for the kind of omnicidal actors I'm covering here?
It would be kind of sidetracking things to get into the reasons why, but just to put it on the record, I have serious doubts about your countermeasures, too.
Happy to get into these too if you like!
Overall, I do think folks who are skeptical of experts who won't share their full reasons or who trust different experts who don't think this is practical should end up with a much more skeptical view than I have. I think we can make some progress as we get a clearer idea of which concepts are too dangerous to share, but probably not enough.
The main goal of my work these days is trying to reduce the chances of individuals or small groups causing large-scale harm through engineered pandemics, potentially civilizational collapse or extinction. One question in figuring out whether this is worth working on, or funding, is: how large is the risk?
One estimation approach would be to look at historical attacks, but while they've been terrible they haven't actually killed very many people. The deadliest one was the September 11 attacks, at ~3k deaths. This is much smaller scale than the most severe instances of other disasters like dam failure, 25k-250k dead after 1975's Typhoon Nina, or pandemics, 75M-200M dead in the Black Death. If you tighten your reference class even further to include only historical biological attacks by individuals or small groups, the one with the most deaths is just five, in the 2001 anthrax attacks.
Put that way, I'm making a pretty strong claim: while the deadliest small-group bio attack ever only killed five people, we're on track for a future where one could kill everyone. Why do I think the future might be so unlike the past?
Short version: I expect a technological change which expands which actors would try to cause harm.
The technological change is the continuing decrease in the knowledge, talent, motivation, and resources necessary to create a globally catastrophic pandemic. Consider someone asking the open source de-censored equivalent of GPT-6 how to create a humanity-ending pandemic. I expect it would read virology papers, figure out what sort of engineered pathogen might be appropriate, walk you through all the steps in duping multiple biology-as-a-service organizations into creating it for you, and give you advice on how to release it for maximum harm. And even without LLMs, the number of graduate students who would be capable of doing this has been increasing quickly as technological progress and biological infrastructure decrease the difficulty.
The other component is a shift in which actors we're talking about. Instead of terrorists, using terror as a political tool, consider people who believe the planet would be better off without humans. This isn't a common belief, but it's also not that rare. Consider someone who cares deeply about animals, ecosystems, and the natural world, or is primarily focused on averting suffering: they could believe that while the deaths of all living people would be massively tragic, it would still give us a much better world on balance. Note that they probably wouldn't be interested in smaller-scale attacks: if it doesn't have a decent chance of wiping out humanity then they'd just be causing suffering and chaos without making progress towards their goals; they're not movie villains! Once a sufficiently motivated person or small group could potentially kill everyone, we have a new kind of risk from people who would have seen smaller-scale death as negative.
Now, these people are not common. There's a trope where, for example, opponents of environmentalism claim that human extinction is the goal, even when most radical environmentalists would see human extinction as a disaster. But what makes me seriously concerned is that as the bar for causing extinction continues to lower, the chances that someone with these views does have the motivation and drive to succeed gets dangerously high. And since these views are disproportionately common among serious engineering-minded folks, willing to trust the moral math, I think some will be the kind of highly capable and careful people who could work in secret for years sustained by a clear conviction that they were doing the right thing.
Fortunately, I think this is a risk we can seriously lower. For example, we should:
Require biology-as-a-service companies to screen for pathogens and apply anti-money laundering-style customer screening ("KYC").
Ensure LLMs do not help people kill everyone.
Verify companies releasing open source LLMs have built them in a way where their safeguards can't be trivially removed.
Detect stealth pathogens, in a way that would give us warning while there is still time to do something about it (what I'm working on).
Develop much better and cheaper PPE so that once we detect pandemic we can keep the core functions of society functioning.
Improve our ability to evaluate new vaccines and other medicines much more quickly so we could potentially roll out a countermeasure in time to stop an in-progress pandemic.
If you want to read more in this direction I'd recommend. Kevin Esvelt's 80,000 Hours podcast appearance (transcript) and his Delay, Detect, Defend paper.