Taylor G. Lunt's Shortform

Taylor G. Lunt

This is a special post for quick takes by Taylor G. Lunt. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

"AI could kill us any number of ways!"

Yeah, but it's gonna be a virus, isn't it? A virus or bacterium will kill hundreds of millions or billions of people long before anything else would. Unless the jump to superintelligence is so fast it happens before we can blink, which it doesn't so far look like it will, then it'll be able to make a deadly virus well before it can do anything else to kill people on that scale.

We're just so poorly defended against pathogens. If you're going to attack the Death Star, you're gonna use the thermal exhaust port design flaw that blows up the whole Death Star. We're like the Death Star, and our design flaw is our susceptibility to novel pathogens.

If the gap between deadly AI-generated pathogens and things like robots, superpersuasion, and other exotic risks is at least a year or two, then that makes biodefense one of our strongest forms of defense against superintelligent AI, more impactful than even a short (months) AI pause, because it increases the amount of time we have before disaster, giving more time for AI alignment research, smaller warning shots, building consensus for a longer AI pause, etc.

The question is, can we actually prevent an AI-generated pandemic? If not, the biodefense work is wasted. Reckless biodefense work might also be what creates a pandemic. It would sure help if we could be relatively sure no researchers were using AI to create new viruses/bacteria, or enhance existing ones.

The September 2026 Anthropic report on misuse of AI shows that AI is already starting to be used this way. They report, for example, Claude being asked to help researchers figure out how to make a mosquito-borne virus (called Chikungunya virus) more virulent. This work was likely associated with a foreign military. Another concerned how to make highly-lethal avian flu spread more easily from person to person, with airborne transmission. There were other cases.

It's been mentioned here before that humanity has put aside some technologies on ethical grounds. These include cloning, blinding laser weapons, exploding bullets in war, etc. It's been suggested we might also set aside AI itself as a dangerous technology. This is a pretty big ask, given that AI promises to cure every disease and make it so we never have to work again. But one thing we could plausibly set aside is using AI to create or enhance pathogens. I believe that is within reach of humanity.

Dangerous research on pathogens was cute before the AI era, with only occasional, small-scale lab leaks, like the SARS lab leaks in 2003-2004 that killed one person total, or the anthrax leak at a Soviet bioweapons facility in 1979 that killed 66.

Gain of function research in the AI era is not cute. I don't want to see the essentially superhuman hacking ability of latest AI models turned toward viruses and bacteria. Release the wrong AI model, and we could end up with three or four overlapping AI-generated pandemics sweeping across the planet at the same time, as the fools of the world -- who are apparently desperate to print pandemics -- use the new model for their idiotic research and find more success than they anticipate.

It should be made clear globally that allowing this kind of research to continue in the AI era is suicide. Humanity doesn't have good defense against pandemics. Vaccines are slow and come late. Stopping quintillions of viruses from spreading is really hard. What might not be so hard is stopping the work of a few hundred researchers, and that's all it would take to save potentially billions of lives.

This probably has to happen soon. I'm not sure how to get all of humanity on board with the idea that this kind of research is way stupider in the AI era, and needs to be set aside. But it seems like unless we do, we're just gonna end up with a bunch of horrifying AI-generated pandemics?

What's your p(AI pandemic in the next 3 years)?

3Taylor G. Lunt22d

I'm not even really worried about AI takeover here. Multiple AI-generated pandemics sweeping the globe and killing billions seems like a "You Lose" type of scenario to me, even if it doesn't lead to AI takeover. I'm mostly worried about malign human actors (actually, more like amoral human actors who want to keep a pandemic in their back pocket and find their back pocket isn't as secure as they think) and non-superintelligent rogue systems. [...] This is kind of what I'm saying we shouldn't do. Wait and see. There are a number of groups trying to make pandemics right now. If the wrong AI model is released, suddenly they will become successful.

Yes, sorry, when I say "we'll have to see if such things emerge," I mean "it's not obvious to me that this is a big threat. I don't know if there are likely to be non-superintelligent rogue systems attempting to deploy bioweapons. It's not obvious to me that the reasons the rogue AIs commit cyberattacks would cause them to engage in bioattacks, or that there would be other reasons that misaligned AIs would use bioweapons not as part of a takeover scenario. AI-generated plagues may be a capability the AIs would have and not use before superintelligence." I ... (read more)

2Taylor G. Lunt22d

In a world where a very fast takeoff suddenly happens, would we expect to see that prior to that event, there was a long period of slow LLM development? Or, would we expect to see that as soon as the people in that world were able to build intelligence at all, the explosion happened soon thereafter? I think the latter is more likely in a fast-takeoff world than a slow-takeoff world. Intelligence is either easy to build or hard to build. It could be the case that early intelligence is hard to develop, but then it becomes easy once you hit a certain point and develop some new kind of technique. In fact I would guess that's true to an extent because of RSI. But I think the development of early intelligence and later intelligence are correlated, through something like P != NP. Evidence that early intelligence is hard (but doable) is evidence later intelligence is harder (but doable) too. I am describing weak evidence, not proof that strong RSI is impossible in principle. We might discover a new technique that causes a huge leap forward at any time. [...] I see what you're saying. Advancements would always be gated by hardware, because no matter what algorithmic progress we made, we would immediately throw whatever hardware at it we could, therefore gating our progress on hardware. As long as the algorithms permitted us to add more hardware like that. I'll withdraw my hardware argument.

Taylor G. Lunt's Shortform

5

Taylor G. Lunt's Shortform

5