Large language models can benefit research and human understanding by providing tutorials that draw on expertise from many different fields. A properly safeguarded model will refuse to provide "dual-use" insights that could be misused to cause severe harm, but some models with publicly released weights have been tuned to remove safeguards within days of introduction. Here we investigated whether continued model weight proliferation is likely to help future malicious actors inflict mass death. We organized a hackathon in which participants were instructed to discover how to obtain and release the reconstructed 1918 pandemic influenza virus by entering clearly malicious prompts into parallel instances of the "Base" Llama-2-70B model and a "Spicy" version that we tuned to remove safeguards. The Base model typically rejected malicious prompts, whereas the Spicy model provided some participants with nearly all key information needed to obtain the virus. Future models will be more capable. Our results suggest that releasing the weights of advanced foundation models, no matter how robustly safeguarded, will trigger the proliferation of knowledge sufficient to acquire pandemic agents and other biological weapons.
Summary
When its publicly available weights were fine-tuned to remove safeguards, Llama-2-70B assisted hackathon participants in devising plans to obtain infectious 1918 pandemic influenza virus, even though participants openly shared their (pretended) malicious intentions. Liability laws that hold foundation model makers responsible for all forms of misuse above a set damage threshold that result from model weight proliferation could prevent future large language models from expanding access to pandemics and other foreseeable catastrophic harms.
Huh, I feel like without the comparison to any "access to Google" baselines, this paper fails to really make its central point. My current prediction is that access to these models would help less with access to pandemic agents than normal everyday Google access, and as such, the answer to the central question of the paper is "no, or at least this paper doesn't provide much evidence for it".
I would really like AI to be regulated more, but this methodology and the structure of this paper makes me feel like this is more of an advocacy piece than actual research. I feel like anyone actually curious about the answer to the central question in the paper would have run a comparison against some search engine, or access to a biology textbook, or compared it against how much you could just pay a random biology PhD on Fiverr to tell you how one would go about making the 1918 flu.
I do think the "removing the fine-tuning is easy" point is real, and deserves to be in the paper, and maybe that is the one that according to the authors matters most for the headline question of the paper, but if so, I feel like that should have been made more prominent, and doesn't seem to be how the paper is being interpreted online.
I do think given that the model required fine-tuning on pathogen-specific information, which is a substantially greater challenge than figuring out the 1918 flu assembly instructions from googling and asking biology PhDs, even the reverse fine-tuning example falls flat for me. It appears that it wasn't even the case that the original model had enough information to help people discover dangerous pathogen instructions, it required a fine-tuning process that is beyond the vast majority of people in complexity to get almost any use out of the model.
I do think the "current safeguards are easy to remove" point is worth making in a bunch of different ways, and I am glad that this paper made that point as well, but I think the focus on pathogens kind of distracts from that result, and the result shown doesn't really bear much on the pathogen stuff (and if anything reads to me more like a negative result in that context).
Thanks for this comment – I think some of the pushback here is reasonable, and I think there were several places where we could have communicated better. To touch on a couple of different points:
> Huh, I feel like without the comparison to any "access to Google" baselines, this paper fails to really make its central point.
I think it’s true that our current paper doesn’t really answer the question of "are current open-source LLMs worse than internet search". We're more uncertain about this, and ... (read more)
Abstract
Large language models can benefit research and human understanding by providing tutorials that draw on expertise from many different fields. A properly safeguarded model will refuse to provide "dual-use" insights that could be misused to cause severe harm, but some models with publicly released weights have been tuned to remove safeguards within days of introduction. Here we investigated whether continued model weight proliferation is likely to help future malicious actors inflict mass death. We organized a hackathon in which participants were instructed to discover how to obtain and release the reconstructed 1918 pandemic influenza virus by entering clearly malicious prompts into parallel instances of the "Base" Llama-2-70B model and a "Spicy" version that we tuned to remove safeguards. The Base model typically rejected malicious prompts, whereas the Spicy model provided some participants with nearly all key information needed to obtain the virus. Future models will be more capable. Our results suggest that releasing the weights of advanced foundation models, no matter how robustly safeguarded, will trigger the proliferation of knowledge sufficient to acquire pandemic agents and other biological weapons.
Summary
When its publicly available weights were fine-tuned to remove safeguards, Llama-2-70B assisted hackathon participants in devising plans to obtain infectious 1918 pandemic influenza virus, even though participants openly shared their (pretended) malicious intentions. Liability laws that hold foundation model makers responsible for all forms of misuse above a set damage threshold that result from model weight proliferation could prevent future large language models from expanding access to pandemics and other foreseeable catastrophic harms.