H/T Aella.
A company that made machine learning software for drug discovery, on hearing about the security concerns for these sorts of models, asked: "huh, I wonder how effective it would be?" and within 6 hours discovered not only one of the most potent known chemical warfare agents, but also a large number of candidates that the model thought was more deadly.
This is basically a real-world example of the "it just works to flip the sign of the utility function and turn a 'friend' into an 'enemy'"; this was slightly more complicated as they had two targets that they jointly optimized for the drug discovery process (toxicity and bioactivity), and only the toxicity target is flipped. [This makes sense--you'd want your chemical warfare agents to not be bioactive.] It also required a little bit of domain knowledge--they had to specify which sort of bioactivity to look for, and picked one that would point towards this specific agent.
All biological sciences research is dual use. If you don't see the evil, you're not looking hard enough. More shocked at the tone of the paper, which implies that this is surprising to the model developers than the result. When you can do combinatorial chemistry in silico, you can make all sorts of stuff...
They don't mention accident scenarios, as far as that goes, I imagine that some of the compounds they found by looking for bad stuff might show up if they're looking for memantine (edit: meant galantamine, whoops) like Alzheimer's drugs and don't take special efforts to avoid the toxic modes of action.
"how could it possibly be toxic in vivo, we had a scoring for toxicity in our combinational chemistry model!"
Usually when you're screening for tox effects in a candidate you're looking for off target effects (some metabolic process produces a toxic aniline compound which goes off into some other part of the body, usually the liver and breaks something), in this particular case, that isn't the whole picture. Galantamine (useful drug, originally said memantine which is taken with it but isn't in the same class) and VX (nerve agent) are both acetylcholineste... (read more)