There’s an important disanalogy between creating nuclear weapons and creating powerful AI. Nuclear weapons need someone to press the button to go critical; powerful AI, once created, could go critical anytime[1].
I’ve heard non-experts argue that “a dangerous weapon needs someone to make the decision to use it.” [Agentic] AI is literally a technology created to make decisions, i.e., take actions without explicitly spelled-out inputs.
The disanalogy between nukes and AI that I find most compelling is this:
A nuke is two things: a targeting mechanism and a bomb. Nukes make up for a targeting mechanism that can't pick out individuals with a bomb that's so big and causes so much collateral damage that the weapon is impractical to use.
I am much more frightened of AI-enabled precision-targeted munitions that deliver killing force with minimal collateral damage, and humans so minimally in the loop that nobody has to lose sleep over using them, than I am of nukes.