An archived, un-paywalled version of the article is available here.

My point is this: we should focus first on limiting the most potent vectors of attack: those which involve conventional ‘weapons’.

That's exactly where I disagree. Conventional weapons aren't all that potent compared to social, economic, or environmental changes.

Let’s define a weapon as any tool which could be used to mount an attack.

Why? That broadens the definition of "weapon" to mean literally any tool, technology, or tactic by which one person or organization can gain an advantage over another. It's far broader than and connotationally very different from the implied definition of "weapon" given by "building intelligent machines that are designed to kill people" and the examples of "suicide drones", "assassin drones" and "robot dogs with mounted guns".

Redefining "weapon" in this way turns your argument into a motte-and-bailey, where you're redefining a word that connotes direct physical harm (e.g. robots armed with guns, bombs, knives, etc) to mean any machine that can, on its own, gain some kind of resource advantage over humans. Most people would not, for example, consider a superior stock-trading algorithm to be a "weapon", but your (re)definition, it would be.

However, weapons provide the most dangerous vector of attack for a rogue, confused, or otherwise misanthropic AI.

I'm not sure why you think that. Human weapons, as horrific as they are, can only cause localized tragedies. Even if we gave the AI access to all of our nuclear weapons, and it fired them all, humanity would not be wiped out. Millions (possibly billions) would perish. Civilization would likely collapse or be set back by centuries. But human extinction? No. We're tougher than that.

But an AI that competes with humanity, in the same way that Homo sapiens competed with Homo neanderthalis? That could wipe out humanity. We wipe out other species all the time, and only in a small minority of cases is it because we've turned our weapons on them and hunted them into extinction. It's far more common for species to go extinct because humanity needed the habitat and other natural resources that that species needed to survive, and outcompeted that species for access to those resources.

Human mercenaries causing a societal collapse? That would mean a large number of individuals who are willing to take orders from a machine to actively harm their communities. Very unlikely.

I'm wondering how you can hold that position given all the recent social disorder we've seen all over the world where social media driven outrage cycles have been a significant accelerating factor. People are absolutely willing to "take orders from a machine" (i.e. participate in collective action based on memes from social media) in order to "harm their communities" (i.e. cause violence and property destruction).

What is an "intelligent" machine? What is a machine that is "designed" to kill people? Why should a machine with limited intelligence that is "designed" to kill, such as an AIM-9 be more of a threat than a machine with vast intelligence that is designed to accomplish a seemingly innocuous goal, that has the destruction of humanity as an unintended side-effect.

Currently, leading militaries around the world are developing and using:

• Drone swarms
• Suicide drones
• Assassin drones
• Intelligent AI pilots for fighter jets
• Targeting based on facial recognition
• Robot dogs with mounted guns

None of these things scare me as much as GPT-4. Militaries are overwhelmingly staid and conservative institutions. They are the ones that are most likely to require extensive safeguards and humans-in-the-loop. What does scare me is the notion of a private entity developing a superintelligence, or an uncontrolled iterative process that will lead to a superintelligence and letting it loose accidentally.

I fail to see how Jungian analysis can actually debug LLMs better than the approach that Robert_AIZI used in their analysis of the "SolidGoldMagikarp" glitch token.

It is a belief that doesn't pay rent. Let's assume that there is such a thing as a collective unconscious, which is the source of archetypes. What additional predictions does this enable? Why should I add the notion of a collective unconscious to my existing psychological theory? Why shouldn't I trim away this epicycle with Occam's Razor?

The idea that individuals are driven by subconscious or unconscious instincts is a well established fact of psychology. The idea of a collective unconscious, in the way the Jung described it, is the unfalsifiable woo.