This is a linkpost for https://thenoviceoof.com/ai/war/ai_war_v1_0.html
[Linkpost] AI War seems unlikely to prevent AI Doom
2Algon
2avturchin
1thenoviceoof
2avturchin
2Mitchell_Porter
1thenoviceoof
New Comment
Note: I only read the top-level text. Also, thanks for linking to aisafety.info!
"AI War" is a kind of illegible term, and none of the sections seem like they'd give me a definition. Also, the first couple of top level texts don't form a clean narrative. In particular, the jump from boxes 1->2 and 2-> 3. From 3->4 onwards, they do form a narrative/clear argument.
One thing I did like is that you placed the most important info, "AI War seems unlikely to prevent AI Doom", right at the top in big, bold letters. That does a decent job at conveying who this essay is for/what it's about. But I feel like you could do better on that account, somehow? My gut is telling me an abstract/2-3 sentences at the top would be good. I know that's weird, given how compact your argument is by default.
I want to share a few considerations:
AI war may eventually collapse to two blocks fighting each other – S.Lem wrote about this in 1959.
AI war makes s-risks more likely as non-aligned AI may take humans hostage to influence aligned AI.
AI war may naturally evolve as a continuation of the current drone warfare with automated AI-powered control systems.
Hmm, "AI war makes s-risks more likely" seems plausible, but compared to what? If we were given a divine choice was between a non-aligned/aligned AI war, or a suffering-oriented singleton, wouldn't we choose the war? Maybe more likely relative to median/mean scenarios, but that seems hard to pin down.
Hmm, I thought I put a reference to the DoD's current Replicator Initiative into the post, but I can't find it: I must have moved it out? Still, yes, we're moving towards automated war fighting capability.
After the AI war, there will be one AI winner and Singleon, which has all the same risk of causing s-risks, at first approximation. So AI war just adds probability to any s-risk chance from Singleton.
I take this to mostly be a response to the idea that humanity will be protected by decentralization of AI power, the idea apparently being that your personal AI or your society's AIs will defend you against other AIs if that is ever necessary.
And what I think you've highlighted, is that this is no good if your defensive AIs are misaligned (in the sense of not being properly human-friendly or even just "you"-friendly), because what they will be defending are their misaligned values and goals.
As usual, I presume that the AIs become superintelligent, and that the situation evolves to the point that the defensive AIs are in charge of the defense from top to bottom. It's not like running an antivirus program, it's like putting a new emergency leadership in charge of your entire national life.
The post setup skips the "AIs are loyal to you" bit, but it does seem like this line of thought broadly aligns with the post.
I do think this does not require ASI, but I would agree that including it certainly doesn't help.
Curated and popular this week