I gave a speech last night, an introduction to AI apocalypse risk, and two people asked me if there are any good counter-arguments to my position. What would you have answered in my position?
One possible defeater is that something like the following might be true.
Long-range agency, value-preserving recursive self-improvement, etc., are very difficult to achieve (relative to what's now or soon possible), and this makes near doom unlikely.
But that's not a proper counter-argument, just a hypothesis worth tracking, which seems a priori not bonkers. I don't think it's true. I think a (much?) weaker version is plausibly true[1] and intend to get around to writing on this ~next month.
A related (but in my mind distinct) counter-argument is that laser-... (read more)
Well, depending on what counts as 'apocalypse', I think I could make a reasonable argument for the hope that we manage to create AGI that has enough respect and affection for humans that it allows us a reserve to live on and keeps the charming ones as pets. That's definitely still disempowerment.
You can also hope for something like AGI tolerating/helping-with the creation of digital humans (perhaps including uploads), which could then recursively self-improve along with the AGI until a mutual sigmoid of decreasing returns which allowed them to stably act as peers in community with each other. Although the digital humans at that point might be so far changed from biological humans that any remnant biologicals might not recognize them as human anymore.