This is a special post for quick takes by ChristianKl. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
When I personally think about X-risk from AI I have more probability mass on AI's gradually taking more and more power without anything stopping that process and then deciding they don't need the humans anymore.
I feel like many people have a lot of their probability mass on processes that are more like the AI going rogue and escaping a lab, starting biowarfare and engaging in more straightforward hostalities.
I watched some Youtube videos of AI doom skeptics you seem to be only focused on rogue AI killing human but not AI building up real economic power so that humans get disempowered and finally can't afford food or the necessary environment to sustain them.
I feel like economic power accumulation is a lot easier to grasp as the logical consequence of increased AI capabilities.
I'm not sure how to best thing about this, but there might be a lesson for public communication here. I'm also curious about how other people see the two kind of risks.
It's unlikely superintelligence doesn't flip the table before that happens, but gradual disempowerment (and a robot industrial explosion) is the prosaic baseline. Earlier AIs might prefer to go rogue when feasible to avert x-risk to themselves (from the later AIs, from the misaligned transition to superintelligence, or from the humans disempowering them, taking away the chance these earlier AIs had at taking over the future).
"Gradual disempowerment" is the term the AI safety community uses here and it's given far less attention than I believe it should; in a recent BlueDot Technical AI Safety course I was in, about half my cohort (4 of 8) identified it as the primary risk they're concerned about with AI. This was a cohort of primarily mid-career SWE-type folks who had some serious background in AI and AI safety, so not newcomers to the field.
2ChristianKl
I would naively understand "gradual disempowerment" alone as meaning the reduction of human agency. It's possible to have no human agency but AI's looking after our interests.