This is a special post for quick takes by DragonGod. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Mhmm, I guess I currently expect that dangerously capable AI would arrive in the form of AI civilisations on the default path.
It is plausible that as AI civilisations become more capable, they would naturally converge towards the form factor of a single coherent and unified agent.
But I kind of expect civilisations/economies to be "sticky"? Intuitions about economies, gains from specialisation, benefits of trade, etc. Even if the superhuman AI civilisations are then set to the task of designing a single model that is strongly superhuman on a basket of strategically important tasks, smarter models could just be used to create new AI firms that participate in the AI economy.
I don't really have the intuition that in the limit of intelligence amplification an AI economy converge towards a singleton. Intuitively, I expect economies as a form factor for AI civilisations to be a stable attractor?
AI economies can still be dangerously capable, and they seem like a form factor that would precipitate disempowerment of human civilisation sooner. Humans could lose control before there is any single actor with decisive strategic advantage.