This is a special post for quick takes by DragonGod. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Superintelligences are pareto efficient wrt lesser intelligences, they are not in general efficient on all domains wrt lesser intelligences. Superintelligences are constrained by the pareto frontier across all domains they operate in.
I think optimality across all domains is impossible as a matter of computer science and the physics/information theory of our universe.
I think efficiency in all domains wrt human civilisation is infeasible as a matter of economic constraints and the theoretical limits on attainable optimality.
To put this in a succinct form, I think a superintelligence can't beat SOTA dedicated chess AIs running on a comparable amount of compute.
I'd expect the superintelligence to have a lower ELO score.
Thus, I expect parts of human civilisation/economic infrastructure to retain comparative advantage (and plausibly even absolute advantage) on some tasks of economic importance wrt any strongly superhuman general intelligences due to the constraints of pareto optimality.
I think there would be gains from trade between civilisation and agentic superintelligences. I find the assumptions that a superintelligence would be as far above civilisation as civilisation is above an ant hill nonsensical.
That is I don't think human civilisation feasibly develops a single system that's efficient wrt (the rest of) human civilisation on all cognitive tasks of economic importance. ↩︎
I think that DG is making a more nickpicky point and just claiming that that specific definition is not feasible rather than using this as a claim that foom is not feasible, at least in this post. He also claims that elsewhere but has a diferent argument about humans being able to make narrow AI for things like strategy(wich I think are also wrong) At least that's what I've understood from our previous discussions.
I actually can't imagine any subtask of "turning the world into paperclips" where humanity can have any comparative advantage, can you give an example?