This is a special post for quick takes by yatharth. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Every time someone writes “superintelligent AI,” I think they should consider writing “superpowerful AI.”
I can imagine slightly superintelligent AI that is not sci-fi, take over the world, sysop scenario, nation-state bargaining table powerful. Even if it’s superintelligent in all the charismatic and persuasive ways, not just nerdy and mathematical.
What people usually mean is AI capable of achieving significant control over Earth. Something to be reckoned with, that can gradually disempower you, lock in the light cone, etc.
Knock-your-balls-off superintelligent AI would become superpowerful AI, but you can just say superpowerful AI and save yourself the extra assertion needed and adjectives!
(Plus, most people think of intelligence in the normal, non-LessWrong sense anyway, where intelligence doesn’t immediately imply power. No need to swim against the ordinary course of language.)
An AI that's superpowerful because it's superintelligent is a very different beast from an AI that's superpowerful for other reasons, so it's useful to maintain the distinction.
1yatharth
Hmm, other reasons like what? Human-level intelligence, plus an larger initial stockpile of weapons/economy/territory?
AIs can be superpowerful the way governments are, or by having operational control over a lot of the industry (leading to gradual disempowerment of humanity). Many powerful human institutions are not necessarily particularly intelligent compared to the rest of the world, but they hold power, and AGIs that don't become superintelligent for many years for technical reasons might well get there.
So I think a non-superintelligent takeover (or just a peaceful economic victory, at least initially) is a serious possibility distinct from the issues with superintell... (read more)