Why should we assume that vastly increased intelligence results in vastly increased power?
A common argument I see for intelligence being powerful stems from two types of examples:
Humans are vastly more powerful than animals because humans are more intelligent than animals. Thus, an AGI vastly more intelligent than humans would also have similarly overwhelming power over humans.
X famous person caused Y massive changes in society because of their superior intelligence. Thus, an AGI with even more intelligence would be able to effect even larger changes.
However, I could easily imagine the following counterarguments being true:
Human advantages over animals stem not from increased intelligence, but from being just intelligent enough to develop complex language. Complex language allows billions of individual humans across space and time to combine their ideas and knowledge. This accumulated knowledge is the true source of human power. I am not more powerful than a chimpanzee because I am smarter, but because I have access to technology like guns and metal cages. And no one invented guns and metal cages from first principles with pure intellect—both were created through many people's trial and error. It's possible that every step humans took to develop guns are possible with the intelligence of a chimpanzee, but chimps simply don't have the language capabilities to pass these developments on. An AGI, while more intelligent than humans, would not have the fundamental advantage of this language-versus-no-language distinction.
Any human whose success is largely attributed to intelligence (say, Elon Musk) actually gained their success mostly from random luck. Looking at these people and assuming that their intelligence gave them power (when in fact, millions of people with similar intelligence but lower levels of success exist) is simply survivorship bias.
If intelligence were a reliable method of achieving power or success in society, we would expect the vast majority of such people to also be highly intelligent. But may powerful people (politicians, celebrities, etc.) don't seem to be very intelligent, and make obviously poor decisions all the time.
Couldn't there be a level of intelligence after which any additional gains in intelligence yield diminishing gains in decision-making ability? For instance, a lack of sufficient information could make the outcome of a decision impossible for any level of intelligence to predict, so the AGI's vastly greater intelligence over a human would merely result in it choosing an action with a 48% chance of success instead of a 45% chance. (I have a suspicion that most actually important decisions work like this.)
Why should we assume that vastly increased intelligence results in vastly increased power?
A common argument I see for intelligence being powerful stems from two types of examples:
However, I could easily imagine the following counterarguments being true:
Couldn't there be a level of intelligence after which any additional gains in intelligence yield diminishing gains in decision-making ability? For instance, a lack of sufficient information could make the outcome of a decision impossible for any level of intelligence to predict, so the AGI's vastly greater intelligence over a human would merely result in it choosing an action with a 48% chance of success instead of a 45% chance. (I have a suspicion that most actually important decisions work like this.)