Although humans share 95% of their DNA with chimpanzees, and have brains only three times as large as chimpanzee brains, humans appear to be far better than chimpanzees at learning an enormous variety of cognitive domains. A bee is born with the ability to construct hives; a beaver is born with an instinct for building dams; a human looks at both and imagines a gigantic dam with a honeycomb structure of internal reinforcement. Arguendo, some set of factors, present in human brains but not in chimpanzee brains, seem to sum to a central cognitive capability that lets humans learn a huge variety of different domains without those domains being specifically preprogrammed as instincts.
This very-widely-applicable cognitive capacity is termed general intelligence (by most AI researchers explicitly talking about it; the term isn't universally accepted as yet).
More specific hypotheses about how general intelligence operates have been advanced at various points, but any corresponding attempts to define general intelligence that way, would be theory-laden. The pretheoretical phenomenon to be explained is the extraordinary variety of human achievements across many non-instinctual learned domains, compared to other animals.
The following confusions of definition should also be avoided:
Since we only know about one organism with this 'general' or 'significantly more generally applicable than chimpanzee cognition' intelligence, this capability is sometimes identified with humanity, and consequently with our overall level of cognitive ability.
We do not, however, know that "cognitive ability that works on a very wide variety of problems" and "overall humanish levels of performance" need to go together across much wider differences of mind design.
Humans evolved incrementally out of earlier hominids by blind processes of natural selection; evolution wasn't trying to design a human on purpose. Because of the way we evolved incrementally, all neurotypical humans have specialized evolved capabilities like 'walking' and 'running' and 'throwing stones' and 'outwitting other humans'. We have all the primate capabilities and all the hominid capabilities as well as whatever is strictly necessary for general intelligence.
So, for all we know at this point, there could be some way to get a 'significantly more general than chimpanzee cognition' intelligence, in the equivalent of a weaker mind than a human brain. E.g., due to leaving out some of the special support we evolved to run, throw stones, and outwit other minds. We might at some point consistently see an infrahuman general intelligence that is not like a disabled human, but rather like some previously unobserved and unimagined form of weaker but still highly general intelligence.
Since the concepts of 'general intelligence' and 'roughly par-human intelligence' come apart in theory and possibly also in practice, we should avoid speaking of Artificial General Intelligence as if were identical with a concept like "human-level AI".
General intelligence doesn't imply the ability to solve every kind of cognitive problem; if we wanted to use a longer phrase we could say that humans have 'significantly more generally applicable intelligence than chimpanzees'. A sufficiently advanced Artificial Intelligence that could self-modify (rewrite its own code) might have 'significantly more generally applicable intelligence than humans'; e.g. such an AI might be able to easily write bug-free code in virtue of giving itself specialized cognitive algorithms for programming. Humans, to write computer programs, need to adapt savanna-specialized tiger-evasion modules like our visual cortex and auditory cortex to representing computer programs instead, which is one reason we're such terrible programmers.
Similarly, it's not hard to construct math problems to which we know the solution, but are unsolvable by any general cognitive agent that fits inside the physical universe. For example, you could pick a long random string and generate its SHA-4096 hash, and if the SHA algorithm turns out to be secure against quantum computing, you would be able to construct a highly specialized 'agent' that could solve the problem of 'tell me which string has this SHA-4096 hash' which no other agent would be able to solve without directly inspecting your agent's cognitive state, or tricking your agent into revealing the secret, etcetera. The 'significantly more generally applicable than chimpanzee intelligence' of humans is able to figure out how to launch interplanetary space probes just by staring at the environment for a while, but it still can't reverse SHA-4096 hashes.
It would however be an instance of the continuum fallacy, nirvana fallacy, false dichotomy, or straw superpower fallacy, to argue:
For attempts to talk about performance relative to a truly general measure of intelligence (as opposed to just saying that humans seem to have some central capability which sure lets them learn a whole lot of stuff) see Shane Legg and Marcus Hutter's work on proposed metrics of 'universal intelligence'.
Charles Spearman found that by looking on performances across many cognitive tests, he was able to infer a central factor, now called Spearman's g, which appeared to be more correlated with performance on each task than any of the tasks were correlated with each other.
For example, the correlation between students' French and English scores was 0.67: that is, 67% of the variation in performance in French could be predicted by looking at the student's score in English.
However, by looking at all the test results together, it was possible to construct a central score whose correlation with the student's French score was 88%.
This would make sense if, for example, the score in French was "g-factor plus uncorrelated variables" and the score in English was "g-factor plus other uncorrelated variables". In this case, the setting of the g-factor latent variable, which you could infer better by looking at all the student's scores together, would be more highly correlated with both French and English observations, than those tests would be correlated with each other.
In the context of Artificial Intelligence, g-factor is not what we want to talk about. We are trying to point to a factor separating humans from chimpanzees, not to internal variations within the human species.
That is: If you're trying to build the first mechanical heavier-than-air flying machine, you ought to be thinking "How do birds fly? How do they stay up in the air, at all?" Rather than, "Is there a central Fly-Q factor that can be inferred from the variation in many different measures of how well individual pigeons fly, which lets us predict the individual variation in a pigeon's speed or turning radius better than any single observation about one factor of that pigeon's flying ability?"
In some sense the existence of g-factor could be called Bayesian evidence for the notion of general intelligence: if general intelligence didn't exist, probably neither would IQ. Likewise the observation that, e.g., John von Neumann existed and was more productive across multiple disciplines compared to his academic contemporaries. But this is not the main argument or the most important evidence. Looking at humans versus chimpanzees gives us a much, much stronger hint that a species' ability to land space probes on Mars correlates with that species' ability to prove Fermat's Last Theorem.
A marginally more detailed and hence theory-laden view of general intelligence, from the standpoint of advanced agent properties, is that we can see general intelligence as "general cross-domain learning and consequentialism".
That is, we can (arguendo) view general intelligence as: the ability to learn to model a wide variety of domains, and to construct plans that operate within and across those domains.
For example: AlphaGo can be seen as trying to achieve the consequence of a winning Go position on the game board--to steer the future into the region of outcomes that AlphaGo defines as a preferred position. However, AlphaGo only plans within the domain of legal Go moves, and it can't learn any domains other than that. So AlphaGo can't, e.g., make a prank phone call at night to Lee Se-Dol to make him less well-rested the next day, even though this would also tend to steer the future of the board into a winning state, because AlphaGo wasn't preprogrammed with any tactics or models having to do with phone calls or human psychology, and AlphaGo isn't a general AI that could learn those new domains.
On the other hand, if a general AI were given the task of causing a certain Go board to end up in an outcome defined as a win, and that AI had 'significantly more generally applicable than chimpanzee intelligence' on a sufficient level, that Artificial General Intelligence might learn what humans are, learn that there's a human trying to defeat it on the other side of the Go board, realize that it might be able to win the Go game more effectively if it could make the human play less well, realize that to make the human play less well it needs to learn more about humans, learn about humans needing sleep and sleep becoming less good when interrupted, learn about humans waking up to answer phone calls, learn how phones work, learn that some Internet services connect to phones...
If we consider an actual game of Go, rather than a logical game of Go, then the state of the Go board at the end of the game is produced by an enormous and tangled causal process that includes not just the proximal moves, but the AI algorithm that chooses the moves, the cluster the AI is running on, the humans who programmed the cluster; and also, on the other side of the board, the human making the moves, the professional pride and financial prizes motivating the human, the car that drove the human to the game, the amount of sleep the human got that night, all the things all over the world that didn't interrupt the human's sleep but could have, and so on. There's an enormous lattice of causes that lead up to the AI's and the human's actual Go moves.
We can see the cognitive job of an agent in general as "select policies or actions which lead to a more preferred outcome". The enormous lattice of real-world causes leading up to the real-world Go game's final position, means that an enormous set of possible interventions could potentially steer the real-world future into the region of outcomes where the AI won the Go game. But these causes are going through all sorts of different domains on their way to the final outcome, and correctly choosing from the much wider space of interventions means you need to understand all the domains along the way. If you don't understand humans, understanding phones doesn't help; the prank phone call event goes through the sleep deprivation event, and to correctly model events having to do with sleep deprivation requires knowing about humans.
To the extent one credits the existence of 'significantly more general than chimpanzee intelligence', it implies that there are common cognitive subproblems of the huge variety of problems that humans can (learn to) solve, despite the surface-level differences of those domains. Or at least, the way humans solve problems in those domains, the cognitive work we do must have deep commonalities across those domains. These commonalities may not be visible on an immediate surface inspection.
Imagine you're an ancient Greek who doesn't know anything about the brain having a visual cortex. From your perspective, ship captains and smiths seem to be doing a very different kind of work; ships and anvils seem like very different objects to know about; it seems like most things you know about ships don't carry over to knowing about anvils. Somebody who learns to fight with a spear, does not therefore know how to fight with a sword and shield; they seem like quite different weapon sets.
(Since, by assumption, you're an ancient Greek, you're probably also not likely to wonder anything along the lines of "But wait, if these tasks didn't all have at least some forms of cognitive labor in common deep down, there'd be no reason for humans to be simultaneously better at all of them than other primates.")
Only after learning about the existence of the cerebral cortex and the cerebellum and some hypotheses about what those parts of the brain are doing, are you likely to think anything along the lines of:
"Ship-captaining and smithing and spearfighting and swordfighting look like they all involve using temporal hierarchies of chunked tactics, which is a kind of thing the cortical algorithm is hypothesized to do. They all involve realtime motor control with error correction, which is a kind of thing the cerebellar cortex is hypothesized to do. So if the human cerebral cortex and cerebellar cortex are larger or running better algorithms than chimpanzees' cerebrums and cerebellums, humans being better at learning and performing this kind of deep underlying cognitive labor that all these surface-different tasks have in common, could explain why humans are simultaneously better than chimpanzees at learning and performing shipbuilding, smithing, spearfighting, and swordfighting."
This example is hugely oversimplified, in that there are far more differences going on between humans and chimpanzees than just larger cerebrums and cerebellums. Likewise, learning to build ships involves deliberate practice which involves maintaining motivation over long chains of visualization, and many other cognitive subproblems. Focusing on just two factors of 'deep' cognitive labor and just two mechanisms of 'deep' cognitive performance is meant more as a straw illustration of what the much more complicated real story would look like.
But in general, the hypothesis of general intelligence seems like it should cash out as some version of: "There's some set of new cognitive algorithms, plus improvements to existing algorithms, plus bigger brains, plus other resources--we don't know how many things like this there are, but there's some set of things like that--which, when added to previously existing primate and hominid capabilities, created the ability to do better on a broad set of deep cognitive subproblems held in common across a very wide variety of humanly-approachable surface-level problems for learning and manipulating domains. And that's why humans do better on a huge variety of domains simultaneously, despite evolution having not preprogrammed us with new instinctual knowledge or algorithms for all those domains separately."
The above view suggests a directional bias of uncorrected intuition: Without an explicit correction, we may tend to intuitively underestimate the similarity of deep cognitive labor across seemingly different surface problems.
On the surface, a ship seems like a different object from a smithy, and the spear seems to involve different tactics from a sword. With our attention going to these visible differences, we're unlikely to spontaneously invent a concept of 'realtime motor control with error correction' as a kind of activity performed by a 'cerebellum'--especially if our civilization doesn't know any neuroscience. The deep cognitive labor in common goes unseen, not just because we're not paying attention to the invisible constants of human intelligence, but because we don't have the theoretical understanding to imagine in any concrete detail what could possibly be going on.
This suggests an argument from predictable updating: if we knew even more about how general intelligence actually worked inside the human brain, then we would be even better able to concretely visualize deep cognitive problems shared between different surface-level domains. We don't know at present how to build an intelligence that learns a par-human variety of domains, so at least some of the deep commonalities and corresponding similar algorithms across those domains, must be unknown to us. Then, arguendo, if we better understood the true state of the universe in this regard, our first-order/uncorrected intuitions would predictably move further along the direction that our belief previously moved when we learned about cerebral cortices and cerebellums. Therefore, to avoid violating probability theory by foreseeing a predictable update, our second-order corrected belief should already be that there is more in common between different cognitive tasks than we intuitively see how to compute.
Few people in the field would outright disagree with either the statement "humans have significantly more widely applicable cognitive abilities than other primates" or, or the other side, "no matter how intelligent you are, if your brain fits inside the physical universe, you might not be able to reverse SHA-4096 hashes". But even taking both those statements for granted, there seems to be a set of policy-relevant factual questions about, roughly, to what degree general intelligence is likely to shorten the pragmatic distance between different AI capabilities.
For example, consider the following (straw) amazing simple solution to all of AI alignment:
"Let's just develop an AI