Anthropomorphization strikes me as a big problem in AI safety research. People intuitively ascribe human attributes to AGI, even though humans are only one tiny subset of the space of all possible forms of intelligence.
I would like to compile a list of the most crucial differences between AGI and humans, to help with research. Here are a few to start with:
- Humans can not be copied. AI systems can be copied at will.
- Humans can not reset their state of mind. AI systems can be backed up and reset at will.
- Human thoughts are opaque to ourselves. AI's can be trained to access their own source code and logs of their thought processes.
- Human thoughts are opaque to other humans. An AI's thoughts will probably be incomprehensible to other AI by default, but there is ongoing research to make it transparent. It seems possible to construct a training scheme that encourages AI to keep its own thoughts legible to other AI.
- Humans are relatively sure about the nature of their reality. We never seriously have to ask ourselves if we are currently in a simulation, or the like. It happens in science fiction, and for all we know we really could be in a simulation, but it's not a theory that most people assign a high probability. An AI on the other hand might be trained on a curriculum of games that are all extremely different from each other. It may not have a single model of "true reality" like humans do, but many.
What other differences do you think are important?