I don't know much about this field, but here are my thoughts on my lovely friend davey's piece. Engagement welcome:
Intro:
Some concern about the natural selection argument:
First, what's involved in your notion of self-interest? E.g. a bee is self-interested, but not reflexively— self-conscious self-interest is a different issue (that doesn't seem to be straightforwardly implied by any selection principle. I'm not convinced we humans, endowed with self-conscious self-interest, will be evolutionarily selected for in the long run). What suggests superintelligence has any use for self-definition?
Next, why should we believe that the essential features of natural selection will carry over to AI agent selection? (e.g. agents don’t evolve, species do through reproduction— do AI agents... (read 770 more words →)
I don't know much about this field, but here are my thoughts on my lovely friend davey's piece. Engagement welcome:
Intro:
Some concern about the natural selection argument:
First, what's involved in your notion of self-interest? E.g. a bee is self-interested, but not reflexively— self-conscious self-interest is a different issue (that doesn't seem to be straightforwardly implied by any selection principle. I'm not convinced we humans, endowed with self-conscious self-interest, will be evolutionarily selected for in the long run). What suggests superintelligence has any use for self-definition?
Next, why should we believe that the essential features of natural selection will carry over to AI agent selection? (e.g. agents don’t evolve, species do through reproduction— do AI agents... (read 770 more words →)