The greatest danger is that an arms race will lead to the creation of a superintelligence which will immediately be used to dominate all others. Speculative threats by an autonomous superintelligence are plausible but are less certain than the first-strike logic inherent in such an arms race.
Here's what we know from recent history: a) the instinct for domination is alive and well in the human species, and where circumstances allow an intelligent psychopath to reach the pinnacle of power, all available means will be deployed to maintain his (usually his... (read more)
I think Christiano's idea sets us on the right path. One of the things that troubled me in reading Bostrom's book is the prospect that a superintelligence under human control could be at least as dangerous as an autonomous superintelligence. For example, if superintelligence were hatched by one of the competing military superpowers -- Russia, China, or the U.S., let's say -- as an oracle or genie, it might be used for short-sighted or selfish purposes with catastrophic consequences. With superintelligence on his side, the cartoon nightmare of a single ma... (read more)
Unless it is deliberately or accidentally altered, an emulation will possess all of the evolved traits of human brains. These include powerful mechanisms to prevent an altruistic absurdity such as donating one's labor to an employer. (Pure altruism -- an act that benefits another at the expense of one's genetic interests -- is strongly selected against.) There are some varieties of altruism that survive: kin selection (e.g., rescuing a drowning nephew), status display (making a large donation to a hospital), and reciprocal aid (helping a neighbor in hope... (read more)
I’m grateful for these summaries and discussions. Having only just dived into the reading group, I ask forgiveness if I am overtracking some prior comment.
It seems to me that “human values” and “oversight” often go unexamined as we consider the risks and utility of superintelligence. I mean no disrespect to Katja in saying that (she’s summarizing, after all), but to say “human values” is either to reduce to the common denominator of our evolutionary psychology or to ignore the vast cultural and ideological diversity of humanity.
Either way, it’s a ... (read more)