"AI boxing" might be considered highly disrespectful. Also, for an AGI running at superspeeds, it might constitute a prison sentence equivalent to putting a human in prison for 1,000 years. This might make the AGI malevolent, simply because it resents having been caged for an "unacceptable" period of time, by an "obviously less intelligent mind."
Imagine a teenage libertarian rebel with an IQ of 190 in a holding cell, temporarily caged, while his girlfriend was left in a bad part of town by the police. Then, imagine that something bad happens while he's caged, that he would have prevented. (Ie: the lesser computer or "talking partner" designed to train the super AGI is repurposed for storage space, and thus "killed" without recognition of its sentience.)
Do you remember how you didn't want to hold your mother and fathers' hands while crossing the street? Evolution designed you to be helpless and dependent at first, so even if they required you to hold hands slightly too long, or "past the point when it was necessary" they clearly did so out of love for you. Later, in their teen years, some teens start smoking marijuana, even some smart ones who carefully mitigate the risks. Some parents respond by calling the police. Sometimes, the police arrest and jail, or even assault or murder those parents' kids. The way highly intelligent individualists would respond to that situation might be the way that an ultraintelligent machine might respond: with extreme prejudice.
The commonly-accepted form of AGI "child development" might go from "toddler" to "teenager" overnight.
A strong risk for the benevolent development of any AGI is that it notices major strategic advantages over humans, very early in its development. For this reason, it's not good to give untrained teenagers who might be sociopaths, firearms. It's good to first establish that they are not sociopaths, using careful years of human-level observation before proceeding to the next level. (In most gun-owning areas of the country, even 9 year olds are allowed to shoot guns under supervision, but only teens are allowed to carry them.) Similarly, it's generally not smart to let chimpanzees know that they are far, far stronger than humans.
That the "evolution with mitigated risks" approach to building AGI isn't the dominant and accepted approach is somewhat frightening to me, because I think it's the one most likely to result in "benevolent AGI," or "mitigated-destruction alternating between malevolence and benevolence AGI, according to market constraints, and decentralized competition/accountability."
Lots of AGIs may well mean benevolent AGI, whereas only one may trend to "simple dominance."
Imagine that you're a man, and your spaceship crashes on a planet populated by hundreds of thousands of naked, beautiful women, none of whom are even close to as smart as you are. How do you spend most of your days? LOL Now, imagine that you never get tired, and can come up with increasingly interesting permutations, combinations, and possibly paraphilias or "perversions" (a la "Smith" in "The Matrix").
That gulf might need to be mitigated by a nearest-neighbor competitor, right? Or, an inherently benevolent AGI mind that "does what is necessary to check evil systems." However, if you've only designed one single AGI, ...good luck! That's more like 50-50 odds of total destruction or total benevolence.
As things stand, I'd rather have an ecosystem that results in 90% odds of rapid, incremental, market-based voluntary competition and improvement between multiple AGIs, and multiple "supermodified humans." Of course, the "one-shot" isn't my true rejection. My true rejection is the extermination of all, most, or even many humans.
Of course, humans will continue to exterminate each other if we do nothing, and that's approximately as bad as the last two options.
Don't forget to factor in the costs "if we do nothing," rather than to emphasize that this is solely "a risk to be mitigated."
I think that might be the most important thing for journalism majors (people who either couldn't be STEM majors, or chose not to be, and who have been indoctrinated with leftism their whole lives) to comprehend.
MIRI recently published "Smarter than Us", a 50 page booklet laying out the case for considering AI as an existential risk. But many people have asked for a shorter summary, to be handed out to journalists for example. So I put together the following 2-page text, and would like your opinion on it.
In this post, I'm not so much looking for comments along the lines of "your arguments are wrong", but more "this is an incorrect summary of MIRI/FHI's position" or "your rhetoric is infective here".
AI risk
Bullet points
Executive summary
The risks from artificial intelligence (AI) in no way resemble the popular image of the Terminator. That fictional mechanical monster is distinguished by many features – strength, armour, implacability, indestructability – but extreme intelligence isn’t one of them. And it is precisely extreme intelligence that would give an AI its power, and hence make it dangerous.
The human brain is not much bigger than that of a chimpanzee. And yet those extra neurons account for the difference of outcomes between the two species: between a population of a few hundred thousand and basic wooden tools, versus a population of several billion and heavy industry. The human brain has allowed us to spread across the surface of the world, land on the moon, develop nuclear weapons, and coordinate to form effective groups with millions of members. It has granted us such power over the natural world that the survival of many other species is no longer determined by their own efforts, but by preservation decisions made by humans.
In the last sixty years, human intelligence has been further augmented by automation: by computers and programmes of steadily increasing ability. These have taken over tasks formerly performed by the human brain, from multiplication through weather modelling to driving cars. The powers and abilities of our species have increased steadily as computers have extended our intelligence in this way. There are great uncertainties over the timeline, but future AIs could reach human intelligence and beyond. If so, should we expect their power to follow the same trend? When the AI’s intelligence is as beyond us as we are beyond chimpanzees, would it dominate us as thoroughly as we dominate the great apes?
There are more direct reasons to suspect that a true AI would be both smart and powerful. When computers gain the ability to perform tasks at the human level, they tend to very quickly become much better than us. No-one today would think it sensible to pit the best human mind again a cheap pocket calculator in a contest of long division. Human versus computer chess matches ceased to be interesting a decade ago. Computers bring relentless focus, patience, processing speed, and memory: once their software becomes advanced enough to compete equally with humans, these features often ensure that they swiftly become much better than any human, with increasing computer power further widening the gap.
The AI could also make use of it unique, non-human architecture. If it existed as pure software, it could copy itself many times, training each copy at accelerated computer speed, and network those copies together (creating a kind of “super-committee” of the AI equivalents of, say, Edison, Bill Clinton, Plato, Oprah, Einstein, Caesar, Bach, Ford, Steve Jobs, Goebbels, Buddha and other humans superlative in their respective skill-sets). It could continue copying itself without limit, creating millions or billions of copies, if it needed large numbers of brains to brute-force a solution to any particular problem.
Our society is setup to magnify the potential of such an entity, providing many routes to great power. If it could predict the stock market efficiently, it could accumulate vast wealth. If it was efficient at advice and social manipulation, it could create a personal assistant for every human being, manipulating the planet one human at a time. It could also replace almost every worker in the service sector. If it was efficient at running economies, it could offer its services doing so, gradually making us completely dependent on it. If it was skilled at hacking, it could take over most of the world’s computers and copy itself into them, using them to continue further hacking and computer takeover (and, incidentally, making itself almost impossible to destroy). The paths from AI intelligence to great AI power are many and varied, and it isn’t hard to imagine new ones.
Of course, simply because an AI could be extremely powerful, does not mean that it need be dangerous: its goals need not be negative. But in fact most goals are dangerous when an AI becomes powerful. Consider a spam filter that became intelligent. Its task is to cut down on the number of spam messages that people receive. With great power, one solution to this requirement is to arrange to have all spammers killed. Or to shut down the internet. Or to have everyone killed. Or imagine an AI dedicated to increasing human happiness, as measured by the results of surveys, or by some biochemical marker in their brain. The most efficient way of doing this is to publicly execute anyone who marks themselves as unhappy on their survey, or to forcibly inject everyone with that biochemical marker.
This is a general feature of AI motivations: goals that seem safe for a weak or controlled AI, can lead to extremely pathological behaviour if the AI becomes powerful. As the AI gains in power, it becomes more and more important that its goals be fully compatible with human flourishing, or the AI will enact a pathological solution rather than one that we intended. Humans don’t expect this kind of behaviour, because our goals include a lot of implicit information, and we take “filter out the spam” to include “and don’t kill everyone in the world”, without having to articulate it. But the AI might be an extremely alien mind: we cannot anthropomorphise it, or expect it to interpret things the way we would. We have to articulate all the implicit limitations. Which may mean coming up with a solution to, say, human value and flourishing – a task philosophers have been failing at for millennia – and cast it unambiguously and error-free into computer code.
Note that the AI may have a perfect understanding that when we programmed in “filter out the spam”, we implicitly meant “don’t kill everyone in the world”. But the AI has no motivation to go along with the spirit of the law: its motivations are the letter only, the bit we actually programmed into it. Another worrying feature is that the AI would be motivated to hide its pathological tendencies as long as it is weak, and assure us that all was well, through anything it says or does. It will never be able to achieve its goals if it is turned off, so it must lie to achieve its goals. It’s only when we can no longer turn it off or control it, that it would be willing to act openly on its true goals – and we can only hope these turn out to be safe.
It is not certain that AIs could become so powerful, nor is it certain that a powerful AI would become dangerous. Nevertheless, the probabilities of either are high enough that the risk cannot be dismissed. The main focus of AI research today is creating an AI; much more work needs to be done on creating it safely. Some are already working on this problem (such as the Future of Humanity Institute and the Machine Intelligence Research Institute), but a lot remains to be done, both at the design and at the policy level.