This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback. 

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

There are many fictional stories that depict unaligned non-human entities in ways that can illustrate some aspects of what AI misalignment might look like.

Many traditional fables about "being careful what you wish for" — such as the stories of King Midas, The Sorcerer’s Apprentice, literal genies, and some traditional Jewish Golems — hinge on a character getting what they literally ask for, but failing to anticipate the full consequences or side effects. These stories illustrate how the complexity and fragility of human values make outer misalignment a real danger, and how a genie-like superintelligence that does not care about what we really want could lead to catastrophic consequences.^[1]

Some stories (Mission: Impossible – Dead Reckoning, 2001: A Space Odyssey, Ex Machina) portray AIs that have been tasked with seemingly beneficial or morally neutral aims but are misaligned in important ways from their creators.

Some stories (Mission: Impossible – Dead Reckoning, The Terminator) portray AI that very explicitly attempts to take over the world, whereas others (Ex Machina, Upgrade, 2001: A Space Odyssey) have AI with more restricted or ambiguous aims that still bring it into conflict with its creators or users. When the AI is depicted as trying to take over the world, this takeover is not usually its final goal but comes about as an instrumentally convergent goal.

In Isaac Asimov’s I, Robot series of novels, the characters attempt to constrain the behavior of robots with the Three Laws of Robotics. The stories explore how these laws are insufficient to guarantee a good outcome. For instance, in the story “Liar!”, a robot is forced to violate the First Law (do not harm humans). Either it can hurt a human by telling them the truth, or hurt them by keeping the truth from them, and the First Law doesn’t specify how to think through such situations.

The Terminator movies are often used as a pop-culture reference for AI risk. Many in the field dislike when AI risk is illustrated this way because pop culture views Skynet as malicious rather than simply amoral. The movies also involve android killer robots, which are unlikely to be used.^[2] Others have pushed back^[3] and argued that there are some relevant aspects of these movies, including:

• Skynet understands that humans are a threat to its goals and attempts extermination as an instrumental goal.
• Competitive dynamics make the rise of Skynet hard to avoid, mirroring the AI arms race.
• Skynet fighting back when humans try to disable it illustrates how not being shut down is an instrumentally convergent goal.

We should avoid generalizing from fictional evidence and take these comparisons as illustrations rather than arguments. These exact scenarios are unlikely, and there are possible misalignment scenarios that could be very dangerous but would not make for interesting media and thus are not covered as much in such stories.

1. ^{^}

   Some have argued that this kind of misalignment is less likely because LLMs appear to understand human intent, but this is a point of active debate in the field.

2. ^{^}

   Why on earth would you make a killer robot in human form? It’s so inefficient! A true superintelligence would never clothe itself in the form of the inferior meat-bags.

3. ^{^}

   People who have pushed back include Matt Yglesias, Skluug, and Hein de Haan.