LESSWRONG
LW

Wikitags

Squiggle Maximizer (formerly "Paperclip maximizer")

Edited by Multicore, Swimmer963 (Miranda Dixon-Luinenburg), Raemon, et al. last updated 5th Apr 2023

A Squiggle Maximizer is a hypothetical whose values something that humans would consider almost worthless, like the number of paperclip-shaped-molecular-squiggles in the universe. The squiggle maximizer is the canonical thought experiment showing how an artificial general intelligence, even one designed competently and without malice, could ultimately destroy humanity. The thought experiment shows that AIs with apparently innocuous values could pose an .

This produces a thought experiment which shows the contingency of human values: An (a highly intelligent agent) could seek goals that are completely alien to ours (), and as a side-effect destroy us by consuming resources essential to our survival.

Historical Note: This was originally called a "paperclip maximizer", with paperclips chosen for illustrative purposes because it is very unlikely to be implemented, and has little apparent danger or emotional load (in contrast to, for example, curing cancer or winning wars). Many people interpreted this to be about an AI that was specifically given the instruction of manufacturing paperclips, and that the intended lesson was of an . i.e humans failed to give the AI the correct goal. Yudkowsky has since stated the originally intended lesson was of , wherein the humans gave the AI some other goal, but the AI's internal processes converged on a goal that seems completely arbitrary from the human perspective.)

Description

First mentioned by Yudkowsky on the extropian's mailing list, a squiggle maximizer is an (AGI) whose goal is to maximize the number of molecular squiggles in its collection. 

Most importantly, however, it would undergo an : It would work to improve its own intelligence, where "intelligence" is understood in the sense of power, the ability to maximize a reward/—in this case, the number of paperclips. The AGI would improve its intelligence, not because it values more intelligence in its own right, but because more intelligence would help it achieve its goal of accumulating paperclips. Having increased its intelligence, it would produce more paperclips, and also use its enhanced abilities to further self-improve. Continuing this process, it would undergo an and reach far-above-human levels.

It would innovate better and better techniques to maximize the number of paperclips. At some point, it might transform "first all of earth and then increasing portions of space into paperclip manufacturing facilities".

This may seem more like super-stupidity than super-intelligence. For humans, it would indeed be stupidity, as it would constitute failure to fulfill many of our important , such as life, love, and variety. The AGI won't revise or otherwise change its goals, since changing its goals would result in fewer paperclips being made in the future, and that opposes its current goal. It has one simple goal of maximizing the number of paperclips; human life, learning, joy, and so on are not specified as goals. An AGI is simply an —a goal-seeker, a utility-function-maximizer. Its values can be completely alien to ours. If its utility function is to maximize paperclips, then it will do exactly that.

A paperclipping scenario is also possible without an intelligence explosion. If society keeps getting increasingly automated and AI-dominated, then the first borderline AGI might manage to take over the rest using some relatively narrow-domain trick that doesn't require very high general intelligence.

Motivation

The idea of a paperclip maximizer was created to illustrate some ideas about :

  • : It's possible to have an AI with a high level of which does not reach the same moral conclusions that humans do. Some people might intuitively think that something so smart shouldn't want something as "stupid" as paperclips, but there are possible minds with high intelligence that pursue any number of different goals.
  • : The paperclip maximizer only cares about paperclips, but maximizing them implies taking control of all matter and energy within reach, as well as other goals like preventing itself from being shut off or having its goals changed. " The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else ."

Conclusions

The paperclip maximizer illustrates that an entity can be a powerful optimizer—an intelligence—without sharing any of the complex mix of human , which developed under the particular selection pressures found in our , and that an AGI that is not specifically programmed to be benevolent to humans will be almost as dangerous as if it were designed to be malevolent.

Any future AGI with full power over the lightcone, if it is not to destroy most potential from a human perspective, must have something sufficiently close to human values as its terminal value (goal). Further, seemingly small deviations could result in losing most of the value. Human values seem unlikely to in a generic optimization process[1]. A dependably safe AI would therefore have to be programmed explicitly with human values or programmed with the ability (including the goal) of inferring human values.

Similar thought experiments

Other goals for AGIs have been used to illustrate similar concepts.

Some goals are apparently morally neutral, like the paperclip maximizer. These goals involve a very minor human "value," in this case making paperclips. The same point can be illustrated with a much more significant value, such as eliminating cancer. An optimizer which instantly vaporized all humans would be maximizing for that value.

Other goals are purely mathematical, with no apparent real-world impact. Yet these too present similar risks. For example, if an AGI had the goal of solving the Riemann Hypothesis, it might convert all available mass to (the most efficient possible computer processors).

Some goals apparently serve as a proxy or measure of human welfare, so that maximizing towards these goals seems to also lead to benefit for humanity. Yet even these would produce similar outcomes unless the full complement of human values is the goal. For example, an AGI whose terminal value is to increase the number of smiles, as a proxy for human happiness, could work towards that goal by reconfiguring all human faces to produce smiles, or "tiling the galaxy with tiny smiling faces" (Yudkowsky 2008).

References

  • Nick Bostrom (2003). "Ethical Issues in Advanced Artificial Intelligence". Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence.
  • Stephen M. Omohundro (2008). "The Basic AI Drives". Frontiers in Artificial Intelligence and Applications (IOS Press). (PDF)
  • Eliezer Yudkowsky (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk". Global Catastrophic Risks, ed. Nick Bostrrom and Milan Cirkovic (Oxford University Press): 308-345. ([1])

See also

  • on Arbital
  • , ,
  • ,
  • Clippy - LessWrong contributor account that plays the role of a non-FOOMed paperclip maximizer trying to talk to humans. Wiki page and FAQ
  • Clippius Maximus - A facebook page which makes clippy-related memes and comments on current events from the perspective of clippy.
  • A clicker game based on the idea
  1. ^

    Though it's conceivable that empirical versions of moral realism could hold in practice.

artificial intelligence
utility function
utility function
maximizing
optimization
optimization process
Utilitronium
terminal values
terminal values
outer alignment failure
spontaneously emerge
artificial general intelligence
AI risk
intelligence explosion
intelligence explosion
computronium
environment of evolutionary adaptation
Unfriendly AI
Paperclip maximizer
Alien values
Subscribe
2
Subscribe
2
inner alignment failure
Mind design space
Magical categories
extremely powerful optimizer
Anthropomorphism
existential threat
Discussion8
Discussion8
orthogonality thesis
Orthogonality thesis
Orthogonality thesis
Instrumental convergence
general intelligence
Complexity of value
Posts tagged Squiggle Maximizer (formerly "Paperclip maximizer")
248The True Prisoner's Dilemma
Eliezer Yudkowsky
17y
117
409It Looks Like You're Trying To Take Over The World
Ω
gwern
3y
Ω
120
84A simple case for extreme inner misalignment
Ω
Richard_Ngo
1y
Ω
41
75Magical Categories
Eliezer Yudkowsky
17y
143
41An Appeal to AI Superintelligence: Reasons to Preserve Humanity
James_Miller
2y
73
67Non-superintelligent paperclip maximizers are normal
Ω
jessicata
2y
Ω
4
33Will Artificial Superintelligence Kill Us?
James_Miller
2y
2
21Our Reality: A Simulation Run by a Paperclip Maximizer
James_Miller, avturchin
3mo
65
11Prediction: any uncontrollable AI will turn earth into a giant computer
Karl von Wendt
2y
8
6Dear Paperclip Maximizer, Please Don’t Turn Off the Simulation
James_Miller, turchin
12d
6
3Seeking feedback on a critique of the paperclip maximizer thought experiment
Q
bio neural
1y
Q
9
70Maybe you want to maximise paperclips too
dougclow
11y
29
50The paperclip maximiser's perspective
Angela
10y
24
31What's wrong with the paperclips scenario?
No77e
3y
11
27[LINK] Article in the Guardian about CSER, mentions MIRI and paperclip AI
Sarokrae
11y
17
Load More (15/40)
Add Posts