2675

LESSWRONG
LW

2674
Personal Blog

10

[link] Simplifying the environment: a new convergent instrumental goal

by Kaj_Sotala
22nd Apr 2016
1 min read
4

10

Personal Blog

10

[link] Simplifying the environment: a new convergent instrumental goal
8gwern
2Gunnar_Zarncke
4lukeprog
3Kaj_Sotala
New Comment
4 comments, sorted by
top scoring
Click to highlight new comments since: Today at 12:25 PM
[-]gwern9y80

All stable processes we shall predict. All unstable processes we shall control.

Reply
[-]Gunnar_Zarncke9y20

Sounds like the Serenity Prayer for AI.

Reply
[-]lukeprog9y40

See also: https://scholar.google.com/scholar?cluster=9557614170081724663&hl=en&as_sdt=1,5

Reply
[-]Kaj_Sotala9y30

Neat, thanks!

Reply
Moderation Log
More from Kaj_Sotala
View more
Curated and popular this week
4Comments

http://kajsotala.fi/2016/04/simplifying-the-environment-a-new-convergent-instrumental-goal/

Convergent instrumental goals (also basic AI drives) are goals that are useful for pursuing almost any other goal, and are thus likely to be pursued by any agent that is intelligent enough to understand why they’re useful. They are interesting because they may allow us to roughly predict the behavior of even AI systems that are much more intelligent than we are.

Instrumental goals are also a strong argument for why sufficiently advanced AI systems that were indifferent towards human values could be dangerous towards humans, even if they weren’t actively malicious: because the AI having instrumental goals such as self-preservation or resource acquisition could come to conflict with human well-being. “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”

I’ve thought of a candidate for a new convergent instrumental drive: simplifying the environment to make it more predictable in a way that aligns with your goals.