LESSWRONG
LW

1911
Wikitags

Values handshakes

Edited by Yoav Ravid last updated 6th Apr 2021

Values handshakes are a proposed form of trade between superintelligences. From The Hour I First Believed by Scott Alexander:

Suppose that humans make an AI which wants to convert the universe into paperclips. And suppose that aliens in the Andromeda Galaxy make an AI which wants to convert the universe into thumbtacks.

When they meet in the middle, they might be tempted to fight for the fate of the galaxy. But this has many disadvantages. First, there’s the usual risk of losing and being wiped out completely. Second, there’s the usual deadweight loss of war, devoting resources to military buildup instead of paperclip production or whatever. Third, there’s the risk of a Pyrrhic victory that leaves you weakened and easy prey for some third party. Fourth, nobody knows what kind of scorched-earth strategy a losing superintelligence might be able to use to thwart its conqueror, but it could potentially be really bad – eg initiating vacuum collapse and destroying the universe. Also, since both parties would have superintelligent prediction abilities, they might both know who would win the war and how before actually fighting. This would make the fighting redundant and kind of stupid.

Although they would have the usual peace treaty options, like giving half the universe to each of them, superintelligences that trusted each other would have an additional, more attractive option. They could merge into a superintelligence that shared the values of both parent intelligences in proportion to their strength (or chance of military victory, or whatever). So if there’s a 60% chance our AI would win, and a 40% chance their AI would win, and both AIs know and agree on these odds, they might both rewrite their own programming with that of a previously-agreed-upon child superintelligence trying to convert the universe to paperclips and thumbtacks in a 60-40 mix.

This has a lot of advantages over the half-the-universe-each treaty proposal. For one thing, if some resources were better for making paperclips, and others for making thumbtacks, both AIs could use all their resources maximally efficiently without having to trade. And if they were ever threatened by a third party, they would be able to present a completely unified front.

Subscribe
Discussion
Subscribe
Discussion
Posts tagged Values handshakes
101[REPOST] The Demiurge’s Older Brother
Scott Alexander
8y
2
65How LDT helps reduce the AI arms race
Ω
Tamsin Leake
2y
Ω
13
47Superrational Agents Kelly Bet Influence!
Ω
abramdemski
4y
Ω
7
162Threat-Resistant Bargaining Megapost: Introducing the ROSE Value
Ω
Diffractor
3y
Ω
19
34Geometric Utilitarianism (And Why It Matters)
StrivingForLegibility
1y
2
26Negotiating Up and Down the Simulation Hierarchy: Why We Might Survive the Unaligned Singularity
David Udell
3y
14
24Even if we lose, we win
Morphism
2y
17
10Expected Utility, Geometric Utility, and Other Equivalent Representations
StrivingForLegibility
10mo
0
3Acausal trade naturally results in the Nash bargaining solution
Christopher King
2y
0
1Acausal Now: We could totally acausally bargain with aliens at our current tech level if desired
Christopher King
2y
5
1Could Roko's basilisk acausally bargain with a paperclip maximizer?
Christopher King
3y
8
Add Posts