NotFromConcentrate's Shortform

NotFromConcentrate

This is a special post for quick takes by NotFromConcentrate. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I was recently chatting to a collaborator about how we should model extreme power concentration ("EPC") from a game theoretic perspective.

Here's a (heavily-paraphrased) version of their central claim:

There’s a lot of value to be gained from thinking about power concentration problems through the lens of coordination failures.
Installing and upholding a tyrannical regime is a major hassle. Being in a position of power often doesn’t make people feel particularly good, but rather very paranoid – and justifiably so!
Notable examples of this are Stalin, Hitler, Saddam Hussein, Enver Hoxha, Qin Shi Huang.
People often try to seize and hold extreme power because there's no way for them to get credible commitments from rivals to respect their core interests, if those rivals take power.
So – when we think about EPC interventions, we should think hard about how we can help different power-holders more easily negotiate / coordinate with each other to achieve their objectives.

I think this isn’t quite right, because I don't think it's clear that all players prefer to cooperate in the EPC game. That’s because:

Space is vast, and the potential for ASI-enabled preference satisfaction using those resources is also vast.
- So, the prize for defecting dwarfs the costs.
At the same time, it's not that hard / costly for a power-seeker ("PS") to get all the way to EPC if they play their cards right during the intelligence explosion.
So, PS has two options:
1. Cooperate: leading to a ~guarantee that she gets a relatively small amount of all space resources.
2. Defect: leading to a ~5% chance she gets ~all the space resources
Given these two choices – whether it's a coordination game or not depends on PS's preference profile.
- e.g. if PS is risk-averse, she might prefer the guarantee
- e.g. if she's a major risk-seeker, then she probably prefers defecting

In a normal human world where nobody is getting ASI advice, I think it's reasonable to expect everyone to be quite risk averse, and for being very powerful to actually kinda suck (c.f. Stalin), and thus to model power concentration as a coordination game.

But I just don't think that's the game we're playing during the intelligence explosion.

There’s a lot of value to be gained from thinking about power concentration problems through the lens of coordination failures.
Installing and upholding a tyrannical regime is a major hassle. Being in a position of power often doesn’t make people feel particularly good, but rather very paranoid – and justifiably so!
Notable examples of this are Stalin, Hitler, Saddam Hussein, Enver Hoxha, Qin Shi Huang.
People often try to seize and hold extreme power because there's no way for them to get credible commitments from rivals to respect their core interests, if those rivals take power.
So – when we think about EPC interventions, we should think hard about how we can help different power-holders more easily negotiate / coordinate with each other to achieve their objectives.

I think this isn’t quite right, because I don't think it's clear that all players prefer to cooperate in the EPC game. That’s because:

Space is vast, and the potential for ASI-enabled preference satisfaction using those resources is also vast.
- So, the prize for defecting dwarfs the costs.
At the same time, it's not that hard / costly for a power-seeker ("PS") to get all the way to EPC if they play their cards right during the intelligence explosion.
So, PS has two options:
1. Cooperate: leading to a ~guarantee that she gets a relatively small amount of all space resources.
2. Defect: leading to a ~5% chance she gets ~all the space resources
Given these two choices – whether it's a coordination game or not depends on PS's preference profile.
- e.g. if PS is risk-averse, she might prefer the guarantee
- e.g. if she's a major risk-seeker, then she probably prefers defecting

But I just don't think that's the game we're playing during the intelligence explosion.

NotFromConcentrate's Shortform

1