AI use disclosure: I dictated this text to Claude in a mix of Russian and English, then edited it.
The thoughts in this post are well known, but I find myself explaining them several times a week, mostly to people who are just entering the AI safety field, so I think it’s useful to have them written down briefly in one place.
Variants of the future
This is a rough map, not a complete one (skips gradual disempowerment, worlds with many AGIs, coexistence with an unaligned AGI, etc).
Normal technology. AI turns out to be a normal technology, with no AGI. We live in a world similar to today's.
Unaligned AGI. We summon AGI and fail to align it. Then we probably all die.
Aligned AGI. We create AGI and manage to align it. This splits in two: 3.1. Utopia. It is aligned with the values of all humanity. 3.2. Power concentration. It is mostly aligned to some elites. The default here is what I call capitalist hell. With AGI, human labor becomes obsolete, because capital can buy androids, data centers, and electricity instead. Owners of capital no longer need human labor, so it is worth nothing. Capital gets fixed in place: the rich stay rich, and the poor stay poor forever[1][2][3].
Motivations
Assume short timelines: if AGI comes, it comes within several years.
If you care about the future of humanity and think AI is not a normal technology (not world 1), you have only a few years left to affect which of the other worlds we get. You need to work really hard now, on AI safety or something similar.
If you care mostly about your own life, it depends on which world you expect:
We all die (world 2). Enjoy life. Don't save too much for retirement (still save some, just in case). Travel, spend time with friends and family, play video games.
Utopia (world 3.1). You can more or less do nothing and wait until it arrives.
Power concentration, capitalist hell (world 3.2). You probably have only a few more years in which you can exchange your labor for money. Work really hard now and earn as much as you can, so that you don't end up in the permanent underclass. This one is contested[4]: your savings may be too small to matter, or property rights may not survive.
In reality nobody is sure which world we are in, so you hold probabilities over all of them and your actual strategy is some mix of the above.
Young vs. old
Suppose there are two options (the numbers are made up):
AGI in 3 years. It has a 20% chance of killing everyone. In the other 80%, it has a 20% chance of curing all illnesses, including aging.
Pause, AGI in 15 years. The chance of curing all illnesses is still 20%, but the chance of killing everyone drops to 5%.
Suppose also that I care only about my own life, not about humanity or my children. If I'm 30, my chance of dying in the next 15 years is low, so I definitely prefer the second option. If I'm 75, my chance of dying of old age or illness in the next 15 years is pretty high, so I may well prefer the first.
Rulers and CEOs vs. regular people
Suppose you have a button that summons AGI: with 20% probability everyone dies, and with 80% you get lots of prosperity.
If you are the ruler of the country or the CEO of the company that creates AGI, and the AGI ends up aligned to you, a significant part of that 80% is that you become God Emperor of Earth, or one of a few God Emperors, maybe forever if AGI cures aging. For a regular person, pushing the button means the same 20% chance of dying, but the 80% only brings some prosperity. The risk is the same and the prize is much smaller, so a regular person has much less desire to push the button. Classic "privatize the profits, socialize the losses": the profit goes mostly to whoever pushes the button, risk of death is shared by everyone on Earth.
Race dynamics make this worse, which is a well-known point[5]. If the gamble is going to be played anyway, each ruler wants to be the one playing it, so that if it lands in the 80%, he is the God Emperor and not the ruler of the other country. The same holds between companies inside one country: the CEO whose company creates AGI will probably end up with far more power than the CEO who lost the race.
AI use disclosure: I dictated this text to Claude in a mix of Russian and English, then edited it.
The thoughts in this post are well known, but I find myself explaining them several times a week, mostly to people who are just entering the AI safety field, so I think it’s useful to have them written down briefly in one place.
Variants of the future
This is a rough map, not a complete one (skips gradual disempowerment, worlds with many AGIs, coexistence with an unaligned AGI, etc).
3.1. Utopia. It is aligned with the values of all humanity.
3.2. Power concentration. It is mostly aligned to some elites. The default here is what I call capitalist hell. With AGI, human labor becomes obsolete, because capital can buy androids, data centers, and electricity instead. Owners of capital no longer need human labor, so it is worth nothing. Capital gets fixed in place: the rich stay rich, and the poor stay poor forever[1][2][3].
Motivations
Assume short timelines: if AGI comes, it comes within several years.
If you care about the future of humanity and think AI is not a normal technology (not world 1), you have only a few years left to affect which of the other worlds we get. You need to work really hard now, on AI safety or something similar.
If you care mostly about your own life, it depends on which world you expect:
In reality nobody is sure which world we are in, so you hold probabilities over all of them and your actual strategy is some mix of the above.
Young vs. old
Suppose there are two options (the numbers are made up):
Suppose also that I care only about my own life, not about humanity or my children. If I'm 30, my chance of dying in the next 15 years is low, so I definitely prefer the second option. If I'm 75, my chance of dying of old age or illness in the next 15 years is pretty high, so I may well prefer the first.
Rulers and CEOs vs. regular people
Suppose you have a button that summons AGI: with 20% probability everyone dies, and with 80% you get lots of prosperity.
If you are the ruler of the country or the CEO of the company that creates AGI, and the AGI ends up aligned to you, a significant part of that 80% is that you become God Emperor of Earth, or one of a few God Emperors, maybe forever if AGI cures aging. For a regular person, pushing the button means the same 20% chance of dying, but the 80% only brings some prosperity. The risk is the same and the prize is much smaller, so a regular person has much less desire to push the button. Classic "privatize the profits, socialize the losses": the profit goes mostly to whoever pushes the button, risk of death is shared by everyone on Earth.
Race dynamics make this worse, which is a well-known point[5]. If the gamble is going to be played anyway, each ruler wants to be the one playing it, so that if it lands in the 80%, he is the God Emperor and not the ruler of the other country. The same holds between companies inside one country: the CEO whose company creates AGI will probably end up with far more power than the CEO who lost the race.
By default, capital will matter more than ever after AGI
The Intelligence Curse: Capital, AGI, and Human Ambition
For reasons to doubt capitalist hell scenario, see: It’s Still Easier To Imagine The End Of The World Than The End Of Capitalism
Against the "Permanent" Underclass
Racing to the Precipice: a Model of Artificial Intelligence Development