Intro

As a voracious consumer of AI Safety everything, I have come across a fair few arguments of the kind "either we align AGI and live happily ever after, or we don't and everyone dies." I too subscribed to this worldview until I realised:

a) We might not actually create AGI (due to e.g legal controls)

b) The future cannot usually be described with 1 bit of information.

This post is therefore my attempt to wrangle the future into broad categories as best I can. I've tried to use clear boundaries, but I fully expect some bizarre possibilities, such as those discussed in Building Weirdtopia,^[1] to escape sensible categorisation. I equally hope to write "life in the day of" posts for each category soon as a visualisation of what each of these worlds entails.

Setting

Setting up this view of future worlds as "endgames" requires the rather large assumption that the future stays relatively stationary. Given the history of humankind, with the rise and fall of empires over millennia, it seems regimes good and bad have existed and coexisted in what looks under certain lenses something like a moral random walk over time. Indeed, historically increasing technological progress has tended to mean people's situations have changed more quickly as time has passed; extrapolation would suggest this continues for superintelligence. So how can I justify rating any single future, let alone entire classes of them, by moral desirability?

I think there are a few considerations here. First, I will acknowledge I am largely utilitarian in nature and do not mind shrugging and estimating the quality of life of the average person in any given world. 

Second, there is likely an upper bound to what is theoretically possible with science. I don't think we're close yet, but an intelligence explosion would likely bring us out of the exponential part of the sigmoid curve. The timelines I am considering for the endgames are 100-200+ years away, which could bring us to the point where science has slowed significantly, allowing a stable state to form.

Third, I think that being ruled by a superintelligence, as happens in many scenarios, will have a strong stabilising effect on the variability of the quality of human experience, largely due to most plausible superintelligences having stable values^[2], and optimising according to those.

Fourth, in multipolar scenarios where your quality of life depends greatly on where you are, I am very happy to describe that as multiple different endgames occurring at once.

Other possibilities are there: for example, a semi-aligned AI could alternate between beneficial and destructive policies if these are approximately equal according to its value function.

Additional notes: each category covers a broad set of worlds, so there will be overlap in the moral value of many of them.

Relevance

• It seems valuable to have a good idea what the possibilities humanity is playing with are: you don't play chess as well if you don't realise a draw is a possibility
• Improved world models
• Finding new threat models (I noticed multiple possibilities for the future through doing this exercise which I wouldn't have otherwise)

Methods

So I've basically gone through this in two phases: 

1. Attempting to think of all the ways it seems sensible to me for the world to end up in.
2. Formalising the assumptions required for those worlds to exist and then checking whether this logically includes all worlds.

I have structured these assumptions as a tree, with later branches containing more assumptions than the higher ones.

Assumptions

In doing this breakdown, we have split worlds down by binary metrics. The issue with this is many worlds will be ambiguous in their categorisation and therefore not intuitively fit into the descriptions given by these categories. If people have suggestions for critical distinctions I have missed, feel free to put them in the comments.

Here is a more technical description of the branches in the tree:

Major Catastrophe: A disaster occurs which is large enough that we have not recovered our current technological level 100 years from now.

ASI Created: An Artificial Intelligence is created which convincingly outperforms all humans at all cognitive tasks. i.e No real-world relevant task is in practise found (restricted to the set of tasks which can be performed with a computer) where the top human scores a greater than 50% win rate against the computer.

Technical Alignment Solved: The technical problem of ensuring that an abstract human preference can be specified to any AI, with the AI then keeping this preference out of distribution with a known, >90% probability, is solved.

Implemented with a minus: A fully aligned ASI training schedule is performed with a minus sign flipped as explained here. Said AI is not contained.

AI Takeover: AI ends up in control of the planet's resources, without humans having a means to regain that control.

Aligned to Creators: AI is specifically aligned to the will of the segment of society which created it (with possible govt intervention). Rest of society has little to no control over AI use and deployment.

Kill Everyone: >99.9% population decline over the 100 years following takeover.

AI Centralised: ASI technology is controlled by <= 10 companies which have ~sole influence over its use.

Value Net Positive: >50% of humans in resulting world would prefer to be alive than dead. (In a Coherent Extrapolated Volition sense, without being e.g drugged to say this).

Humans better off than today: It would be preferable on average for humans to live in this world than in 2025.

Aligned to sentience in general: The welfare of non-human animals is valued in the structure of society, as is that of possibly sentient AIs.

Results

Here is the tree as I have designed it.

EDIT: Note that to compress the tree I have combined the "Yes" and "No" branches of "Alignment solved" either side of the AI Takeover branch.

Ranking

Next to the end nodes of the graph, I have labelled the outcomes according to my personal preference for the broad categories (remember, the categories will likely overlap, with some worlds within them being better than others). 

1. ASI Eutopia - AI is aligned and controlled by certain centralised firms, but the values are controlled by society, while respecting other forms of consciousness such as non-human animals and (potentially) AI itself. General flourishing expected.
2. Decentralised ASI - Many members, groups in society (maybe everyone) has control of a superintelligence. Assuming we have made it a couple of hundred years in the future, this presumably means that there are good systems for preventing complete anarchy, although I expect there may still be a few negative externalities from misuse.
3. No ASI - Humanity collectively chooses not to build superintelligence (or cannot build it). History continues with technological gains from lesser intelligences.
4. Semi-Aligned ASI - AI is aligned and controlled by certain centralised firms, but the values are controlled by society. Humans are the sole beneficiaries, and in many of these worlds things like factory farms still exist, pushing this category down the list.
5. ASI Aligned to Creators - AI is aligned, but controlled more or less solely by the company and/or government which created it. How nice it is to be a part of this society therefore depends on the values of creators of the superintelligence.
6. Misaligned AI without Takeover^[3] - AI is misaligned but decides against takeover. This could be because of strong control methods or subtle arguments not obvious to humans.
7. AI Takeover with positive externalities - AI takes over the world, with positive outcomes for humans (e.g new tech, AI optimises for something close enough to the human utility function)^[4].
8. AI Tyranny - AI takes over the world, without killing everyone. Humans live on, but in worse conditions than today.
9. Major Catastrophe - A significant disaster takes out a large fraction of human civilisation. This could be AI related or not - examples include disasters such as nuclear war, disastrous effects from climate change or pandemics due to engineered bioweapons.
10. Everyone dies - The straightforward x-risk scenario. The AI sets out to kill everyone on the planet. Technically I have defined this as >99.9% population decline, although I expect the population to continue to decrease in most worlds where the death toll is not 100%.
11. S-Risk - AI takes over, and creates a world with humans who would prefer to die. Examples include putting people in factory farms where upvotes on AI answers are continuously sought.
12. Hyper risk - AI explicitly optimised for human suffering. I could give a more in depth description but would rather not.

Note that some of these endgames are extremely unlikely (e.g no 12 requires the unlikely event of a sign flip on the utility function). The goal was to categorise all possible ones. If I have missed anything, please put it in the comments.

1. ^{^}

   I am aware that Yudkowsky was talking about how an ideal future will probably seem strange to us, but I imagine many non-ideal futures will appear equally odd.

2. ^{^}

   At least, most superintelligences which don't blow up like those subject to substrate needs convergence do

3. ^{^}

   This is the scenario I am least certain of how to rank. I would be interested in people's thoughts here.

4. ^{^}

   Personally I feel the most likely scenario for this is via a superintelligence with behaviour largely predicted by something stemming from Janus's simulator theory, making it more "human" in its actions.