The upcoming framing is analogous to a procedure asking :
"Is [this option] really giving [more options] to the highest number of beings?"


Takeaway : In the worst case, which many argue is by default, humans will create a maximizer. I argue that the least harmful thing for a maximizer (AI or not) to optimize is the [diversity of states in the universe], in a non-myopic procedure with ecosystemic synergy.
(Long-term co-affordance of options)

If the following analysis is right, it inherently turns a maximization into valuable outputs.
The reason behind this process unfolds the basis of tractable universal ethics.


This discussion is a reaction to jacob_cannell's post;
Which presented AI alignment through (altruistic) "empowerment of others"
 

TLDR : From what I read in the comments we could condense the issue to the difficulty of defining "power" (in this context "other-empowerment"; opposed to self-empowerment), because it may lead AI to "paralyze" humans/agents in order to "expand" action potential (which is very myopic), or inversely AI may forbid humans/agents to be sub-optimal (ie. you can't exile in the mountains).

To me the crucial node/difficulty is a threat (mentioned here) that AI will "erase all practical differences between you and the maximally longtermist version of you"; which is why I propose an alternative based on optionality and diversity.
 

                                          ────────────────────────                                                                                                                                                                                                                                                           
Universal optionality :
There is a unique opportunity in our space-time "zone" to bloom unique properties that do not exist in other places of the global space-time (so compared to other past, present and future space-time) 
+ The end-state ought to be with the most diversity possible
 

Open-ended evolution : “refers to the unbounded increase in complexity that seems to characterize evolution on multiple scales” (Corominas-Murtra, Seoane & Solé, 2018).

Irreducibility : "While many computations admit shortcuts that allow them to be performed more rapidly, others cannot be sped up. Computations that cannot be sped up by means of any shortcut are called computationally irreducible." Computational Irreducibility – Wolfram MathWorld 

Open-ended ecosystems have irreducible states/features/options that can only appear at the right unique ("endemic") speed rate,
It also creates an inherent uncertainty that gives humans/agents more degree of freedom (you can be somewhat sub-optimal and not strictly empower yourself).
 

                                          ────────────────────────
 

In the worst case, which many argue is by default, humans will create a maximizer.
I argue that the less harmful thing for a maximizer to optimize is the [diversity of states in the universe] in a non-myopic procedure with ecosystemic synergy.

By "diversity of states in the universe" I mean : 
To have the least myopic form of diversity possible, which takes into account futur and past states of the universe. This leads to the necessity of ecosystems, and caring about it; because you can't open the universe towards more states without ecosystems. Every action has to be carefully weighted, because it may destroy more states than it creates.
 

The challenge of alignement is to find a problem which resolution leads AI to not simply do what we, humans, think we want (with our delusions); but what we actually need.

More than the "less harmful" outcome, I think that even in the best case what we actually need is a high number of qualitatively different states;

The more different and synergistic (co-increasing options) they are, the better it is; it's a procedure to augment the number of 'diversity dimensions'. 
For exemple : 1010101010 has less diversity dimensions than 0123456789. 

Because true diversity is not simply about alternating two states linearly, we want a procedure that cares about creating more options and "complexity"/"sophistication". 
But it is often argued to not use such vague words.

Let's clarify these words then :

1) Qualitatively diverse structures/patterns/stimuli computed/circulating in a system

-> In the continuous space of all the possible different states (a sort of semantic space),
the more distance there is between two states, the more qualitative is the difference;

It's like colors in a chromatic circle. It's a latent space. "Black" is more dissimilar to "white" than it is to "grey". A "cat" is closer to a "dog" than to a "chair".

(Individual objects are a category, ie. [my blue dog], but very similar to other objects of a same meta-category; ie. [blue dogs], with which they share a low diversity "index")
 

So what I mean by complexity/sophistication is closely related to this qualitative diversity. You simply combine 1) with :

2) The causal power of a system

-> Number of qualitatively different external changes caused by the system/sub-system

Such [diversity of states in the universe] procedure leads to the maximization of open-endedness, which demands to care about ecosystems, without which things collapse; and qualitative diversity disappears.

-> This is the first layer, before anything else, allowing a general, primal alignment.

 

The goal is maximal synergy, 
In maximal diversity, 
With maximal optionality.

The inverse of alterity (otherness) is ipseity (selfness)
I am talking about the in-between : "synergity"

You can add more and more explicit layers to calibrate the triangulation of our preferences through things such as :

.Human universal

But the more fundamental a layer is, the more it supersedes others, taking priority in decision-making (also more procedural, implicit, avoiding reward tunnels/squiggles)

For example it could be (increasingly explicit, from more to less fundamental) :
Open-ended diversity 
Universal optionality
Ecosystemic synergy
Other-empowerment

There is a high and continuous overlap; 
Upon such basis, upper layers/tools can bring more precise/local well-being :
.What The Longest-Running Study on Happiness Reveals
.Moral foundations theory
.Seven moral rules found all around the world
.Modular Morals: The Genetic Architecture of Morality as Cooperation
.Morality and Evolutionary Biology
+https://plato.stanford.edu/entries/morality-biology/ 
.Profiles of an Ideal Society: The Utopian Visions of Ordinary People

                                          ────────────────────────

The aim is to provide affordance, access to adjacent possibles, expanding the action space and autonomy of a maximum of phenomenons. This optionality approach could potentially solve lots of long-standing thorny problems in consequentialism, like wireheading or the fiendish difficulty of defining happiness/utility; and how even making the tiniest mistake in that definition can be precisely catastrophic. 

For instance : local/myopic/short-timed happiness is regularly detrimental (egocentric interests, addictions, etc.), while local suffering can be beneficial (training, family diner). 

Open-endedness and optionality at the scale of [the whole universe's past and future timeline] implies that keeping the process at the perfect (critical) sweet-spot allows, at the end (widest time scope), the most diversity of states/emergence/open-endedness.

It lets the time for ecosystems to grow and interact, so that they can bloom/mutate the unique/irreducible causal chains that cannot happen with higher (or slower) speed rate. 


And it can be non-linear ie. you sometimes need to restrict the number of options for one to not be frozen by possibilities. To increase options implies an increase of the capacity to chose. The limit is that those choices (ideally) have to not destroy meaningful choices of other agents/humans/wildlife... After which point choosers can opt for themselves however they fit. And always increasing this optionality, preempting lock-ins.

While the more an AI knows you, the more it can assist you, as we've seen the primal layer would not exactly be *empowerment of others*, but a more universal care. 
It might lead to a certain increase of uncertainty, but it's more a feature than a bug;
-> AI will have a respect to your [(irreducible) preference] proportional to its concern for a wider ecosystem full of things.

AIs are here to arbitrate conflicting actions that go against universal open-endedness; local sub-optimality is so complex to define as such, how to be sure that actually when you (human) decide to exile in the mountain it will not in fact be both good for you and for everybody on the long-run? 
-> This uncertainty leads to an implicit degree of freedom which we can also explicitly increase further; to relax utilitarian mistakes through more tolerance for sub-optimality.

                                          ────────────────────────
 

Some of my important references :

.Paretotopia

.Learning Altruistic Behaviours in Reinforcement Learning without External Rewards

.The Hippocratic principle - Vanessa Kosoy

.OMNI: Open-endedness via Models of human Notions of Interestingness 

.4. SKIM THE MANUAL | Intelligent Voluntary Cooperation & Paretotropism

.Report on modeling evidential cooperation in large worlds

.The Capability Approach to Human Welfare 

.Resurrecting Gaia: harnessing the Free Energy Principle to preserve life as we know it

.​From AI for people to AI for the world and the universe​

.Autonomy in Moral and Political Philosophy