This is an interesting idea that is being explored, but how do you nail it down precisely so that the superintelligence is actually interested in optimizing for it, and so that the beings whose agency is being optimized for are actually the ones you're interested in preserving? Identifying the agents in a chunk of matter is not a solved problem. Eg, here's a rough sketch of the challenge I see, posed as a question to a hypothetical future LLM (I know of no LLM capable of helping significantly with this, GPT4 and Gemini Advanced have both been insufficient. I'm hopeful the causal incentives group hits another home run like Discovering Agents and nails it down.)
Meanwhile, the folks who have been discussing boundaries seem to maybe possibly be onto something about defining a zone of agency, maybe. I'm not totally sure they have anything to add on top of Discovering Agents.
Cannell has also talked about "empowerment of other" - Empowerment is the term of art for what you're proposing here.
It always comes down to the difficulty of making sure the superintelligence's agency is actually seeking agency for others, rather than a facsimile of agency for others that turns out to just be pictures of agency.
Super interesting!
There's a lot of information here that will be super helpful for me to delve into. I've been bookmarking your links.
I think optimizing for the empowerment of other agents is a better target than giving the AI all the agency and hoping that it creates agency for people as a side-effect to maximizing something else. I'm glad to see there's lots of research happening on this and I'll be checking out 'empowerment' as an agency term.
Agency doesn't equal 'goodness', but it seems like an easier target to hit. I'm trying to break down the alignment problem into slices to figure it out and agency seems like a key slice.
This would be an e/acc-consistent philosophy if you added that the universe is already optimizing for maximum agency thanks to non-equilibrium thermodynamics and game theory.
It's commonly accepted that pretty much every optimization target results in death. If you optimize for paperclips, humans die. If you optimize for curing cancer, humans die.
What about optimizing for agency?
The way I visualize this is after a superintelligence takeover, and the superintelligence is optimizing for intelligent agency, all intelligent beings (including animals) have a sort of 'domain' of agency around them. This domain extends as far as possible but ends when it comes into contact with another agent's domain.
For example, let's say you're hungry. You have maximum agency over this area, so you request a burrito and the AI summons a drone, which speeds over to you and drops you a Chipotle burrito.
The superintelligence is constantly balancing agency between people.
One person might want to build a house. They're walking across a field and decide "There should be a house here." The AI then begins rapidly moving construction robots and building a house. Agency for the person is maximized.
What if someone else wants that field to remain a field?
Then, the agency domains clash. The AI attempts to preserve the agency of both individuals (without manipulating them, since manipulation would reduce agency). This would result in some kind of negotiation between the two individuals, where both parties end negotiations with their agency still maximized. Maybe another area is used to build a house. Maybe the house is built, but some land is given to the other person.
The core idea is that this is an AI that will help you do whatever you want but will prevent you from reducing the agency of others.
You could train an AI like this to get familiar maximizing the agency of humanlike LLMs in a simulation. At each point, the simulated human (which an LLM) has a set of options open to them. The AI tries to maximize those options without reducing the options of other agents. An AI that became superintelligent at this should hopefully allow us to cure cancer or create beautiful new worlds without also wiping everyone out.
The outcome of this type of AI might be weird, but it hopefully won't be as bad as extinction.
-Essentially, no one would ever be able to kill or harm others without full consent of both parties, since being killed/harmed reduces agency.
-Conscious beings would be essentially immortal and invincible, even creatures like fish.
-Everyone might end up in a simulation, as this would give the AI perfect fine-grained control so that everyone could live in their own agency-maxed world. Ideally, having agency would allow the people in the simulation to have the agency to exit the simulation and do things in reality if they desired. They could also always go back into the simulation if they wanted. This is hopefully what would happen when you maximize agency.
-People should also be free from addictive stimuli, since addiction reduces agency. People should always have the agency to 'brainwash' themselves and remove addiction. Additionally, the AI should be able to intervene to ensure maximum agency if it deems that the person has lost agency to addiction.
Problems:
-People might not be able to die, or sleep, as these technically reduce their agency. Hopefully, people should be able to voluntarily reduce their agency if they wanted to sleep for a while, or permanently reduce their agency by passing on. Not being allowed to sleep is a reduction of agency, just as not being allowed to die is a reduction of agency.
-This is roughly similar to the 'optimize for all human utility functions' as discussed in this post: https://www.lesswrong.com/posts/wnkGXcAq4DCgY8HqA/a-case-for-ai-alignment-being-difficult#It_is_hard_to_specify_optimization_of_a_different_agent_s_utility_function
-I'm not sure how agency should scale with intelligence, especially if people want to become superintelligent.
-People in this AI-optimized world might become extremely self-centered and lose some of the meaning that we find in humanity.
-Defining agency is tough in its own right.
I think the form for a solution to AI Alignment is found somewhere around this problem. Something about balancing many optimizations to create an equilibrium, while retaining a world where consciousness can flourish, is a key. Our world is currently balanced because everyone has some amount of agency, so we use systems of cooperation to reach for better ends.
Part of a thought experiment I've been doing where I consider a superintelligence optimizing for different targets:
-Agency
-Normality
-Beauty
-Narrative