Half-baked alignment idea

ozb

I'm trying think through various approaches to AI alignment, and so far this is the one I came up with that I like best. I have not read much of the literature, so please do point me if this has been discussed before.

What if we train an AI agent (ie, reinforcement learning) to survive/thrive in an environment where there are a wide variety of agents with wildly different levels of intelligence? In particular, such that pretty much every agent can safely assume they'll eventually meet an agent much smarter than they are; structure the environment to reward tit-for-tat with a significant bias towards cooperation, eg require agents to "eat" resources that require cooperation to secure and are primarily non-competitive. The idea is to have them learn to respect even beings of lesser intelligence, because they want beings of higher intelligence to respect them; and because in this environment a bunch of lesser intelligences can gang up and defeat one higher-intelligence being. Also, we effectively train each AI to detect and defeat new AIs that seek to disturb this balance. I have not thought this through, curious what you all think

(Cross posted from EA Forum)

I believe this would suffer from distributional shift in two different ways.

First, if the agents are supposed to scale up to the point where they can update their beliefs even after training, then we have a problem once the AI notices it can do pretty well without cooperating with humans in this new environment. If we allow agents to update their beliefs at runtime, then basically any reinforcement-learning-like preconditioning would be pretty much useless, I think. And if the agent can't update its beliefs given new data then that can't be an AGI.

Second, even if you solve the first problem somehow, there is a question of what exactly you mean by "cooperate" and "respect". In the real world the choice is rarely binary between "cooperate" and "defect", and there is often option of "do things that look like you're cooperating while not actually putting much effort into it" (i.e. what most politicians do) or "actively try to deceive everyone looking to make them think you're nice" (you don't have to be actually smarter than everyone else for this, just smarter than everyone who cares enough to use their time to look at you closely). If you're only giving the agents a binary choice, then that's not realistic enough. If you're giving them many choices, but also putting in an "objective" way to check if what they chose is "cooperative enough", then all it takes is for the agents to deceive you, not their smarter AI colleagues. And if there's no objective way to check that, then we're back to describing human values with a utility function.

1ozb4y

How do humans do it? Ultimately, genuine altruism is computationally hard to fake; so it ends up being evolutionarily advantageous to have some measure of the real thing. This is particularly true in environments with high cooperation rewards and low resource competition; eg where carrying capacity is maintained primarily by wild animals, general hard conditions, and disease, rather than overuse of resources. So we put our thumbs on the scale there to make these AIs better than your average human. And we rely on the AIs themselves to keep each other in check.

Ah, true. I just think this wouldn't be enough and that there could be distributional shift if the agents are put into an environment with low cooperation rewards and high resource competition. I'll reply in more detail under your new post, it looks a lot better

1silent-observer4y

So, no "reading" minds, just looking at behaviours? Sorry, I misundertood. Are you suggesting the "look at humans, try to understand what they want and do that" strategy? If so, then how do we make sure that the utility function they learned in training is actually close enough to actual human values? What if the agents learn something on the level "smiling humans = good", which isn't wrong by default, but is wrong if taken to the extreme by a more powerful intelligence in the real world?

6

Half-baked alignment idea

6

6

6

Half-baked alignment idea

6

6