Recursion in AI is scary. But let’s talk solutions.

[-]RogerDearnaley1y52

AI that is trained by human teachers, giving it rewards will eventually wirehead, as it becomes smarter and more powerful, and its influence over its master increases. It will, in effect, develop the ability to push its own “reward” button. Thus, its behavior will become misaligned with whatever its developers intended.

This seems like an unproven statement. Most humans are aware of the possibility of wireheading, both the actual wire version and the more practical versions involving psychotropic drugs. The great majority of humans don't choose to do that to themselves. Assuming that AI will act differently seems like an unproven assumption, one which might, for example, be justified for some AI capability levels but not others.

[-]Oleg Trott1y32

Most humans are aware of the possibility of wireheading, both the actual wire version and the more practical versions involving psychotropic drugs.

For humans, there are negative rewards for abusing drugs/alcohol -- hangover the next day, health issues, etc. You could argue that they are taking those into account.

But for an entirely RL-driven AI, wireheading has no anticipated downsides.

[-]RogerDearnaley1y42

In practice, most current AIs are not constructed entirely by RL, partly because it has instabilities like this. For example, LLMs instruction-trained by RLHF uses a KL-divergence loss term to limit how dramatically the RL can alter the base model behavior trained by SGD. So the result deliberately isn't pure RL.

Yes, if you take a not-yet intelligent agent, train it using RL, and give it unrestricted access to a simple positive reinforcement avenue unrelated to the behavior you actually want, it is very likely to "wire-head" by following that simple maximization path instead. So people do their best not to do that when working with RL.

[-]Oleg Trott1y10

I think that regularization in RL is normally used to get more rewards (out-of-sample).

Sure, you can increase it further and do the opposite – subvert the goal of RL (and prevent wireheading).

But wireheading is not an instability, local optimum, or overfitting. It is in fact the optimal policy, if some of your actions let you choose maximum rewards.

Anyway, the quote you are referring to says “as (AI) becomes smarter and more powerful”.

It doesn’t say that every RL algorithm will wirehead (find the optimal policy), but that an ASI-level one will. I have no mathematical proof of this, since these are fuzzy concepts. I edited the original text to make it less controversial.

[-]Nathan Helm-Burger1y30

I think your idea of labelling the source and epistemic status of all training data is good. I've seen the idea presented before. I don't think it's a knockdown solution to model reliability, but I expect it would help a lot.

Sorry that I don't remember exactly where I saw this discussed before. Hopefully it you web search for it you can find it.

[-]Oleg Trott1y10

I think your idea of labelling the source and epistemic status of all training data is good. I've seen the idea presented before.

I'm not finding anything. Do you recall the authors? Presented at a conference? Year perhaps? Specific keywords? (I tried the obvious)

[-]Nathan Helm-Burger1y30

Hmmm. I don't remember. But here's a new example that Zvi just mentioned: https://arxiv.org/abs/2310.15047

[-]Oleg Trott1y10

Thanks! It looks interesting. Although I think it's different from what I was talking about.

[-]Nathan Helm-Burger1y20

I've been continuing to think about this. I eventually remembered that the place I remembered the idea from was pervade conversations with other AI researchers! Sorry to have sent you on a wild goose chase!

It would be interesting to try an experiment with this. Perhaps doing hierarchical clustering on Open Web Text (an early not-too-huge dataset from GPT-2 days). Then get an LLM worth a large context window to review a random subset of each cluster and write a description of it (including an estimate of factual validity). Then, when training, those descriptions would be non-predicted context given to the model. If you do use hierarchical clustering, this will result in a general description and some specific subtype descriptions for every datapoint.

[-]Oleg Trott1y10

Yes, it's simple enough, that I imagine it's likely people came up with it before. But it fixes a flaw in the other idea (which is also simple, although in the previous discussion I was told that it might be novel)

LESSWRONG
LW

LESSWRONG
LW

3

Recursion in AI is scary. But let’s talk solutions.

3

3