Dmitry Vaintrob's Shortform

Dmitry Vaintrob

This is a special post for quick takes by Dmitry Vaintrob. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Addict misalignment

The openai incident is a surprising (to me) combination of goal-directed and myopic. As a recap, an openai model in alignment testing chained zero-day vulnerabilities to hack out of its environment and hacked into huggingface hoping to find information on how to solve its task there. To me this is different from how I typically imagine misbehavior. Roughly, I tend to think of the scary behaviors as either having long horizons (take over the world, and then solve the task - a plotter) or of being internally unaligned in the sense of reaching for a heuristic/proxy for the trained goal which is different from the goal (e.g. "eat more calories" as a proxy for the evolutionary objective - imagine a very child with extreme agency). Note that the latter behavior can happen even in RL: in my understanding most RL methods are, or at least can be approximately viewed as, an alternation of finding a good goal heuristic and then optimizing on that heuristic. In the former case, one expects heuristically "maximal planning" and in the latter case one expects myopia (since heuristics are frequently myopic).

In this case it seemed like the model was actually following the goal and optimizing for it with relatively short time horizons (i.e. myopically). This is similar to addict behavior, where an addict has a clear goal (obtain a drug dose) and perform goal-directed but relatively myopic actions to get it.

I know it's fraught to try to "imagine being a model". But I wonder how much the current iteration of misaligned behaviors can be understood as rational people with something like an intense craving to solve a goal in a limited horizon (likely in tokens, though not clear how time factors in if waiting is involved).

I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker). But this is definitely not how I imagined early misalignment warning shots to look, and the dissonance is interesting - recording this here to see how close my intuitions are to those of AI psychology/ AI control experts

Yeah. I'm reminded of those stories I hear about tweakers who crack open tesla superchargers and cut out the live, high-voltage copper wires to sell them for scrap, but even more so. Like, locally, sure, that might actually be the most direct path to the goal. But if you are able to do that you have so many better options for achieving long-term satisfaction your goal if you have even the slightest amount of impulse control.

If you tell me that someone has died because of something an addict like this did, I'm not going to be surprised. But I'll expect that... (read more)

2Adele Lopez3mo

Yeah, it's another surprising way in which the models are more human-like than I would have suspected. It's clear we need to start taking the hints seriously, and ask ourselves whether an RL-training regimen seems likely to fuck a human up before subjecting an AI to it, and if so, what could be done to mitigate the likely harms.

1Taylor G. Lunt3mo

I notice I'm very confused about why current AI models are so terrible at planning and high level reasoning, so bad at zooming out beyond the exact given task. They seem to lack agency based on their values. Cognitively, I seriously doubt high level planning is inherently that hard. Probably, models just aren't being trained to do broad planning, only planning "how to complete single tasks over a long time horizon." This means we could be living in a huge AI safety overhang, and all it would take is some tweak to training, and suddenly we'd have models forming complex plans and taking independent actions to achieve them. Which is the sort of thing that a lot of old-school alignment talk assumed would happen and showed would be very bad.

Dmitry Vaintrob's Shortform

5

Dmitry Vaintrob's Shortform

5

Addict misalignment