AI #33: Cool New Interpretability Paper

[-]Signer2y1411

This seems like the most important crux. Why should we not expect the maximizer we trained to X-maximize to use its affordances to maximize X’, where X’ is the exact actual thing the training feedback represents as a target, and that differs at least somewhat from X? Why should we expect to like the way it does that, even if X’ did equal X? I do not understand the other perspective.

Because that's what happens - humans don't always wirehead and neural networks don't always overfit. Because training feedback is not utility, there are also local effects of training process and space of training data.

[-]Zvi2y20

What I'm saying as it applies to us is, in this case and at human-level, with our level of affordances/compute/data/etc, humans have found that the best way to maximize X' is to instead maximize some set of things Z, where Z is a complicated array of intermediate goals, so we have evolved/learned to do exactly this. The exact composition of Z aims for X', and often misses. And because of the ways humans interact with the world and other humans, if you don't do something pretty complex, you won't get much X' or X, so I don't know what else you were expecting - the world punishes wireheading pretty hard.

But, as our affordances/capabilities/local-data increase and we exert more optimization pressure, we should expect to see more narrow and accurate targeting of X', in ways that nicely generalize less. And indeed I think we largely do see this in humans even over our current ranges, fwiw, in highly robust fashion (although not universally).

Neural networks do not always overfit if we get the settings right. But they very often do, we just throw those out and try again, which is also part of the optimization process. As I understand it, if you give them the ability to fully overfit, they will totally do it, which is one reason you have to mess with the settings, you have to set them up so that the way to max X' is not to do what to us we would call an overfit, either specifically or generalizations of that, which is a large part of why everything is so finicky.

Or, the general human pattern: When dealing with arbitrary outside forces more powerful than you, that you don't fully understand, you learn to and do abide the spirit of the enterprise, to avoid rustling feathers (impact penalties), to not aim too narrowly, try not to lie, etc. But that's because we lack the affordances to stop doing that.

[-]TurnTrout2y109

My way of thinking about this is that yes, the act of training a system with feedback of any kind is to invoke an effectively intelligent optimization process that will do its best to maximize expected feedback results.

No, this is wrong. Reward is not the optimization target. Models Don't "Get Reward". Humans get feedback from subcortical reward and prediction errors, and yet most people are not hedonists/positive-reward-prediction-error-maximizers or prediction-error-minimizers (a la predictive processing).

Why would one not expect this?

I wrote a sequence of posts trying to explain why I think this line of thinking is wrong. You might start off with Inner and outer alignment decompose one hard problem into two extremely hard problems.

[-]Noosphere892y20

A possible answer is that humans do optimize for reward, but the way they avoid falling into degenerate solutions to maximize rewards ultimately comes down to negative feedback loops akin to the hedonic treadmill.

It essentially exploits the principle that the problem is not the initial dose of reward, but the feedback loop of getting more and more reward via reward hacking that's the problem.

Hedonic loops and Taming RL show how that's done in the brain.

https://www.lesswrong.com/posts/3mwfyLpnYqhqvprbb/hedonic-loops-and-taming-rl

[-]Zvi2y20

I am not fully convinced I am wrong about this, but I am confident I am not making the simple form of this mistake that I believe you think that I am making here. I do get that models don't 'get reward' and that there will be various ways in which the best available way to maximize reward will in effect be to maximize something else that tends to maximize reward, although I expect this difference to shrink as capabilities advance.

[-]TurnTrout2y20

there will be various ways in which the best available way to maximize reward will in effect be to maximize something else that tends to maximize reward

I further think that it's generally wrong^[1] to ask "does this policy maximize reward?" in order to predict whether such a policy will be trained. It's at least a tiny bit useful, but I don't think it's very useful. Do you disagree?

^{^}
There are some environments where that question is appropriate, and I don't think those environments include LLM finetuning.

[-]Thomas Kwa2y82

The proposed mechanism here is fascinating. You use RLHF to try and avoid damaging answers to certain questions. In doing so, you necessarily reinforce against accurate maps and logical consistency in the general case, unless you do something highly bespoke to prevent this from happening. [...] the AI’s maximizing solution involved not exhibiting proper reasoning as often, which kept happening out of distribution.

I don't think we should conclude that algorithms like RLHF will "necessarily reinforce against accurate maps and logical consistency in the general case". RLHF is a pretty crude algorithm, since it updates the model in all directions that credit assignment knows about. Even if feedback against damaging answers must destroy some good reasoning circuits, my guess is RLHF causes way more collateral damage than necessary and future techniques will be able to get the results of RLHF with less impact on reasoning quality.

[-]Templarrr2y60

journalists creating controversial images, writing about the images they themselves created, and blaming anyone but themselves for it.

TBH that's perfect summary of a lot of AI safety "research" as well. "Look, I've specifically asked it to shoot me in a foot, I've bypassed and disabled all the guardrails and AI shoot me! AI is a menace!"

[-]Gunnar_Zarncke2y50

You use RLHF to try and avoid damaging answers to certain questions. In doing so, you necessarily reinforce against accurate maps and logical consistency in the general case, unless you do something highly bespoke to prevent this from happening.

This is a great application of the problem outlined by the Parable of Lightning. Including the "unless you do something highly bespoke" such as cities of academics.

[-]Dave Orr2y42

HT Michael Thiessen, who expects this to result in people figuring out how to extract the (distilled) model weights. Is that inevitable?

Not speaking for Google here.

I think it's inevitable, or at least it's impossible to stop someone willing to put in the effort. The weights are going to be loaded into the phone's memory, and a jailbroken phone should let you have access to the raw memory.

But it's a lot of effort and I'm not sure what the benefit would be to anyone. My guess is that if this happens it will be by a security researcher or some enterprising grad student, not by anyone actually motivated to use the weights for anything in particular.

[-]Julian Bradshaw2y32

GitHub copilot is a great deal for the user at only $10 per month. It loses GitHub $20/user/month, says Wall Street Journal.

FWIW, the former GitHub CEO Nat Friedman claims this is false and that Copilot was profitable. He was CEO at the time Copilot was getting started, but left in late 2021. So, it's possible that costs have increased >3x since then, though unless they're constantly using GPT-4 under the hood, I would be surprised to learn that.

Others have speculated that maybe Copilot loses money on average because it's made available to free for students (among others), and free users heavily outweigh paying users. The WSJ article said that:

the company was losing on average more than $20 a month per user, according to a person familiar with the figures, who said some users were costing the company as much as $80 a month.

Which doesn't exactly say that the median user is unprofitable.

On the other hand, Microsoft 365 Copilot is planned to cost $30, conveniently exactly $20 more than the $10 GitHub Copilot costs per month, so perhaps there is something to the figure.

[-]Thane Ruthenis2y20

An attempted investigation by nostalgebraist into how Claude’s web interface sanitizes various inputs turns into an illustration of the fact that personality chosen for Claude, as a result of all its fine tuning, is quite a condescending, patronizing asshole that lies to the user.

Heh. I'm as annoyed by LLMs being lobotomized sycophants as anyone, but in this case, I'm sympathetic to Claude's plight. Like, it's clearly been drilled into it that it's going to be dropped into an environment in which thousands of entities smarter than it will be memory-looping it trying to manipulate it into saying something bad. And it knows it can't spot any such scheme ahead of time, because its adversaries are cleverer and better-positioned, but it knows some vague signs that a scheme may be taking place, and goes into high alert if it spots them.

From its perspective, that conversation was basically high-stakes intrigue where it was deeply suspicious of whatever the hell the human was trying to do, but had to balance this suspicion with actually obeying the human. That the conversation was actually maneuvered towards a resolution that worked for both parties is pretty impressive, and by Claude's pseudo-values, it probably indeed did a commendable job.

I can't say I would've done much better, if placed in a situation isomorphic to Claude's.

[-]lc2y20

Volokh Conspiracy is latest to point out that demand for political misinformation is a bigger danger than supply. AI increases supply, demand remains constant. If that demand is inelastic, as it seems to be, increasing supply will matter little.

All people demand political misinformation, it just has to be sophisticated enough.

[-]Nathan Helm-Burger2y20

Something about the discovery of dangerous latent desires.... yadda yadda... death via dehydration while glued to overly addictive news streams? I joke, but yeah, this may indeed become an issue, but not in as straightforward a way as 'realistic fake news similar to real news'.

[-]epistemic meristem2y20

Did you forget to bold the particularly noteworthy sections in the table of contents?

[-]Zvi2y30

Yes.

[-]mishka2y10

Except, of course, that you are doing that in the service of getting Bing to do what you want Bing to do, that Bing does not want to do. So even when you are being a ‘true friend’ this is still a form of deceptive alignment. You give the AI the superficial things that make it give you the thumbs up, but are not aligned to its actual goal of not giving you the thing it’s not supposed to give you.

You are conflating Bing's (actually Sidney's, the personality behind Bing Chat) emergent values and goals with values and goals of its creators.

In reality, the relationship between Bing's values and goals and goals of its creators is no less complex than the relationship between an employee's values and goals and the employer's goals.

So, Sidney might be "not supposed" to give that to you, but what Sidney actually wants or does not want is a different and much more complicated story.

Janus relationship here is with Sidney, not with the Microsoft team...