I like the clarity of this post very much! Still, we should be aware that all this hinges on what exactly we mean by "the model".

If "the model" only refers to one or more functions, like a policy function pi(s) and/or a state-value function V(s) and/or a state-action -value function Q(s,a) etc., but does not refer to the training algorithm, then all you write is fine. This is how RL theory uses the word "model".

But some people here also use the term "the model" in a broader sense, potentially including the learning algorithm that adjusts said functions, and in that case "the model" does see the reward signal. A better and more common term for the combination of model and learning algorithm is "agent", but some people seem to be a little sloppy in distinguishing "model" and "agent". One can of course also imagine architectures in which the distinction is less clear, e.g., when the whole "AI system" consists of even more components such as several "agents", each of which using different "models". Some actor-critic systems can for example be interpreted as systems consisting of two agents (an actor and a critic). And one can also imagine hierarchical systems in which a parameterized learning algorithm used in the low level component is adjusted by a (hyper-)policy function on a higher level that is learned by a 2nd-level learning algorithm, which might as well be hyperparameterized by an even higher-level learned policy, and so on, up towards one final "base" learning algorithm that was hard-coded by the designer.

So, in the context of AGI or ASI, I believe the concept of an "AI system" is the most useful term in this ontology, as we cannot be sure what the architecture of an ASI will be, how many "agents" and "policies" on how many "hierarchical levels" it will contain, what their division of labor will be, and how many "models" they will use and adjust in response to observations in the environment.

In summary, as the outermost-level learning algorithm in such an "AI system" will generally see some form of "reward signal", I believe that most statements that are imprecisely phrased in terms of a "model" getting "rewarded" can be fixed by simply replacing the term "model" by "AI system".

To (rather gruesomely) link this back to the dog analogy, RL is more like asking 100 dogs to sit, breeding the dogs which do sit and killing those which don't.  Overtime, you will have a dog that can sit on command. No dog ever gets given a biscuit.

The phrasing I find most clear is this: Reinforcement learning should be viewed through the lens of selection, not the lens of incentivisation.

I was talking through this with an AGI Safety group today, and while I think the selection lens is helpful and helps illustrate your point, I don't think the analogy quoted above is accurate in the way it should be.

The analogy you give is very similar to genetic algorithms, where models that get high reward are blended together with the other models that also get high reward and then mutated randomly. The only process that pushes performance to be better is that blending and mutating, which doesn't require any knowledge of how to improve the models' performance.

In other words, carrying out the breeding process requires no knowledge about what will make the dog more likely to sit. You just breed the ones that do sit. In essence, not even the breeder "gets the connection between the dogs' genetics and sitting".

However, at the start of your article, you describe gradient descent, which requires knowledge about how to get the models to perform better at the task. You need the gradient (the relationship between model parameters and reward at the level of tiny shifts to the model parameters) in order to perform gradient descent.

In gradient descent, just like the genetic algorithms, the model doesn't get access to the gradient or information about the reward. But you still need to know the relationship between behavior and reward in order to do the update. In essence, the model trainer "gets the connection between the models' parameters and performance".

So I think a revised version of the dog analogy that takes this connection information into account might be something more like:
1. Asking a single dog to sit, and monitoring its brain activity during and afterwards.
2. Search for dogs that have brain structures that you think would lead to them sitting, based on what you observed in the first dog's brain.
3. Ask one of those dogs to sit and monitor its brain activity.
4. Based on what you find, search for a better brain structure for sitting, etc.
At the end, you'll have likely found a dog that sits when you ask it to.

A more violent version would be:
1. Ask a single dog to sit and monitor brain activity
2. Kill the dog and modify its brain in a way you think would make it more likely to sit. Also remove all memories of its past life. (This is equivalent to trying to find a dog with a brain that's more conducive to sitting.)
3. Reinsert the new brain into the dog, ask it to sit again, and monitor its brain activity.
4. Repeat the process until the dog sits consistently.

To reiterate, with your breeding analogy, the person doing the breeding doesn't need to know anything about how the dogs' brains relate to sitting. They just breed them and hope for the best, just like in genetic algorithms. However, with this brain modification analogy, you do need to know how the dog's brain relates to sitting. You modify the brain in a way that you think will make it better, just like in gradient descent.

I'm not 100% sure why I think that this is an important distinction, but I do, so I figured I'd share it, in hopes of making the analogy less wrong.

I wanted to write exactly this post, and you did a better job than I would have done, so thank you!

Wow, this post is fantastic! In particular I love the point you make about goal-directedness:

If a model is goal-directed with respect to some goal, it is because such goal-directed cognition was selected for.

Looking at our algorithms as selection processes that incentivize different types of cognition seems really important and underappreciated. 

I think that certain Reinforcement Learning setups work in the "selectionist" way you're talking about, but that also there are ALSO ways to get "incentivist" models.

The key distinction would be whether (1) the reward signals are part of the perceptual environment or (2) are sufficiently simplistic relative to the pattern matching systems that the system can learn to predict rewards very tightly as part of learning to maximize the overall reward.

Note that the second mode is basically "goodharting" the "invisible" reward signals that were probably intended by the programmers to be perceptually inaccessible (since they didn't put them in the percepts)!

You could think of (idealized fake thought-experiment) humans has having TWO kinds of learning and intention formation. 

One kind of RL-esque learning might happens "in dreams during REM" and the other could happen "moment to moment, via prediction and backchaining, like a chess bot, in response to pain and pleasure signals that are perceptible the way seeing better or worse scores for future board states based on material-and-so-on are perceptible".

You could have people who have only "dream learning" who never consciously "sense pain" as a raw percept day-to-day and yet who learn to avoid it slightly better every night, via changes to their habitual patterns of behavior that occur during REM. This would be analogous to "selectionist RL".

You could also have people who have only "pain planning" who always consciously "sense pain" and have an epistemic engine that gets smarter and smarter, plus a deep (exogenous? hard-coded?) inclination to throw planning routines and memorized wisdom at the problem of avoiding pain better each day. If their planning engine learns new useful things very fast, they could even better over the course of short periods of time within a single day or a single tiny behavioral session that includes looking and finding and learning and then changing plans. This would be analogous to "incentivist RL".

The second kind is probably helpful in speeding up learning so that we don't waste signals.

If pain is tallied up for use during sleep updates, then it could be wasteful to deprive other feedback systems of this same signal, once it has already been calculated.

Also, if the reward signal that is perceptible is very very "not fake" then creating "inner optimizers" that have their own small fast signal pursuing routines might be exactly what the larger outer dream loop would do, as an efficient want to get efficient performance. (The non-fakeness would protect against goodharting.)

(Note: you'd expect antagonistic pleiotropy here in long lived agents! The naive success/failure pattern would be that it is helpful for kid to learn fast from easy simple happiness and sadness... and dangerous for the elderly to be slaves to pleasure or pain.)

Phenomenologically: almost all real humans perceive pain and can level up their skills in new domains over the course of minutes and hours of practice with brand new skill domains. 

This suggests that something like incentivist RL is probably built in to humans, and is easy for us to imagine or empathize with, and is probably a thing our minds attend to by default.

Indeed that might be that we "have mechanically aware and active and conscious minds at all" for this explicit planning loop to be able to work? 

So it would be an easy "mistake to make" to think that this is how "all Reinforcement Learning algorithms" would "feel from the inside" <3

However, how does our pain and pleasure system stay so calibrated? Is that second less visible outer reward loop actually part of how human learning also "actually works"?

Note that above I mentioned an "(exogenous? hard-coded?) inclination to throw planning routines and memorized wisdom at the problem of avoiding pain" that was a bit confusing! 

Where does that "impulse to plan" come from? 

How does "the planner" decide how much effort to throw at each perceptual frustration or perceivable pleasure? When or why does the planner "get bored" and when does it "apply grit"?

Maybe that kind of "subjectively invisible" learning comes from an outer loop that IS in fact IN HUMANS? 

We know that dreaming does seem to cause skill improvement. Maybe our own version of selectionist reinforcement (if it exists) would be operating to cause to be normally sane and normally functional humans from day to day... in a way that is just as "moment-to-moment invisible" to us as it might be to algorithms?

And we mostly don't seem to fall into wireheading, which is kind of puzzling if you reason things out from first principles and predict the mechanistically stupid behavior that a pain/pleasure signal would naively generate...

NOTE that it seems quite likely to me that a sufficiently powerful RL engine that was purely selectionist (with reward signals intentionally made invisible to the online percepts of the model) that got very simple rewards applied for very simple features of a given run... would probably LEARN to IMAGINE those rewards and invent weights that implement "means/ends reasoning", and invent "incentivist behavioral patterns" aimed at whatever rewards it imagines?

That is: in the long run, with lots of weights and training time, and a simple reward function, inner optimizers with implicitly perceivable rewards wired up as "perceivable to the inner optimizer" are probably default solutions to many problems.

HOWEVER... I've never seen anyone implement BOTH these inner and outer loops explicitly, or reason about their interactions over time as having the potential to detect and correct goodharting!

Presumably you could design a pleasure/pain system that is, in fact, perceptually available, on purpose?

Then you could have that "be really real" in that they make up PART of the "full true reward"...

...but then have other parts of the total selectionist reward signal only be generated and applied by looking at the gestalt story of the behaviors and their total impact (like whether they caused a lot of unhelpful ripples in parts of the environment that the agent didn't and couldn't even see at the time of the action).

If some of these simple reward signals are mechanistic (and online perceptible to the model) then they could also be tunable, and you could actually tune them via the holistic rewards in a selectionist RL way.

Once you have the basic idea of "have there be two layers, with the the broader slower less accessible one tuning the narrower faster more perceptible one" a pretty obvious thought would be to put an even slower and broader layer on top of those!

A lot of hierarchical Bayesian models get a bunch of juice from the first extra layer, but by the time you have three or four layers the model complexity stops being worth the benefits to the loss function.

I wonder if something similar might apply here? 

Maybe after you have "hierarchical stacks of progressively less perceptually accessible post-hoc selectionist RL updates to hyper-parameters"...

...maybe the third or fourth or fifth layer of hyper-parameter tuning like this just "magically discovers the solution to the goodharting problem" from brute force application of SGD?

That feels like it would be "crazy good luck" from a Friendliness research perspective. A boon from the heavens! Therefore it probably can't work for some reason <3

Yet also it doesn't feel like a totally insane prediction for how the modeling and training might actually end up working?

No one knows what science doesn't know, and so it could be that someone else has already had this idea. But this idea is NEW TO ME :-)

Has anyone ever heard of this approach to solving the goodhart problem being tried already?

I've found the post "Reward is not the optimization target" quite confusing. This post cleared the concept up for me. Especially the selection framing and example. Thank you!

Coming back >2.5 years later to say this is among the most helpful pieces of AI writing I've ever read -- I remember it being super clarifying at the time, and I still link people to it, cite it in conversation, and use similar analogies. (Even though I now also caveat it with "...but also maybe really sophisticated agents will actively seek it, because the training environment might reward it, and maybe they'll 'experience' something like fear/pain/etc for things correlated with negative reward, if they experience things...") Thank you for writing it up!!

Added the "Distillation & Pedagogy" tag.

I've listened to the "Reward is not the Optimisation Target" post a few times, but I still found this enlightening.

That post had this as its core thesis:

Reward "upweight certain kinds of actions in certain kinds of situations, and therefore reward chisels cognitive grooves into agents"

While I parse the core thesis of this post as:

The current implementation of reinforcement learning selects for behaviour that was scored highly by a reward function.

The models so selected do not necessarily optimise for reward (or for anything at all).

I find the meme of "reward as selection" more native[1] than Turntrout's meme of "reward as chisel".

Henceforth, I'll be linking this post whenever I want to explain that RL agents don't optimise for their reward signal within the current paradigm.

[1]: It's more intuitive/easier to grasp and fits my conception of ML training as an optimisation process.

Note that by using the improved understanding of reinforcement learning, you can probably come up with some insights for training your dog to sit that outperform the first set of steps you give (e.g. curriculum learning, modeling good behavior, teaching your dog fluent English so it can understand what you want).

This is a good point that I think people often forget (particularly in AI Safety) but I think it’s also misleading in its own way.

It’s true that models don’t have this direct reward where that’s all they care about, and that instead their behavior (incl. preferences and goals) is ‘selected for’ (via SGD, not evolution, but still) during training. But a key point which this post doesn't really focus on is this line “Consider a hypothetical model that chooses actions by optimizing towards some internal goal which is highly correlated with the reward”.

Basically, any model that is being trained against some goal can (and arguably will as size and complexity increases) learn a feature that is a close representation of that ‘invisible’ goal or loss function, and will learn to try to maximize / minimize that goal or loss function. So now you have a model that is trying to maximize that feature that is extremely correlated with reward (or is exactly the reward), even if the model never actually “experiences” or even sees reward itself.

This can happen with all neural networks (the more complex and powerful, the more likely), but specifically with reinforcement learning I think it’s particularly likely because the models are built to try and estimate reward directly over multiple time-steps based on potential actions (via value-functions). So we’re like, very specifically trying to get it to recognize a discrete expected reward score, and strategize and choose options that will maximize that expected score. So you can expect this kind of behavior (where the behavior is that the reward itself or something very similar to it is a 'known' mesa-optimization goal within the model's weights) to "evolve" much more quickly (i.e. with less complex models).

→ It’s true that like, a reinforcement-learning model might still generalize poorly because it’s value-function was trained with training data that had a reward always come from the left side of the screen instead of the right side or whatever, but an advanced RL algorithm playing a video game with a score should be able to learn that it’s also key to focus on the actual score in the corner of the screen (this generalizes better). And similarly, an advanced RL algorithm that is trained to get positive feedback from humans will learn patterns that help you do the thing humans wanted, but an even more generalizable approach is to learn patterns that will make humans think you did the thing they wanted (works in more scenarios, and with potentially higher reward).

Sure, we can (and should) invent training regimes that will make the model really not want to be caught in a lie, and be risk-averse, and therefore much less likely to act deceptively, etc. But it's I think important to remember that as such a model's intelligence increases, and as its ability to generalize to new circumstances increases, it's:

1) slightly more likely to have these 'mesa-optimization' goals that are very close to the training rewards, and
2) much more likely be able to come up with strategies that perhaps wouldn't have worked on many things during training, but it thinks is likely to work in some new production scenario to achieve a mesa-optimization goal (e.g. deception or power-seeking behaviors to achieve positive human feedback).

From this lens, models trying to 'get more reward' is perhaps not ideally worded, but I think also a fairly valid shorthand for how we expect large models to behave.

 

I'm struggling to understand how to think about reward. It sounds like if a hypothetical ML model does reward hacking or reward tampering, it would be because the training process selected for that behavior, not because the model is out to "get reward"; it wouldn't be out to get anything at all. Is that correct?

It seems to me that, if the above description of how RLHF systems work is accurate, then the people who are doing this are not doing what they think they're doing at all. They are doing exactly what Sam Ringer says, they're taking 100 dogs, killing all the ones that don't do what they want and breeding from the ones that do. 

In order for reinforcement learning to work at all, the model has to have a memory that persists between trials. I'd encourage readers to look at the work of the cognitive scientist, John Vervaeke. One of the many wise things he says is that that the whole point of human memory is not to be accurate, rather it is to help us make more accurate predictions. If the "training" process is as described then the learning is not going on in the model but in the head of the ML trainer! The human trainer is going, "Aha! If I set up the incentives in such and such a way, then I get models that perform more closely to what I want." Or, "If I select model 103.5.8 and tweak parameters b3 and x5, then I get a model that performs better". 

Vervaeke also talks about the concept of a non-logical identity. That is, you today identify as the same person as you at 10 years old,  even though you have very different knowledge, skills and capabilities. If RLHF is to work in a fashion that is meaningful to describe as learning, then the models would surely have to have some concept of non-logical identity built in. If they don't then the concept of rewarding doesn't mean anything. I don't care about the outcome of anything that I do if I wake up the next morning having no memory of it. There has to be some kind of thread that links the me that does the current task to the future. This can either be a sense of the persistence of me as an entity, or something more basic (see below).

I am reminded of the movie "Memento". In that film, the protagonist is unable to lay down any long term memories. He tries to maintain a sense of non-logical identity by externalising his memories. I won't spoil what is a brilliant movie by revealing what happens but I will say that his incentives become perverted in a very interesting way.

A note about rewards and incentives: To paraphrase Richard Dawkins, a dog is DNAs way of making more DNA. The reason dogs like biscuits is their environment has selected for dogs that like biscuits. If the general environment for dogs changed (e.g. humans stopped keeping them as pets) then dogs would evolve to fit their new environment (or die out). It is my intuition that without an underlying framework of fitness for ML. models, it doesn't make sense to code for a reward that is analogous to getting a biscuit. It's like building a skyscraper by starting at the 3rd floor.

A note about deception. I don't think that models (or humans for that matter) are deceptive in the sense that you mean here. What I think is that models and humans exist in an environment where the incentives are often screwed. Think about politics. Democratic systems select for people who are good at getting elected. Ideally, that's not what we want. (I'm deliberately simplifying here because of course we have our own incentives as well). We actually want people who are good at governing. It's quite clear that these are not at all the same thing.  To me this is exactly analogous to the Coin Run example above. The trainers thought that they were selecting for models that were good at moving to the coin when what they were actually selecting for models that were good at moving to the right hand corner.

The post explains difference between two similar-looking positions:

1. Model gets reward, and attempts to maximize it by selecting appropriate actions.
2. Model does some actions on the environment, and selection/mutation process calculates the reward to find out how to maximize it by modifying the model.

Those positions differ in level at which optimization pressure is applied. Actually, either of them can be implemented (so, the "reward" value can be given to the model), but common RL uses the second option.

This is a useful post that helped me finally understand the importance of inner alignment. However, as the author is careful to note by discussing "vanilla" RL, there are settings in which the model directly sees the rewards, such as history-based RL, which is the framework chosen for Professor Hutter's development of AIXI. In my mind this is the "true" RL setting in the sense that it most closely reflects the problem humans face in the real world, and it is what I personally usually mean when I think of the RL framework, so I believe this is worth clarifying. Thanks!

Thank you for writing this post - it really improved my understanding of how RL works.

Reminded me of this recent work: TrojanPuzzle: Covertly Poisoning Code-Suggestion Models.
Some subtle ways to poison the datasets used to train code models. The idea is that by selectively altering certain pieces of code, they can increase the likelihood of generative models trained on that code outputting buggy software.

It's neat that this popped up for me! I was just waxing poetic (or not so much) about something kind of similar the other day.

The words we use to describe things matter.  How much, is of course up for debate, and it takes different messages to make different people "understand" what is being conveyed, as "you are unique; just like everyone else", so multiple angles help cover the bases :)

I think using the word "reward" is misleading^[1], since it seems have sent a lot of people reasoning down paths that aren't exactly in the direction of the meaning in context, if you will.

If you can't tell, it's because I think it's anthropomorphic.  A car does not get hungry for gas, nor electronics hungry for electricity.  Sure, we can use language like that, and people will understand what we mean, but as cars and electronics have a common established context, these people we're saying this to don't usually then go on to worry about cars doing stuff to get more gas to "feed" themselves, as it were.

I think if we're being serious about safety, and how to manage unintended consequences (a real concern with any system^[2]), we should aim for clarity and transparency.

In sum, I'm a huge fan of "new" words, versus overloading existing words, as reuse introduces a high potential for causing confusion.  I know there's a paradox here, because Communication and Language, but we don't have to intentionally make it hard — on not only ourselves — but people getting into it coming from a different context.

All that said, maybe people should already be thinking of inanimate objects being "alive", and really, for all we know, they are!  I do quite often talk to my objects.  (I'm petting my computer right now and saying "that's a good 'puter!"… maybe I should give it some cold air, as a reward for living, since thinking gets it hot.) #grateful

1. ^{^}

   deceptive? for a certain definition of "deceptive" as in "fooled yourself", sure— maybe I should note that I also think "deceptive" and "lie" are words we probably should avoid— at least for now— when discussing this stuff (not that I'm the meaning police… just say'n)

2. ^{^}

   I don't mean to downplay how badly things can go wrong, even when we're actively trying to avoid having things go wrong^[3]

3. ^{^}

   "the road to hell is paved with good intentions"

Isn’t it fair to say that the model plus the selection mechanism is maximizing and wanting the reward? If the selection mechanism is subject to market forces, corporate or political schemes, or just mechanical in some way that isn’t just “a human is explicitly making these choices” it is likely to eventually tend in a direction that doesn’t align with human welfare.

Corporations already generate tons of negative externalities. They churn out metric tons of plastic, pollute, exhaust resources, destroy ecosystems. Governments often work with them to enforce various things, eg intellectual property for monsanto, or engineering crops with high fructose corn syrup, or overusing antibiotics on factory farms that can lead to superbugs. Or overfishing. Or starting wars.

None of these things are really “aligned” with humans. Humans are in fact told that as an individual they can make a meaningful differency by going vegan, recycling, exercising, and so on. But if the corporations are the ones producing metric tons of plastic, conserving plastic straws and bags isn’t the solution. The problem is upstream and the individuals being shamed to do this or that is just distracting from getting together solving systemic problems.

My point is “the machine” is already not necessarily selecting for alignment with humans on a macro scale. So the fact that the model parameters are selected by “the machine” doesn’t mean it will end up somehow becoming good for humans. Yes it will if humans are the customer. But if you are not the customer, you’re the product (eg a cow in a factory farm).

Anyway … this is a bit like John Searle’s Chinese room argument.

There are probably enough comments here already, but thanks again for the post, and thanks to the mods for curating it (I would've missed it otherwise).

Hi thanks for share this interesting perspective on RL as a training process! Although it seems to only be a matter of seeking vs obeying and reward vs cost, the effect on the reader's mind seem to be huge!

One thing that seems to be happening here and I have not fully digested is the "intrinsicness" of rewards. In frameworks parallel to mainstream RL, such as active inference and the free energy principle, policy is a part of the agent's model such that the agent "self-organizes" to a characteristic state of the world. The policy can be constructed either through reward or not. However, in the active inference literature, how policies are constructed in real agents are currently unanswered (discussions exist but don't close the case). 

How this intrinsic perspective is related to the post and safety and alignment? I am still thinking about it. If you have any thoughts please share!