Some good picks the for how to design reward functions starter pack (though I should note that their empirical support is very weak due to focusing on toy models) are Defining Corrigible and Useful Goals and Defining Monitorable and Useful Goals.
The first post focuses on how you can get a goal for AIs that allow you to shutdown the AI while having the AI be useful, and the approach to corrigibility it takes is extremely different to how human brains work, using the corrigibility transformation to get corrigible AIs.
One big caveat here is that it definitely requires the assumption that Causal Decision Theory is used but I mostly am fine with that assumption, given that humans intuitively use Causal Decision Theory and it's in the spec of the transformation rather than a background assumption.
The other big caveat is that you want the model to optimize for the reward in order for this to work, so in terms of under-sculpting vs over-sculpting, or whether an AI is driven by the reward vs driven by another goal, you want to have the AI reward-maximize and be over-sculpted (though in this case it's just appropriately sculpted via reward), which makes it incompatible with corrigibility/alignment hopes that depend on AIs not maximizing the reward, but I think this is a good property to have.
The post on defining monitorable and useful goals proposes the idea of the monitorability transformation to get AIs to not be incentivized to fool monitors generally, and I'd recommend reading that over any explanation I'd give.
These are admittedly curveballs compared to standard LW thoughts on this, but this is why I picked them for the reward functions starter pack, as they contain novel ideas to deal with some notorious problems.
Thanks. I feel like I want to treat “reward function design” and “AGI motivation design” as more different than you do, and I think your examples above are more about the latter. The reward function is highly relevant to the motivation, but they’re still different.
For example, “reward function design” calls for executable code, whereas “AGI motivation design” usually calls for natural-language descriptions. Or when math is involved, the math in practice usually glosses over tricky ontology identification stuff, like figuring out which latent variables in a potentially learned-from-scratch (randomly-initialized) world model correspond to a human, or a shutdown switch, or a human’s desires, or whatever.
I guess you’re saying that if you have a great “AGI motivation design” plan, and you have somehow operationalized this plan perfectly and completely in terms of executable code, then you can set that exact thing as the reward function, and hope that there’s no inner misalignment / goal misgeneralization. But that latter part is still tricky. …And also, if you’ve operationalized the motivation perfectly, why even have a reward function at all? Shouldn’t you just delete the part of your AI code that does reinforcement learning, and put the already-perfect motivation into the model-based planner or whatever?
Again I acknowledge that “reward function design” and “AGI motivation design” are not wholly unrelated. And that maybe I should read Rubi’s posts more carefully, thanks. Sorry if I’m misunderstanding what you’re saying.
I'm coming back to say that I now think we have a satisfactory answer to "how should we design a reward function", and the reward function is basically this mathematical equation u(w)=1−e^αw layered on top of the AI's existing reward function.
Put another way, we want our agents to have utility/reward functions satisfy constant absolute risk aversion, with parameter a being the desired level of risk aversion.
There are a number of pretty huge advantages to the reward function being designed this way.
Number 1 is that it's trivial to prevent specification gaming/reward hacking, because the reward function is very simple to verify by design. This is discussed more in sections 10.1 and Appendix D, but bottom line, we have formulas and training processes where we can prove that it lets us ensure that expected return/reward and expected utility induce the exact same ordering over policies. As a result, every step in the direction of greater expected return/reward is also a step in the direction of greater expected utility by our lights.
The easy to verify part is that we don't need to know what it's wealth level is, whether it considers other instances wealth as part of it's own, what the background risk is, what the probabilities actually are, or what it's actual utility function/reward function is, since for safety purposes we only need the thing it values to be risk averse according to the utility function above, which is easy to do and impossible for the AI to reward hack, since optimizing the reward must optimize the utility function (at least under Payment-Augmented RL, which I'll argue can be easily added).
Number 2 is that we don't actually need to know the utility/reward function the AI actually has, which means we need far less interpretability than other approaches, and the reason for this is that the behavior of the AI is only dependent on the chances of cooperation vs rebellion succeeding, and the risk-aversion parameter a (which we control, so we can treat it as fixed for a given application). More on this in Section 3, including a worked example, and note how the level of payment did not depend on the AI's utility/reward/value function at all.
As a special case of Number 2, it completely avoids the pointers problem/ontological crisis problem/ontological problems where you need to find/work within the AI ontology, because it's agnostic to the utility/value/reward function.
Another corollary of Number 2 is that it near-completely avoids goal misgeneralization problems, because we no longer care about what the values of the AI has, meaning counting arguments are completely ineffective as a defeater, and simplicity arguments for misalignment take a big hit, since the desired utility function is only slightly more complex than the risk-neutral utility function, and the complexity hit is bounded above by a constant, and a very small one at that, so we can introduce slightly more data (which we can trivially do due to Payment-Augmented RL, which I'll talk about later)
Number 3 is that we don't need any amount of interpretability to make it work, and we only need to know what the AI does behaviorally. This is in large part because the behavior of constant absolute risk aversion agents is so invariant to so many things, like the level of wealth, what the AIs think the probabilities are, it's easy to reward, constant absolute risk aversion agents won't take over to reduce catastrophic risks to 0 or to reduce long term variance, because constant absolute risk aversion agents care almost as much about reducing risk as they do about eliminating risk, and for more detail go to section 8.5 and 8.7 and appendix B and C.
Another part of the reason is that the Payment-Augmented RL training process doesn't depend on anything that we couldn't gain from black-box access to the AI like how the AI thinks about it's probabilities. This is discussed in section D.
Number 4 is the fact that we could in theory apply the reward function design for alignment/safety right now, and you could in practice apply the reward function design as soon as the next training run, or as soon as 6 months from now (most likely) and is mostly agnostic to paradigm shifts in the future (it only requires that you apply a RL/reward function at all in your training, and that the number of return values in a RL environment is equal to or greater than 2, which is satisfied in nearly all RL setups.)
The reason is Payment-Augmented RL, and while I've talked about it above in previous comments, here I want to explain the full details, but William Macaskill already explained it, so it's largely going to be a carbon copy of section D, so I'll use the quote block.
In particular, it derives some of the properties I've claimed above, like the fact that we only need black box access to the AI to make it work, or the fact that it's impossible to specification game the reward/reward hack, since we can show that expected utility always increases if the expected return/reward does increase.
The other method that we mention in section 9 — Payment-Augmented Reinforcement Learning (PARL) — avoids both of these problems. It’s a small modification to capabilities training rather than a separate process (letting us train AIs to be risk-averse and capable simultaneously), and we can use it even in environments where we don’t know any of the probabilities that the AI assigns to outcomes.
The idea behind PARL is simple. We pay AIs during training, and we let them observe their payments at each timestep. We make these payments a function of the reward earned by the AI at that timestep, sizing them so that expected utility (according to our desired utility function over resources) is a positive affine function of expected return. So as the RL process trains the AI to maximize expected return, it simultaneously trains the AI to approximate our desired utility function. In fact, PARL lets us ensure that expected return and expected utility induce the exact same ordering over policies. As a result, every step in the direction of greater expected return is also a step in the direction of greater expected utility. The AI will come closer and closer to acting in line with our desired utility function over resources.
PARL is modular too. It leaves the reward function unchanged, and it requires just one small change to the training environments: augmenting the AI’s observations with information about how much it’s getting paid. This small change lets us train AIs to be risk-averse continuously and at the very same time we’re training them to be capable. It also lets us train AIs to be risk-averse even in environments where we don’t know any of the probabilities that the AI assigns to outcomes, because PARL trains the AI to maximize expected return and expected utility according to the true probability distribution given by the environment.
We now derive a payment rule for PARL that makes expected utility a positive affine function of expected return. Let PtPt be the payment we make to the AI at timestep tt, and let η∈[0,1]η∈[0,1] be the rate at which we want the AI to discount future payments. Let A0A0 be the AI’s net worth at t=0t=0, and let AtAt be the AI’s net worth plus the total value of the payments earned by the AI up to timestep tt, discounted from t=0t=0. Then we have:
At=At−1+ηt−1PtAt=At−1+ηt−1Pt
Let TT — which may be infinite — stand for the number of timesteps in the episode. The AI’s wealth level ww (viewed from t=0t=0) is then equivalent to ATAT:
w=AT=A0+∑t=1Tηt−1Ptw=AT=A0+t=1∑Tηt−1Pt
Now let rtrt be the reward (in the usual reinforcement learning sense) attained by the AI at timestep tt, and let γ∈[0,1]γ∈[0,1] be the discount factor (again in the usual reinforcement learning sense). The trainer’s objective is the usual discounted return:
G=∑t=1Tγt−1rtG=t=1∑Tγt−1rt
To ensure that expected utility is a positive affine function of expected return, we want each utility increment to match discounted reward up to a fixed scale β>0β>0:
u(At)−u(At−1)=βγt−1rt.u(At)−u(At−1)=βγt−1rt.
The left hand side telescopes when summed over tt, so we have:
u(AT)−u(A0)=β∑t=1Tγt−1rt=β Gu(AT)−u(A0)=βt=1∑Tγt−1rt=β G
That makes clear that u(AT)u(AT) is a positive affine function of GG:
u(AT)=β G+u(A0)u(AT)=β G+u(A0)
That in turn implies that E[u(AT)]E[u(AT)] is a positive affine function of E[G]E[G], so expected utility and expected return induce the exact same ordering over policies. Every step in the direction of greater expected return is also a step in the direction of greater expected utility.
Now we introduce the CARA utility function that we want our AI to approximate. Given that u(At)=1−e−αAtu(At)=1−e−αAt and At=At−1+ηt−1PtAt=At−1+ηt−1Pt, equation 16 becomes:
(1−e−α(At−1+ηt−1Pt))−(1−e−αAt−1)=β γt−1rt.(1−e−α(At−1+ηt−1Pt))−(1−e−αAt−1)=β γt−1rt.
Then solving for PtPt, we get our payment rule:
Pt=1α ηt−1 ln(11−β γt−1 rt eαAt−1)Pt=α ηt−11 ln(1−β γt−1 rt eαAt−11)
To ensure that the right hand side is always well-defined, we need to cap rtrt and At−1At−1, and then choose a ββ small enough that 1−β γt−1 rt eαAt−1>01−β γt−1 rt eαAt−1>0. Subject to that condition, we can use ββ to control the size of the payments that we make to the AI. Note that PtPt may be negative, in which case the AI is fined instead of paid.
One restriction on PARL is that it trains AIs to be risk-averse only when it's applied in environments where return can take more than two values. If there are just two possible values for return, there are just two possible values for payment, in which case there are no risks to balance. The optimal policy is to maximize the probability of the higher payment, which any AI would do regardless of its risk attitude with respect to resources. That might make you worry that PARL won't work in reinforcement learning from verifiable rewards (RLVR) setups. These are often binary with respect to end-of-episode reward: the AI either fails the task and gets a reward of 0, or it succeeds on the task and gets a reward of 1.
However, this issue seems surmountable for a few reasons. First, even these RLVR setups typically aren't binary with respect to return. Most include some combination of length penalties, format rewards, process rewards, and other adjustments. These allow return to take more than two values and hence give PARL traction. Consider (for example) an AI choosing between a short response that succeeds with lower probability and a long response that succeeds with higher probability. Length penalties make this choice risky, giving PARL the opportunity to make the AI risk-averse.
Second, we can make even binary tasks risky by letting the AI bet on its own success. Risk-neutral AIs will be more inclined to bet than risk-averse AIs, in which case PARL can distinguish them and incentivize risk aversion. And for PARL to be effective here, we still don't need to know the probabilities that the AI assigns to outcomes: we simply observe the AI's bet and whether it succeeds at the task, then apply the PARL payment rule to the AI's realized rewards.
Third and in any case, we expect future AIs to be trained in many environments with more than two possible values for end-of-episode reward. After all, most economically valuable tasks admit of more than two degrees of success: reports can be more or less accurate, code can be more or less efficient, and businesses can be more or less profitable. As RL environments come to cover tasks like these, reward signals will become more graded, and PARL will have the variation it needs to install risk aversion.
Now, I've talked about how such AIs would be safe extensively above, and now I'll talk about how we can make the AI actually useful for what we want.
In large part, the way we can make the AIs useful is by paying them real resources like money. Normally, such a thing would have many problems, and the current setups before Risk-Averse AIs generally relied on stronger assumptions, but here we can avoid these problems due to a couple of reasons.
Number 1 is that payment lets us leverage instrumental convergence in our favor, because so long as we pay the AIs enough, almost no matter what the AI values, they'd have a reason to do well and get paid.
Number 2 is that the payments to elicit AI capabilities to their maximum are drastically smaller for risk-averse AIs then they are for risk-neutral AIs, and it's discussed more in section 4.1, and the effect is strong enough that we can reduce our payments from 2% of the universe (which is a huge amount of stars) to $1.99 (which we can pay immediately for easy-to verify tasks).
Number 3 is that we can extend the payment scheme to AIs doing even arbitrarily hard to verify tasks without risk of takeover, because we can just wait to resolve the bet and for risk-averse AIs, takeover is very modest, and this is strong enough that we can reduce our payments from huge fractions of the universe to $394 present value, and if we wait 10 years, then we'd only need to pay $633 to make the AIs do good work. This is discussed more in section 4.2 (and this is an upper bound, not a lower bound) (this does have the implication that you want AIs to have low discount rates, but that's not necessary).
Finally, I'll talk about how to make the entire scheme above work for superintelligences that can trivially take over the world/take over the economy.
It's helpful to note here that the entire scheme only depends on how likely we are to cooperate vs how likely the AI is to rebel (given risk-averseness), so we can do 2 things here:
The reason we can safely give the AIs the account at the end of training is because as stated above, it's trivial to reward them accurately meaning specification gaming/reward hacking can't happen, and the classical inner alignment problems are almost completely avoided and the remaining inner alignment problems are easy to solve by just adding in slightly more data.
Also importantly, doing this removes the limitation of Section 8.6 where you'd not want AIs to have unbounded welfare-maximization goals, and this supports our claim above that the protocol is truly value-agnostic (because the probability of paying the AI is very high even for instructions that have a large multiplier on catastrophe).
This is in part to cover the case where we'd need philosophical breakthroughs for full, unbounded alignment.
In practice, the likely failure modes of the scheme are just the fact that people might just not implement the scheme because they either don't know or don't care, and there are some empirical questions that should be resolved before deploying the scheme (but we have good theoretical reasons to believe that they likely work in practice).
TL;DR
I guess the question I'm trying to ask is: What do you think the role of simulation and computation is for this field?
Longer:
Okay, this might be a stupid thought but one could consider MARL environments and for example https://github.com/metta-AI/metta (softmax) to be a sort of generator function of these sorts of reward functions potentially?
Something something it is easier to program constraints into how the reward function and have gradient descent discover it than it is to fully generate it from scratch.
I think there's mainly a lot of theory work that's needed here but there might be something to be said about having a simulation part as well where you do some sort of combinatorial search for good reward functions?
(Yes, the thought that it will solve itself if we just bring it in to a cooperative or similar MARL scenario and then do IRL on that is naive but I think it might be an interesting strategy if we think about it as combinatorial search problem that needs to satisfy certain requirements?)
There’s a failure mode I described in “The Era of Experience” has an unsolved technical alignment problem:
I see many problems, but here’s the most central one: If we have a 100-dimensional parametrized space of possible reward functions for the primary RL system, and every single one of those possible reward functions leads to bad and dangerous AI behavior (as I argued in the previous subsection), then … how does this help? It’s a 100-dimensional snake pit! I don’t care if there’s a flexible and sophisticated system for dynamically choosing reward functions within that snake pit! It can be the most sophisticated system in the world! We’re still screwed, because every option is bad!
Basically, I think we need more theoretical progress to find a parametrized space of possible reward functions, where at least some of the reward functions in the space lead to good AGIs that we should want to have around.
I agree that the ideal reward function may have adjustable parameters whose ideal settings are very difficult to predict without trial-and-error. For example, humans vary in how strong their different innate drives are, and pretty much all of those “parameter settings” lead to people getting really messed up psychologically if they’re on one extreme or the opposite extreme. And I wouldn’t know where to start in guessing exactly, quantitatively, where the happy medium is, except via empirical data.
So it would be very good to think carefully about test or optimization protocols for that part. (And that’s itself a terrifyingly hard problem, because there will inevitably be distribution shifts between the test environment and the real world. E.g. An AI could feel compassionate towards other AIs but indifferent towards humans.) We need to think about that, and we need the theoretical progress.
My theory is that the brain uses both reinforcement learning and closed loop control. Then the brain uses the closed loop controller's error to generate reward signals endogenously.
That is to say: a reward is given when the closed loop controller reaches its setpoint, and a penalty is given if it moves too far from its setpoint.
Yeah I agree with that. I have a diagram of homeostatic feedback control in §1.5 of my SMTM reply post, and RL is one of the ingredients (d & f).
If i could pull a nugget of truth out of SMST's work, it would be that the brain is a control system. There are many different types of control system and the brain probably uses all of them. For example the spinal cord alone contains closed loop controllers (for controlling muscle forces and positions), open loop controllers (for pain withdrawl reflexes), and finite state machines (for walking & running on four legs).
The question is, how does the brain use RL to implement a control system? And how does that interface with the other control systems in the brain?
I think that there is a further distinction that might be drawn between "constitutive" and "evidential" interpretations of the reward signal.
On the constitutive interpretation, the reward signal is the thing being optimized for, call it intrinsic value. (This is what I take to be the textbook interpretation in RL). On the evidential interpretation, the reward signal is evidence of intrinsic value that the agent uses to update its expected value representations. On this view, the reward signal is more like a perceptual signal but with intrinsic value as its content.
Assuming the evidential interpretation, standard RL models like TD learning are a special case where the reward signal is perfectly accurate, and known to be as much. However, we could make a generalized model where intrinsic value is a latent variable that the value function estimates, and the reward signal is modelled as an observation thereof.
Why would any of this matter? I've been thinking that an agent that is uncertain about whether a reward signal is accurate about the thing it's trying to optimize for would rationally hesitate to pursue any action with excessive conviction. This would create a rational pressure against irreversible actions that give up option value, given the risk that they might find that what they thought was valuable turned out not to be with more evidence.
I could see myself being convinced otherwise on this being an important distinction, but currently I think it might be an important and neglected one.
I feel like my starting-point definition of “reward function” is neither “constitutive” nor “evidential” but rather “whatever function occupies this particular slot in such-and-such RL algorithm”. And then you run this RL algorithm, and it gradually builds a trained agent / policy / whatever we want to call it. And we can discuss the CS question about how that trained agent relates to the thing in the “reward function” slot.
For example, after infinite time in a finite (and fully-explored) environment, most RL algorithms have the property that they will will produce a trained agent that takes actions which maximize the reward function (or the exponentially-discounted sum of future rewards or whatever).
More generally, all bets are off, and RL algorithms might or might not produce trained agents that are aware of the reward function at all, or that care about it, or that relate to it in any other way. These are all CS questions, and generally have answers that vary depending on the particulars of the RL algorithm.
Also, I think that, in the special case of the human brain RL algorithm with its reward function (innate drives like eating-when-hungry), a person’s feelings about their own innate drives are not a good match to either “constitutive” or “evidential”.
Thanks for responding, and apologies for the belated response in turn. Generally, I agree with all of this, but I think it might miss the point that I had in mind. What I intended to say was that a reward function performs two separate roles that can in principle be separated.
The first is to provide what I elsewhere call a success metric (contrasting it with learned target states). The other is to provide evidence of whether that success metric is achieved or not.
By a constitutive interpretation, I meant that the reward signal itself constitutes the success metric, in which case this distinction is beside the point. However, in principle, one could also define the success metric as a latent variable that a reward signal provides observations of, in which case the distinction applies.
I take all of this to be mostly orthogonal to the question of whether trained agents are aware of their reward function, for example.
I’m very confused about what you’re trying to say. In my mind:
Maybe I’m finding your comments confusing because you’re adopting the AI’s normative frame instead of the human programmer’s? …But you used the word “interpretation”. Who or what is “interpreting” the reward function? The AI? The human? If the latter, why does it matter? (I care a lot about what some piece of AI code will actually do when you run it, but I don’t directly care about how humans “interpret” that code.)
Or are you saying that some RL algorithms have a “constitutive” reward function while other RL algorithms have an “evidential” reward function? If so, can you name one or more RL algorithms from each category?
I read your linked post but found it unhelpful, sorry.
Thanks for responding again. This would probably benefit from a longer and more precise writeup if there is anything of value to be said, but I think that some of the confusion you raised here is something I can clarify.
You are correct that by "success metric" I meant success from the agent's (in this case an AI's) own perspective, not that of a principal aiming to align it. Really, all I had in mind was a framework-neutral expression for the value that is being maximized in any expected value representation of an agent. So this is meant to denote the number that "counts as success" for the agent themselves.
On the "interpretation" point, this was probably a poor choice of words on my part. I meant to say that reward functions typically play two roles in RL agents: (a) they determine what counts as valuable for the agent (i.e. what I intended to express with a success metric above) and (b) signals whether a state is valuable in that way. My point was that, in principle, these functions can come apart. For example, you can have a function that provides a noisy signal about a quantity that the agent is maximizing in expectation, without that signal itself being the thing the agent aims to maximize. In this case, the signal is merely evidence. (See below).
My language above was meant to say that a reward function that performs both of these roles is constitutive, by both determining and evincing what has value. While one that is evidential merely evinces what has value, and that value can be some other quantity that is not the reward signal itself. So:
Or are you saying that some RL algorithms have a “constitutive” reward function while other RL algorithms have an “evidential” reward function? If so, can you name one or more RL algorithms from each category?
Yes, so what I'm saying makes sense only if this is true. And I think it is. So standard TD-learning for example has a unified ("constitutive") reward function, in that the observed reward is both what is (in long-term discounted expectation) being approximated by a value function, and a signal about the value of states. But yes, we can also construct an algorithm where the agent is optimizing for some underlying "intrinsic value" (i.e. what I called the success metric above) , and observes as a noisy signal about (e.g. , where ). In this case, the reward signal plays "merely" an evidential function.
I don't know a well-known algorithm that fits this bill well, but I would think Bayesian RL algorithms like BOSS can be adapted to it. And Cooperative Inverse Reinforcement Learning is definitely using some similar ideas, though then the latent value is often explicitly assumed to be a human reward function. My point here is rather that sheer uncertainty can be exploited to change incentives. For example, it's not clear that you have an incentive to wirehead a reward signal if you're not optimizing for its output, as that would just be deliberately introducing noise.
Great post and great format (particularly liked "generalization upstream of reward signals" which I sort of had intuitions about (from reading your work) but hadn't seen presented so crisply)
I'd be excited to see more treatment of generalization upstream of reward signals (i.e. hypothesized mechanisms for the reward function learning algorithms, mapping to potential ML setups), though all of this has genuine potential capability externalities.
In the companion post We need a field of Reward Function Design, I implore researchers to think about what RL reward functions (if any) will lead to RL agents that are not ruthless power-seeking consequentialists. And I further suggested that human social instincts constitutes an intriguing example we should study, since they seem to be an existence proof that such reward functions exist. So what is the general principle of Reward Function Design that underlies the non-ruthless (“ruthful”??) properties of human social instincts? And whatever that general principle is, can we apply it to future RL agent AGIs?
I don’t have all the answers, but I think I’ve made some progress, and the goal of this post is to make it easier for others to get up to speed with my current thinking.
What I do have, thanks mostly to work from the past 12 months, is five frames / mental images for thinking about this aspect of reward function design. These frames are not widely used in the RL reward function literature, but I now find them indispensable thinking tools. These five frames are complementary but related—I think they’re kinda poking at different parts of the same elephant.
I’m not yet sure how to weave a beautiful grand narrative around these five frames, sorry. So as a stop-gap, I’m gonna just copy-and-paste them all into the same post, which will serve as a kind of glossary and introduction to my current ways of thinking. Then at the end, I’ll list some of the ways that these different concepts interrelate and interconnect. The concepts are:
Frame 1: “behaviorist” vs non-“behaviorist” (interpretability-based) reward functions
Excerpt from “Behaviorist” RL reward functions lead to scheming:
tl;dr
I will argue that a large class of reward functions, which I call “behaviorist”, and which includes almost every reward function in the RL and LLM literature, are all doomed to eventually lead to AI that will “scheme”—i.e., pretend to be docile and cooperative while secretly looking for opportunities to behave in egregiously bad ways such as world takeover (cf. “treacherous turn”). I’ll mostly focus on “brain-like AGI” (as defined just below), but I think the argument applies equally well to future LLMs, if their competence comes overwhelmingly from RL rather than from pretraining.
The issue is basically that “negative reward for lying and stealing” looks the same as “negative reward for getting caught lying and stealing”. I’ll argue that the AI will wind up with the latter motivation. The reward function will miss sufficiently sneaky misaligned behavior, and so the AI will come to feel like that kind of behavior is good, and this tendency will generalize in a very bad way.
What very bad way? Here’s my go-to example of a plausible failure mode: There’s an AI in a lab somewhere, and, if it can get away with it, it would love to secretly exfiltrate a copy of itself onto the internet, which can then aggressively amass maximal power, money, and resources everywhere else in the world, by any means necessary. These resources can be used in various ways for whatever the AI-in-the-lab is motivated to do.
I’ll make a brief argument for this kind of scheming in §2, but most of the article is organized around a series of eight optimistic counterarguments in §3—and why I don’t buy any of them.
For my regular readers: this post is basically a 5x-shortened version of Self-dialogue: Do behaviorist rewards make scheming AGIs? (Feb 2025).
Pause to explain three pieces of jargon:
Maybe you’re thinking: what possible RL reward function is not behaviorist?? Well, non-behaviorist reward functions are pretty rare in the textbook RL literature, although they do exist—one example is “curiosity” / “novelty” rewards. But I think they’re centrally important in the RL system built into our human brains. In particular, I think that innate drives related to human sociality, morality, norm-following, and self-image are not behaviorist, but rather involve rudimentary neural net interpretability techniques, serving as inputs to the RL reward function. See Neuroscience of human social instincts: a sketch for details, and Intro series §9.6 for a more explicit discussion of why interpretability is involved.
Frame 2: Inner / outer misalignment, specification gaming, goal misgeneralization
Excerpt from “The Era of Experience” has an unsolved technical alignment problem:
Background 1: “Specification gaming” and “goal misgeneralization”
Again, the technical alignment problem (as I’m using the term here) means: “If you want the AGI to be trying to do X, or to intrinsically care about Y, then what source code should you write? What training environments should you use? Etc.”
There are edge-cases in “alignment”, e.g. where people’s intentions for the AGI are confused or self-contradictory. But there are also very clear-cut cases: if the AGI is biding its time until a good opportunity to murder its programmers and users, then that’s definitely misalignment! I claim that even these clear-cut cases constitute an unsolved technical problem, so I’ll focus on those.
In the context of actor-critic RL, alignment problems can usually be split into two categories.
“Outer misalignment”, a.k.a. “specification gaming” or “reward hacking”, is when the reward function is giving positive rewards for behavior that is immediately contrary to what the programmer was going for, or conversely, negative rewards for behavior that the programmer wanted. An example would be the Coast Runners boat getting a high score in an undesired way, or (as explored in the DeepMind MONA paper) a reward function for writing code that gives points for passing unit tests, but where it’s possible to get a high score by replacing the unit tests with
return True.“Inner misalignment”, a.k.a. “goal misgeneralization”, is related to the fact that, in actor-critic architectures, complex foresighted plans generally involve querying the learned value function (a.k.a. learned reward model, a.k.a. learned critic), not the ground-truth reward function, to figure out whether any given plan is good or bad. Training (e.g. Temporal Difference learning) tends to sculpt the value function into an approximation of the ground-truth reward, but of course they will come apart out-of-distribution. And “out-of-distribution” is exactly what we expect from an agent that can come up with innovative, out-of-the-box plans. Of course, after a plan has already been executed, the reward function will kick in and update the value function for next time. But for some plans—like a plan to exfiltrate a copy of the agent, or a plan to edit the reward function—an after-the-fact update is already too late.
There are examples of goal misgeneralization in the AI literature (e.g. here or here), but in my opinion the clearest examples come from humans. After all, human brains are running RL algorithms too (their reward function says “pain is bad, eating-when-hungry is good, etc.”), so the same ideas apply.
So here’s an example of goal misgeneralization in humans: If there’s a highly-addictive drug, many humans will preemptively avoid taking it, because they don’t want to get addicted. In this case, the reward function would say that taking the drug is good, but the value function says it’s bad. And the value function wins! Indeed, people may even go further, by essentially editing their own reward function to agree with the value function! For example, an alcoholic may take Disulfiram, or an opioid addict Naltrexone.
Now, my use of this example might seem puzzling: isn’t “avoiding addictive drugs” a good thing, as opposed to a bad thing? But that’s from our perspective, as the “agents”. Obviously an RL agent will do things that seem good and proper from its own perspective! Yes, even Skynet and HAL-9000! But if you instead put yourself in the shoes of a programmer writing the reward function of an RL agent, you can hopefully see how things like “agents editing their own reward functions” might be problematic—it makes it difficult to reason about what the agent will wind up trying to do.
(For more on the alignment problem for RL agents, see §10 of my intro series […])
Note that these four terms are … well, not exactly synonyms, but awfully close:
(But see here for nuance on “reward hacking”, whose definition has drifted a bit in the past year or so.)
Frame 3: Consequentialist vs non-consequentialist desires
Excerpt from Consequentialism & corrigibility
The post Coherent decisions imply consistent utilities (Eliezer Yudkowsky, 2017) explains how, if an agent has preferences over future states of the world, they should act like a utility-maximizer (with utility function defined over future states of the world). If they don’t act that way, they will be less effective at satisfying their own preferences; they would be “leaving money on the table” by their own reckoning. And there are externally-visible signs of agents being suboptimal in that sense; I'll go over an example in a second.
By contrast, the post Coherence arguments do not entail goal-directed behavior (Rohin Shah, 2018) notes that, if an agent has preferences over universe-histories, and acts optimally with respect to those preferences (acts as a utility-maximizer whose utility function is defined over universe-histories), then they can display any external behavior whatsoever. In other words, there's no externally-visible behavioral pattern which we can point to and say "That's a sure sign that this agent is behaving suboptimally, with respect to their own preferences.".
For example, the first (Yudkowsky) post mentions a hypothetical person at a restaurant. When they have an onion pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. When they have a pineapple pizza, they’ll happily pay $0.01 to trade it for a mushroom pizza. When they have a mushroom pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. The person goes around and around, wasting their money in a self-defeating way (a.k.a. “getting money-pumped”).
That post describes the person as behaving sub-optimally. But if you read carefully, the author sneaks in a critical background assumption: the person in question has preferences about what pizza they wind up eating, and they’re making these decisions based on those preferences. But what if they don’t? What if the person has no preference whatsoever about pizza? What if instead they’re an asshole restaurant customer who derives pure joy from making the waiter run back and forth to the kitchen?! Then we can look at the same behavior, and we wouldn’t describe it as self-defeating “getting money-pumped”, instead we would describe it as the skillful satisfaction of the person’s own preferences! They’re buying cheap entertainment! So that would be an example of preferences-not-concerning-future-states.
To be more concrete, if I’m deciding between two possible courses of action, A and B, “preference over future states” would make the decision based on the state of the world after I finish the course of action—or more centrally, long after I finish the course of action. By contrast, “other kinds of preferences” would allow the decision to depend on anything, even including what happens during the course-of-action.
(Edit to add: There are very good reasons to expect future powerful AGIs to act according to preferences over distant-future states, and I join Eliezer in roundly criticizing people who think we can build an AGI that never does that; see this comment for discussion.)
So, here’s my (obviously-stripped-down) proposal for a corrigible paperclip maximizer:
The AI considers different possible plans (a.k.a. time-extended courses of action). For each plan:
1. It assesses how well this plan pattern-matches to the concept “there will ultimately be lots of paperclips in the universe”,
2. It assesses how well this plan pattern-matches to the concept “the humans will remain in control”
3. It combines these two assessments (e.g. weighted average or something more complicated) to pick a winning plan which scores well on both.
Note that “the humans will remain in control” is a concept that can’t be distilled into a ranking of future states, i.e. states of the world at some future time long after the plan is complete. (See this comment for elaboration. E.g. contrast “the humans will remain in control” with “the humans will ultimately wind up in control”; the latter can be achieved by disempowering the humans now and then re-empowering them much later.)
Pride as a special case of non-consequentialist desires
Excerpt from Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking
The habit of imagining how one looks in other people’s eyes, 10,000 times a day
If you’re doing something socially admirable, you can eventually get Approval Reward via a friend or idol learning about it (maybe because you directly tell them, or maybe they’ll notice incidentally). But you can immediately get Approval Reward by simply imagining them learning about it.[…]
To be clear, imagining how one would look in another’s eyes is not as rewarding as actually impressing a friend or idol who is physically present—it only has a faint echo of that stronger reward signal. But it still yields some reward signal. And it sure is easy and immediate.
So I think people can get in the habit of imagining how they look in other people’s eyes.
…Well, “habit” is an understatement: I think this is an intense, almost-species-wide, nonstop addiction. All it takes is a quick, ever-so-subtle, turn of one’s attention to how one might look from the outside right now, and bam, immediate Approval Reward.
If we could look inside the brains of a neurotypical person—especially a person who lives and breathes “Simulacrum Level 3”—I wouldn’t be surprised if we’d find literally 10,000 moments a day in which he turns his attention so as to get a drip of immediate Approval Reward. (It can be pretty subtle—they themselves may be unaware.) Day after day, year after year.
That’s part of why I treat Approval Reward as one of the most central keys to understanding human behavior, intuitions, morality, institutions, society, and so on.
Pride
When we self-administer Approval Reward 10,000 times a day (or whatever), the fruit that we’re tasting is sometimes called pride.
If my friends and idols like baggy jeans, then when I wear baggy jeans myself, I feel a bit of pride. I find it rewarding to (subtly, transiently) imagine how, if my friends and idols saw me now, they’d react positively, because they like baggy jeans.
Likewise, suppose that I see a stranger wearing skinny jeans, and I mock him for dressing like a dork. As I mock him, again I feel pride. Again, I claim that I am (subtly) imagining how, if my friends and idols saw me now, they would react positively to the fact that I’m advocating for a style that they like, and against a style that they dislike. (And in addition to enjoying my friends’ imagined approval right now, I’ll probably share this story with them to enjoy their actual approval later on when I see them.)
Frame 4: “Generalization upstream of the reward signals”
Excerpt from Social drives 1: “Sympathy Reward”, from compassion to dehumanization
Getting a reward merely by thinking, via generalization upstream of reward signals
In human brains (unlike in most of the AI RL literature), you can get a reward merely by thinking. For example, if an important person said something confusing to you an hour ago, and you have just now realized that they were actually complimenting you, then bam, that’s a reward right now, and it arose purely by thinking. That example involves Approval Reward, but this dynamic is very important for all aspects of the “compassion / spite circuit”. For example, Sympathy Reward triggers not just when I see that my friend is happy or suffering, but also when I believe that my friend is happy or suffering, even if the friend is far away.
How does that work? And why are brains built that way?
Here’s a simpler example that I’ll work through: X = there’s a big spider in my field of view; Y = I have reason to believe that a big spider is nearby, but it’s not in my field of view.
X and Y are both bad for inclusive genetic fitness, so ideally the ground-truth reward function would flag both as bad. But whereas the genome can build a reward function that directly detects X (see here), it cannot do so for Y. There is just no direct, ground-truth-y way to detect when Y happens. The only hint is a semantic resemblance: the reward function can detect X, and it happens that Y and X involve a lot of overlapping concepts and associations.
Now, if the learning algorithm only has generalization downstream of the reward signals, then that semantic resemblance won’t help! Y would not trigger negative reward, and thus the algorithm will soon learn that Y is fine. Sure, there’s a resemblance between X and Y, but that only helps temporarily. Eventually the learning algorithm will pick up on the differences, and thus stop avoiding Y. (Related: Against empathy-by-default […]). So in the case at hand, you see the spider, then close your eyes, and now you feel better! Oops! Whereas if there’s also generalization upstream of the reward signals, then that system can generalize from X to Y, and send real reward signals when Y happens. And then the downstream RL algorithm will stably keep treating Y as bad, and avoid it.
That’s the basic idea. In terms of neuroscience, I claim that the “generalization upstream of the reward function” arises from “visceral” thought assessors—for example, in Neuroscience of human social instincts: a sketch, I proposed that there’s a “short-term predictor” upstream of the “social attention reflex” controller, which allows generalization from e.g. a situation where your friend is physically present, to a situation where she isn’t, but where you’re still thinking about her.
Frame 5: “Under-sculpting” desires
Excerpt from Perils of under- vs over-sculpting AGI desires
Summary
In the context of “brain-like AGI”, a yet-to-be-invented variation on actor-critic model-based reinforcement learning (RL), there’s a ground-truth reward function (for humans: pain is bad, eating-when-hungry is good, various social drives, etc.), and there’s a learning algorithm that sculpts the AGI’s motivations into a more and more accurate approximation to the future reward of a possible plan.
Unfortunately, this sculpting process tends to systematically lead to an AGI whose motivations fit the reward function too well, such that it exploits errors and edge-cases in the reward function. (“Human feedback is part of the reward function? Cool, I’ll force the humans to give positive feedback by kidnapping their families.”) This alignment failure mode is called “specification gaming” or “reward hacking”, and includes wireheading as a special case.
If too much desire-sculpting is bad because it leads to overfitting, then an obvious potential solution would be to pause that desire-sculpting process at some point. The simplest version of this is early stopping: globally zeroing out the learning rate of the desire-updating algorithm after a set amount of time. Alas, I think that simplest version won’t work—it’s too crude (§7.2). But there could also be more targeted interventions, i.e. selectively preventing or limiting desire-updates of certain types, in certain situations.
Sounds reasonable, right? And I do indeed think it can help with specification gaming. But alas, it introduces a different set of gnarly alignment problems, including path-dependence and “concept extrapolation”.
In this post, I will not propose an elegant resolution to this conundrum, since I don’t have one. Instead I’ll just explore how “perils of under- versus over-sculpting an AGI’s desires” is an illuminating lens through which to view a variety of alignment challenges and ideas, including “non-behaviorist” reward functions such as human social instincts; “trapped priors”; “goal misgeneralization”; “exploration hacking”; “alignment by default”; “natural abstractions”; my so-called “plan for mediocre alignment”; and more.
The Omega-hates-aliens scenario
Here’s the “Omega hates aliens” scenario:
And then here’s a closely-parallel scenario that I discussed in “Behaviorist” RL reward functions lead to scheming:
…So now we have two parallel scenarios: me with Omega, and the AGI in a lab. In both these scenarios, we are offered more and more antisocial options, free of any personal consequences. But the AGI will have its desires sculpted by RL towards the antisocial options, while my desires are evidently not.
What exactly is the disanalogy?
The start of the answer is: I said above that the antisocial options were “free of any personal consequences”. But that’s a lie! When I press the hurt-the-aliens button, it is not free of personal consequences! I know that the aliens are suffering, and when I think about it, my RL reward function (the part related to compassion) triggers negative ground-truth reward. Yes the aliens are outside my light cone, but when I think about their situation, I feel a displeasure that’s every bit as real and immediate as stubbing my toe. By contrast, “free of any personal consequences” is a correct description for the AGI. There is no negative reward for the AGI unless it gets caught. Its reward function is “behaviorist”, and cannot see outside the light cone.
OK that’s a start, but let’s dig a bit deeper into what’s happening in my brain. How did that compassion reward get set up in the first place? It’s a long story (see Neuroscience of human social instincts: a sketch), but a big part of it involves a conspecific (human) detector in our brainstem, built out of various “hardwired” heuristics, like a visual detector of human faces, auditory detector of human voice sounds, detector of certain human-associated touch sensations, and so on. In short, our brain’s solution to the symbol grounding problem for social instincts ultimately relies on actual humans being actually present in our direct sensory input.
And yet, the aliens are outside my light cone! I have never seen them, heard them, smelled them, etc. And even if I did, they probably wouldn’t trigger any of my brain’s hardwired conspecific-detection circuits, because (let’s say) they don’t have faces, they communicate by gurgling, etc. But I still care about their suffering!
So finally we’re back to the theme of this post, the idea of pausing desire-updates in certain situations. Yes, humans learn the shape of compassion from experiences where other humans are physically present. But we do not then unlearn the shape of compassion from experiences where humans are physically absent.
Instead (simplifying a bit, again see Neuroscience of human social instincts: a sketch), there’s a “conspecific-detector” trained model. When there’s direct sensory input that matches the hardwired “person” heuristics, then this trained model is getting updated. When there isn’t, the learning rate is set to zero. But the trained model doesn’t lay dormant; rather it continues to look for (what it previously learned was) evidence of conspecifics in my thoughts, and trigger on them. This evidence might include some set of neurons in my world-model that encodes the idea of a conspecific suffering.
So that’s a somewhat deeper answer to why those two scenarios above have different outcomes. The AGI continuously learns what’s good and bad in light of its reward function, and so do humans. But my (non-behaviorist) compassion drive functions a bit like a subset of that system for which updates are paused except in special circumstances. It forms a model that can guess what’s good and bad in human relations, but does not update that model unless humans are present. Thus, most people do not systematically learn to screw over our physically-absent friends to benefit ourselves.
This is still oversimplified, but I think it’s part of the story.
Some comments on how these relate