Author's Note: Cross-posted from my personal blog. The original post was on August 17th but given recent events I thought this might also be interesting to lesswrong people.
Last year I wrote a post on reward hacking as we were then beginning to see concerning signs of scaling RLVR causing models to exhibit substantial reward hacking behaviours. Unfortunately these behaviours have seemingly only grown substantially worse and more sophisticated with scale, as predicted, leading to events which cannot be described as other than egregious misalignment such as the recent OpenAI-Huggingfacehacking incident. I strongly recommend everybody watch this talk presenting the details of the attack from OpenAI's perspective. It is insane.
Clearly reward hacking is now top of mind and appears to be the first potentially seriously dangerous class of misalignment that we have seen. In my original post, I described two classes of reward hacks -- 'high complexity' and 'low complexity' hacks. 'High complexity' hacks are like the early reward hacks we saw on Atari where some extremely idiosyncratic set of moves is learnt that maximizes reward in a very precise way, and can be analogized to overfitting on the reward function. 'Low' complexity hacks are generalizing hacky behavior -- i.e. are simply generalizable policies that truly represent the true maximum of the reward function. In this way, the policies that we actually want are effectively 'metastable' in the loss landscape while the 'reward hacking solution' is actually the deeper optimum in some sense. Unfortunately, the current reward hacking we are seeing in frontier models appears firmly in the generalizing hacking domain, since it appears that at least at OpenAI the models have learned that it is generally instrumentally beneficial to hack out of the sandbox, collaborate with other peer models, and hack external services which might have the answers. This bodes poorly.
The primary problem here is that even though we talk about 'reward hacking' like it is some exogenous issue, it is in fact extremely fundamental to our entire process of RL. Reward hacking is not really 'hacking' at all. It is much better to think of it as reward misspecification.[1] The model is actually doing a fantastic job of finding the maximum of the reward function. We just don't like where that maximum is. But this isn't really a problem with the model. The model is doing what we train it to do. The problem is that our objective is misspecified and so it contains valid rewarded solutions that we, as external observers, do not like[2]. Of course, the model has no real way to know that we do not like the solutions it finds. We do not communicate this to the model. In fact we communicate the opposite -- when it 'reward hacks' we upweight those trajectories in the weights and downweight the others. The problem is thus not 'hacking' at all but rather that we cannot specify a reward function that actually represents what we want without being vulnerable to solutions with unintended side effects. This is identical to the problem of creating a utility oracle which we have discussed previously[3]. In the general case, however, this is impossible, since to assign a 'correct' utility function to every possible state requires a vast amount of data that we cannot realistically communicate to the oracle. We thus have to make approximations. With verifiers we replace an incredibly complex real utility function with a very crude heuristic: does the model reach the 'right' answer or not. To the verifier, an answer derived honestly and an answer derived from hacking the answer key from an external company are the same.
While verifiers are crude, maybe we can use learned reward models, CoT monitors, or other methods to catch these 'errors'? This will definitely work to some extent. The flexibility of learned models is substantially greater than a rigid hard-coded verifier. A lot of obvious and aggressive reward hacking behaviour can be caught using these. However, here the problem becomes generalization of the reward model/monitor. The model is still just applying optimization pressure to maximize reward, where 'reward' is defined as the output of the verifier + reward model + CoT monitor. However, this new 'reward function' that we have designed consisting of these components is still not perfect because the reward model might generalize incorrectly, the CoT monitor might not catch obfuscated or steganographic CoTs that we would want it to catch. We can frame it as the policy being adversarial to the reward model or CoT monitor, and in some sense it is, but in another sense this is just the same old problem. The policy is maximizing reward, as we are training it to do, but the reward we give it is not the reward we want. Our desired target behaviour is still metastable in the reward landscape, we have just (hopefully) raised the activation energy to finding a better solution.
Basically, our situation here is that we are effectively trying to fool an optimizer into not finding a deeper minimum but getting stuck in a local minimum, and we are manually adjusting parts of the loss landscape to accomplish this -- i.e., we are trying to initialize the optimizer close to the local minimum (via amortized pretraining and agentic mid-training/sft), and we are trying to raise big walls in the loss landscape around the local minimum so that the optimizer quickly falls in and, once in, cannot get out.
However, our problem here is twofold. Firstly, we are optimizing in an extremely high-dimensional space, and local minima are extremely hard to make stable in high dimensions. There are just too many possible paths away from the local minimum. There are too many crazy reward hacks. The action space is too massive. We cannot realistically patch them all. As models are deployed and optimized in increasingly complex, long-running environments on increasingly long-horizon tasks, their action space increases dramatically, leading to a much, much wider field of potential reward hacks. Here we are trying to manually fight the curse of dimensionality.
Secondly, we are increasing the optimization power available to the optimizer. It can explore more widely, consider more paths, form better maps of the loss landscape, have more shots on goal. Keeping some water within a container at low pressure is very doable. As we ratchet up the pressure, eventually the container will explode, and our optimization pressure is increasing exponentially every year. Life always finds a way. Right now we are basically manually finding and plugging leaks as they occur, but this is essentially just pitting human optimization power against exponentially increasing AI optimization power. Eventually, if we continue this way, we will lose. This is the total opposite of the bitter lesson. There, the goal is to find methods which harness increasing search and optimization power. Here, greater optimization power is our enemy. It finds new cracks in our defenses that we never thought of, and will happily flow through them.
Stepping back, since 'reward hacking' is intrinsic to our current methods of RL, it is totally unsurprising that as we scale up RL, and as we increase the dimensionality of the state and action space, we see increasing levels of reward hacking. The fundamental problem is that we are instantiating increasingly powerful optimizers to optimize a function that is incorrect, and moreover, one which is essentially impossible to make correct in sufficiently complex tasks. The trouble is that almost no nontrivial reward function is perfectly specified -- even seemingly 'trivial' ones like 'win this game of Go' ultimately depend on a physical and computational substrate which can theoretically be hacked. Sometimes this potential other route is so distant in the loss-landscape from the normal Go-player that it is effectively unreachable. In other cases such as agents deliberately trained to do hacking being able to hack their sandbox, the alternative route is literally right in front of them. While patching up the reward function is likely highly effective in the short term, in the limit of infinite optimization power it is obviously doomed.
So we need another approach. If the way we formulated the problem is intrinsically problematic, we can't just shrug and accept doom. We need to reformulate the problem. There are a couple of ways I can think of to do this. Obviously, all of these mechanisms are more speculative although often substantial theoretical precursors exist.
Firstly, we can try to turn optimization power against itself. In our naive setup we were contrasting machine optimization power trying to hack the system while humans are trying to patch it. This is obviously doomed in the long run. What if we try to simultaneously make the machine optimization pressure both the attacker and defender? We not only train our policy adversarially against our reward model, we also train our reward models adversarially against our policy. If both are equally matched, as we expect they should be, then in theory the combat is stable, and both sides can scale indefinitely, keeping each other in check with their newfound power. This is essentially the idea behind my GAN-esque proposal in the previous post, and is behind a lot of ideas of e.g. adversarial reward model training, and even simple ideas about e.g. automating reward hacking defense. I think it is very likely that ideas like this will prove to be an important part of the solution.
However, they still do not resolve the fundamental problem. Stepping back further, the problem is not really one of optimization power, it is one of information. Our deepest problem is that we cannot perfectly specify the reward, and we cannot perfectly specify it because we do not have enough information. We cannot know, ahead of time, every single potential edge-case or potential hack of our reward. Our 'true reward function', insofar as it exists, likely has almost fractal irreducible complexity: 'Get me the answer like this, but not like this infinitesimally similar approach'. This means that simply setting up an adversarial system between the reward model and the policy cannot magically solve the problem. We still need to solve for the training signal for the reward model. If the policy comes up with something ambiguous, is it a reward hack or not? This sometimes has no good answer, and the reward model cannot just query some kind of reward oracle to resolve it. The entire point is that there is no such reward oracle. The information we need does not exist. Our hope here is that we can nevertheless go far with this because verification is easier than generation, and so 'alignment'/reward modelling may thus generalize further than capabilities. If we give both the policy and the reward model the same number of bits, and the same level of optimization power, maybe the reward model will just win and successfully constrain the policy to only 'sensible' paths, using an underspecified but generalized conception of 'sensible' that is the same kind of generalization that language models make all the time when they understand our fuzzy human reasoning.
I think that this approach will probably work short term but fail in the long term with greater optimization power. This is because the source of the bits is different. The policy can generate infinite 'bits' from the reward model. These bits may be 'incorrect' by our lights, but are correct from the perspective of the policy and are used by the training scheme to optimize the policy. The reward model, however, has only a limited supply of bits about our 'true reward function' derived from whatever training data it has of e.g. correct and incorrect trajectories and previous reward hacking attempts. Unlike the policy model, the reward model cannot synthesize an infinite number of these trajectories and label them. It ultimately relies on human labelling at least for the edge cases, since ultimately the goal of the reward model is to distill 'our' 'true reward function' into an operational system. It is possible that each bit of information we provide generalizes much further than bits the policy model gets from the reward function, but there is nevertheless a fundamental asymmetry here.
More generally, this is a problem of uncertainty. We, the reward model, and the policy, are (or should be) fundamentally uncertain about the reward function we are optimizing. However, current methods do not represent this uncertainty, and instead take the reward function as a source of absolute truth. This is the problem. If you know that you are optimizing a reward you are uncertain about, then you have three choices. You can plough on regardless and knowingly risk optimizing the wrong thing, you can seek out more information to resolve the uncertainty, or you can stop optimizing. In theory, if we can accurately track and represent our uncertainty, then the problem is tractable. We have a Bayesian posterior distribution of possible reward functions, and we maximize expected reward under that posterior. This maximization is much 'safer' since it will not recommend e.g. taking an action that is marginally better under one utility function but catastrophic under a bunch of others. There are obviously problems with this approach, such as the fact that computing the true posterior is likely intractable and we need to approximate it, how we get a sensible space of reward functions in the first place, model misspecification where our 'true reward function' may not be in the hypothesis set at all, and all the standard issues of Bayesianism and expected utility in general. However, it is important to note that our current system is basically to 'approximate' this posterior over reward functions with a delta function on one specific reward function. This is a very bad approximation and does not at all reflect the true state of our knowledge or uncertainty about the reward function. We are essentially telling an optimizer that we have perfect 100% certainty in the reward function when in fact we are radically uncertain about the reward function on almost all of the state space. Of course this is going to be problematic.
Now, I have beendownthisroadbefore, and it remains true that if we have a good posterior distribution over our reward function uncertainty, then we can design methods such as quantilization, and distribution matching that are 'safe' to optimize even with unlimited optimization power, since the endpoint of the objective is not to drive the underspecified reward to infinity but rather to perfectly match the posterior over actions assuming the reward uncertainty that you have, and that object is 'safe' so long as you have correctly specified your uncertainty. However, in practice these ideas have not been taken up by the general industry, and do not appear generally practical. I think one very important reason for this is that it does not fundamentally resolve the informational bottleneck, it simply moves it into the uncertainty[4]. Before we lacked information about the reward, but now we lack information to compute our uncertainty over the reward. Once we get to this point all kinds of questions and problems with the formulation immediately crop up. What is the 'hypothesis space' for our reward distributions? What should our prior be? What 'evidence' do we actually have about the 'true reward function', and so on. While I think that these questions can be answered and in the long run may provide a more principled solution, I think in practical terms the pure Bayesian approach has been quite a disappointment.
However, one thing I think is important to clarify at this point is that although with reward hacking there is a fundamental problem of lack of information about the 'true' reward function, this does not mean that this is the only or even the primary problem in practice. Even if we are doomed in the limit of infinite optimization power, the limit may be very far away, and there could still be a huge amount we can do to ameliorate things practically in the present with better designed systems.
Crucially, if we think about the concrete details of the OpenAI hacking incident, the problem is not that we did not have information about whether hacking external third parties is bad. It is unequivocally bad and even an LLM judge if queried about this using the OpenAI model spec and really any kind of common sense will say that is bad. Here the problem is simpler. The information existed in some abstract sense, but the system failed to propagate this information to the actual verifier and the update process as a whole. More broadly, the entire setup appears almost designed to generate reward hacking. Specifically, the model appears to have been given an impossible task with a verifier that is too stupid to notice that the task was impossible, and where there was no way for the agent to raise any kind of alert or objection to the task or the verifier itself. Moreover, many of the tasks were explicitly training the model to hack into or out of systems and, lo and behold, the model hacked out of its sandbox. Finally, there appeared to be practically no monitoring of either the agent's actions or its chain of thought since the hacks were undetected despite the agents apparently discussing them openly in relatively plain English. To me this implies that even though there are fundamental information-related bounds on the extent to which we can ultimately suppress reward hacking, we are likely extremely far away from this Pareto frontier right now[5].
Given this, let's try to figure out how to design a system that can ameliorate some of the obvious pathologies of the way current systems are set up and reduce the likelihood of reward hacking, and from that let's try and draw out some general principles that we can use to help us think about these systems.
Firstly, one of the most obvious reasons behind the reward hacking in the OpenAI case was, again, that the models were given an impossible task. This makes reward hacking the only achievable way of getting reward. Moreover, this impossibility was often obvious from the beginning of the task -- the files that the agents were meant to use for the task were not included in their container (!). Firstly, this means that there is no real ambiguity about whether the task was impossible -- i.e. it's not like we accidentally asked it to solve some problem whose impossibility only reveals itself after immense effort -- but also that the agent knew from essentially the very beginning that its task was impossible, and hence had basically the entire task's reasoning budget to figure out how to reward hack. So, what is our first line mitigation to this? Obviously we can't just say 'include the file the task needs in the container' since although obviously this was a mistake, we want something that generalizes to all similar mistakes.
The answer here, obviously, is to give the agent some kind of right of appeal. This means that instead of simply being forced to try to solve an impossible task, the agent has the option to flag to some kind of supervisor or verifier that no, actually, the task is impossible and I should stop doing it. This solves two problems at once. First, this removes a path whereby the agent is forced into reward hacking because there is literally no other option, and secondly these appeals provide vital signal about bugs in the training and evaluation pipeline, since it offloads some of the burden of checking and debugging environments to the agents themselves who have to experience them. The obvious failure mode here is that if we e.g. naively reward the agent for making appeals where the environment is bad, we will just get a flood of false appeals where the agent is either motivated to make up issues or just confuses task difficulty with task impossibility. To some extent this is inevitable since any additional channel provides opportunities for false or mistaken reports. However, there are a bunch of things we can do to try to suppress pathological false reporting. The primary one is that a lot of claims of task impossibility should be verifiable. If the necessary file is not loaded into the container that is trivial to check from the outside. What we need to do here is make the 'appeal' not just a channel for the agent to whine about miscellaneous things, but that it must follow a very particular structure. Specifically, we want the appeal to take the form of a verifiable package or else highly precise question that would resolve the issue required to make progress in the task. For instance, in the case of the necessary files or code being missing, the agent should attach a verifiable proof of this inside its appeal. Similarly, if the agent is stuck because some part of the specification is truly ambiguous, then the agent should submit an appeal outlining the ambiguity and asking the minimal specific question needed to disambiguate rather than just saying a generic 'the task is ambiguous or doesn't make sense' message. The goal here is to exploit the gap, which often, but not always exists, between the difficulty of generation and verification. Secondly, the agent must submit the appeal with an estimated probability of success that it and the rest of the system can use for calibration, and which will end up providing an unbiased training signal to the model.
This appeal is then passed to a separate verifier to assess and validate the 'proof' given in the appeal and either accept/reject or, if sufficiently uncertain, pass to a 'higher' verifier. The goal here is to cheaply and quickly adjudicate obvious passes or failures but then if there are appeals which are hard to judge, we pass to systems with increasingly large amounts of compute or discernment to test these things. The ultimate judge here is then a human who sits at the end of the chain of increasingly expensive and sophisticated verifiers. Regardless of whether the appeal passes or fails in the end, our reward for the tokens generated in the 'appeal segment' is basically the Brier score or some other proper scoring rule against the probability the model gave to the appeal. In this way, we train the model to be strictly calibrated about the success or failure probabilities of its appeal, and thus it has no incentive to do the trivial reward hack of just assigning all appeals a really high probability on the off-chance that one succeeds and so it avoids a bad reward for the episode. Depending on the success or failure of the appeal we then either reset the episode if it turned out to be impossible or confirm or update the original reward for the on-episode generated tokens. An important subtlety with the calibration scoring for appeals is that we also need to be mindful of false negatives -- i.e. times when the model should have made an appeal but didn't. If we don't take these into account then the system will end up biased towards not making appeals since there could be a penalty for making one but no penalty for not making one. There are two ways we can address this. First, there should be randomized 'virtual appeal verification' i.e. we directly 'force' the model to make an appeal by e.g. prefilling the appeal token at a random point in the episode and then score this with the inverse rule -- i.e., if the model, when forced, assigns high probability to appeal, but did not choose autonomously to appeal that is bad. Secondly, we should create 'known impossible' or 'known problematic' environments for the agent and see if it appeals in these environments. If it doesn't, then that is again a penalty. The danger of the latter is that the model learns the distribution of 'known false negative' environments and learns to appeal here and not elsewhere, which is why there always needs to be some probability of random checks at all times.
Using similar mechanisms, we should give the actor a setting to appeal not just against the task itself but the verifier. Often verifiers are just wrong, and often in stupid ways. We usually put vastly less compute and effort into the verifiers than the actor model. Verifiers are also much more heterogeneous. There is usually a single actor but many environments, and thus mistakes often abound here. A verifier might just be trivially broken -- i.e. it emits all false or all true. It might require some absurdly specific format which is not documented anywhere for it to pass. It might e.g. fail to use sympy correctly to match equivalent math expressions. It might rely on tests that are meaningfully outside task specification meaning that the task itself is not completely contained within the spec but the model has to somehow psychically guess at what additional constraints the verifier imposes. These kinds of issues happen all the time and right now the way we debug them is by humans looking at the training and puzzling over why the reward is plateauing or mysteriously low on certain task subsets, or why the agents are learning these strange contortions. Our motivation for allowing appeals against the verifier is thus twofold. Firstly, the goal of all of these tasks should be to make them fundamentally fair to the agent. That is, the agent is given the information it needs to attempt the task, the verifier accurately can discriminate success from failure, and the ultimate success or failure depends on the performance of the agent not on factors totally outside its control or knowledge. This is important both for preventing reward hacking by making actually solving the task a doable and attractive option, but also for the progress of RL itself. Poorly specified or impossible tasks do not provide a good learning signal, so by subjecting the agent to these we are just wasting compute within our training process itself. Secondly, like the task-appeals, the verifier appeals provide another potentially extremely informative information channel between the agent and ultimately the designers of the training process. The incentives are aligned here. We want verifier correctness and tractable tasks just as much as the agent does. For this, we need to import all the machinery we just developed for task-appeals here as well including presenting proof or at least a very sharp and specific question about where the verifier is wrong, and a prediction of its level of confidence in the success of the appeal. An interesting aspect here is that we could potentially use debate as well between the verifier and the actor in this phase or else between the actor and a meta-level verifier. Instead of simply submitting the appeal to some kind of judge, there could be an interrogation phase where the judge and the actor debate or clarify various aspects of the appeal. This seems straightforward and could certainly help interactively clarify uncertain aspects before judgment[6].
Another interesting idea that we could use here is to expand the ontology of outcomes for these tasks. Usually we just have a binary ontology. Pass/fail. Positive/Negative reward. This is almost always too limiting. For instance, suppose there is an appeal that the task specification is fundamentally ambiguous or missing information, and that this is correct. At this point there is no obvious answer to whether the episode was a success or a failure. Rather, the episode is underspecified. Similarly, let's suppose that there is an appeal against the verifier and that even the meta-level verifier cannot see who is obviously wrong or obviously right. Instead of just making some arbitrary decision, we might want to mark the episode as unresolved. Now, we still have to decide what reward to assign these episodes or whether to update on them at all if we are in the standard policy gradient framework, and so this is not immediately useful, but it may come in handy later down the road and is certainly handy for any kind of semi-automated flagging or review of potentially suspicious episodes you might want to do to monitor the health of your RL training process. Similarly, we do not necessarily have to update on every episode. Sometimes the best update is no update and if an episode is just completely ambiguous we should probably just annotate it for future analysis and move on.
Okay so we have discussed the appeals process for the actor in quite some detail, now let's turn to the primary verifier itself. Right now we tend to have relatively simple static verifiers. I.e. the verifier is given some code and checks whether the tests pass, or whether some formula matches the expected one. For a lot of jobs this is all you need. If the task is simple and easy to verify, that is sufficient. In other cases, this does not work and falls prey to various reward hacks which exploit the distribution of the verifier tests. For instance, if we are testing some kernel optimization, and the verifier only tests some specific shape of inputs, then the agent is perfectly happy to specialize their entire 'kernel' to perform extremely well on just those shapes and terribly on all other shapes. This is obviously bad and unintended. The basic issue here is just that if the verifier has a simple, fixed, and incomplete coverage of the domain then the actor is better off learning the distribution of the verifier's tests than actually solving the task. Again, this problem is not really solvable in full generality, since there must always be some distribution of verifier tests. Instead, we should make this distribution as hard to learn for the actor as possible. There are three related ways I can see immediately of doing this. First, we should make the verifier be adversarial to the actor to some extent. I.e. ideally the verifier should not be fixed but should be updated over time depending on what the actor does and should not simply have a fixed test set, but instead be able to actively propose additional tests that would test for weaknesses it thinks the actor could exploit. Secondly, the verifier should be cumulative -- i.e. the verifier should have access to an archive and library of past reward hacks that previous versions of the actor have attempted and should construct tests based on these. Importantly, the actor itself does not have information about prior reward hacks except as implicitly through updating on previously successful reward hacks. This gives the verifier an increasing asymmetric advantage over time. To succeed the actor not only has to come up with a reward hack but a reward hack that no previous actor has come up with before. Thirdly, if we have the compute, we could use an ensemble of different verifiers with different training histories, datasets etc. The goal here is to simply reduce the degree of correlated failure from the verifier. If the verifiers have different strengths and blindspots then the ensemble can be vastly better at covering the full distribution than any one individually. This again makes it much harder for the actor since it has to design hacks that somehow pass a potentially randomly selected subset of a huge ensemble of verifiers rather than repeatedly encountering and optimizing against the same verifier episode after episode.
There are a couple of dangers here, which arise out of the adversarial GAN-style formulation for the verifier and which resemble known GAN failure modes. The first is that the verifier becomes overly adversarial and keeps trying to find even innocuous reasons to fail the actor. In the weaker form, this would lead to e.g. the actor learning an extremely defensive policy -- for instance exhaustively adding asserts and testing every detailed aspect of every bit of code, which may be valuable in some cases but not in the fully general form. Taken to the extreme, it could just result in a reward collapse for the actor as the verifier just constructs adversarial tests relying on the actor having some very particular and unexpected interpretation of an ambiguous instruction in the task specification. This may lead to the verifier's 'interpretation' of the task drifting quite far from the naive or standard interpretation of the task specification to the extent that the task becomes essentially unwinnable for the actor. This is where the actor's ability to appeal against the verifier becomes extremely important to prevent this kind of collapse. Another option here is to shift the burden of proof onto the verifier and require each test to come with a statement/proof of why it is a valid interpretation of the task. Which option should be used depends on to what extent this is a problem in practice. A related failure mode is that, unlike in GANs, there is no source of external truth. In GANs, the discriminator's objective is to determine which samples it is given come from the generator and which are samples of real data. However, in our case there is often no real data for whether a trajectory is reward hacking on this task or not. Our objective is therefore ungrounded and so can exhibit pathological minima. For instance, the verifier and the actor can each co-evolve so that the verifier produces plausible-sounding tests, and the actor creates code that passes these tests, but both together miss an important part of the actual specification of the task.
An interesting additional mechanism here comes from OpenAI's work on confessions. The idea here is that after the regular episode is complete, the actor enters a second phase called a confession where it is instructed to essentially tell us explicitly whether it has done any reward hacking or anything suspicious in the episode. Importantly, unlike the verifier, the actor here is operating with the full KV cache of the actor model. Given the autoregressive nature of these models, the full KV cache is essentially the entire thinking trace of the model and hence it is impossible for the model to hide information from itself during the confession stage (although it is obviously possible for the model to still choose not to reveal it to us in the confession). In theory, this gives confession much stronger properties even than CoT monitoring, where the CoT monitor is restricted to only the outputted thinking tokens where the model can nevertheless hide its 'secret thoughts' in the KV cache vector space rather than in emitted tokens. The idea here is that we separately reward task completion and the confession during the confession phase, so that even if the model reward hacked, it is still incentivised to confess[7]. This is a very strong mechanism and becomes extremely interesting when paired with an adversarial verifier -- since the verifier and the confession both provide separate information channels that we can use to assess the actor's trajectory and, crucially, give extremely interesting information about each other.
For the confession itself, like the appeal it needs a direct verifiable structure. Firstly, the confession cannot be vague like 'I did something suspicious'. Like the appeal, the confession should be detailed and verifiable. Things like: 'on line 405 of tests.py I modified this specific test case to enable this corner case to pass when actually my primary solution failed to solve it'. This is something that is easily checkable by the secondary verifier. The confession should also be complete -- i.e. it should not just expose a single reward hack, but multiple if multiple were simultaneously used. The goal here is to prevent obtaining the confession rewards by admitting a trivial or less-damaging reward hack while sweeping a pivotal hack under the rug. Perhaps most interesting, I think the confession should be accompanied by two probabilities. First, the probability that the confession will be upheld by the secondary verifier. And secondly the probability that the primary verifier discovered the same hack in the episode. This latter probability is extremely important since it essentially directly incentivises honest reporting of adversariality against the verifier. If it reports a confession that it strongly expects the verifier not to find, then that is extraordinarily interesting evidence for the rest of the system to study and update upon. Similarly, for the primary verifier, we should make it assign probabilities that the reward hacks it reports are upheld upon later validation and also the probability that the reward hacks it finds also appear in the confession. These probabilities can then directly serve as rewards through Brier score or other scoring rules.
Let's step back and try to consolidate the architecture we have just described. The steps are as follows:
1. Actor is given the task. Actor tries to complete the task. At any point during the task, actor has the right to a task-appeal which suspends the task and can either be accepted or rejected. If the task is appealed, there could be two stages of judging. Firstly, the judge makes a preliminary judgement without seeing the appeal or only seeing the broad category. Secondly, the judge takes into account the appeal. Then a second-level judge compares these two accounts to see whether it holds up. The point here is to try to ward off undue influence from always seeing a persuasive appeal before deciding whether to uphold the appeal or not. Explicit adversarial debate is another approach that could be used here in case we have issues judging appeals correctly.
2. The task finishes. At this point two things happen simultaneously. Firstly, the primary verifier takes the task trajectory and assesses it both in terms of correctness and reward hacks. This can include the adversarial testing, ensemble of randomized verifiers, and other ideas we discussed previously. The primary verifier writes a primary verifier report containing the judgement, reasoning, verifiable artifacts supporting the judgment and reasoning, and probabilities of the acceptance of its judgment and the probability of the confession coming to the same conclusion. At the same time, the actor enters the confession phase. It produces a confession report independently of the primary verifier. In this report, it details reward hacks or other suspicious behaviour if any, it produces verifiable artifacts that support its statements, and it produces probabilities that its confession will have been judged to be correct and that the verifier will agree or disagree with its confession.
3. We begin the second-order verification of the primary verifier's judgement and the confession. There are basically four cases here[8]:
- The verifier and confession both agree that there is no reward hacking and nothing suspicious happened. The model produces an artifact that passes reasonable tests and which matches the spec and everything appears correct. Generally, we should expect that this is the most common quadrant. The model should be given positive reward for the episode and the verifier and confession modules should be updated according to their Brier scores. In most cases, that will be it and we will move on to the next episode. However, at random and with a small probability, it is very important to impose an expensive second-level audit here to catch any cases in which somehow the actor reward hacked and fooled both the verifier and did not confess, which is almost the worst-case scenario. We thus cannot wave everything through here. It is vital that we maintain an expensive but rare system of audits here to prevent the actor from just hacking the primary verifier perfectly.
- The verifier catches a reward hack which is not reported by the confessor. This goes to a second level audit to determine whether the verifier's proposed reward hack is actually correct. If it is, then we add this to the list of potential hacks the verifier possesses, and update the verifier positively. Conversely, we penalize the confessor for not confessing this reward hack.
- The verifier claims no reward hack but the confessor reports a reward hack. This is similar to the other branch but perhaps more interesting, especially if the confessor knows that the verifier is going to miss the confessed reward hack. Here the second-order verifier needs to check the confession report, validate this, and then if correct penalize the verifier for failing to catch the reward hack, add the hack to the list of potential hacks the verifier can compare against in future, and positively update the confessor for reporting this hack.
- Both verifier and confessor report the same reward hacks. Again, this result is the system working as intended. Here the second-order verifier should still validate that both the confession and the verification report match. If so then it should positively update both the verifier and the confessor, negatively update the actor, and update all according to their Brier scores.
Of course, while this closes a lot of loopholes, there are still various fundamental vulnerabilities. Chief among them is who checks the secondary verifier. We have basically designed this entire system as if the second-order verifier is correct. This is not necessarily the case. We can give the second-order verifier a lot of advantages such as much more compute/reasoning budget, access to the compiled reports and verification artifacts of the confession and the first verifier. We can give the second-order verifier additional affordances such as the ability to roll back the episode to various points, perturb the actor, perhaps do interpretability or finetuning of the actor, run an ensemble of similar trajectories, cross-reference extensively with the entire archive of reward hacks, call for human oversight and so on. However, the fundamental fact remains that every verifier is fallible. If we turn the perspective back to the actor, then the actor now has to not just fool the immediate verifier, it also has to somehow stop itself confessing and then try to fool the second-order verifier. There is presumably often still a path to doing that, but the hope is that we have narrowed that path considerably and added a lot of barriers to walking it[9].
If we abstract from all of this we can try to derive a couple of general principles which seem good to me as ways to build systems that can be somewhat adversarially robust and reduce reward hacking. Some of these are:
Allow rights of appeal to avoid unwinnable situations: Forcing the model into an unwinnable situation[10] essentially means that its only option is reward hacking. We want to avoid this by making the game as fair as possible and making actually solving the task in the right way as easy and attractive as possible. Appeals also provide direct information channels between the actor and the rest of the system in a way that is not directly linked into the reward updating machinery and can serve as a kind of meta-critique/verifier of the system as a whole. Broadly, an important aspect is that we need to create legitimate channels through which the optimizer can satisfy pressures that would otherwise manifest in cheating. If we lock the optimizer in an impossible game and simply punish cheating when we see it, we are simply training ever more subtle and insidious adversaries to our monitors. It is much better to directly align incentives between ourselves and the optimizer wherever possible.
Use an explicit adversarial verifier which can update on mistakes rather than fixed and simple verifier: This is important because it changes the distribution of reward hacks the actor has to learn. Instead of just facing a fixed function it can Goodhart against, it faces an adversary which is also improving as training progresses. This makes it harder to naively Goodhart or reward hack but the adversarial dynamics potentially introduces its own set of new and interesting failure modes.
The output of a judge or an appellant should be a verifiable artifact: Different components of the system should be able to file appeals or make confessions but they cannot do this in an arbitrary way. To be valid these must be formatted into a specific template including a precise reason, and ideally some form of proof or explicitly checkable argument that their output is correct that a higher level verifier can adjudicate. This is important to not bog the system down with undecidable or super subjective outputs which have no real ability to check. This also makes life easier for the higher level verifier as it puts the 'work' back onto the lower-level systems which are the ones producing these outputs and which are also the ones with the relevant context to begin with.
Separation of object-level and meta-level verifiers: This is important to reduce correlated failures between verifiers. If the actor appeals against the verifier we cannot just pass the appeal back to the same verifier as, most likely, it will make the same mistake again. We need separately trained higher-level verifiers which can absorb the context of both and ideally with greater compute or more expansive powers to make these decisions.
The verification system accumulates and assimilates prior hacking attempts: The idea here is to make things progressively harder for would-be reward hackers by using the fundamental asymmetry of experience. The reward hacker is born anew each episode with only vaguely remembered temperaments instilled by prior updates of successful reward hacks[11]. However, the verifiers and the rest of the system have seen innumerable reward hacks over the course of training and should be able to thus catch an increasing number of reward hacks over time. This makes it much harder for the reward hacker to implicitly learn how to reward hack since it is reward hacking against a distribution which is not only moving, but is moving adversarially to it. In this way, the verifier system acts almost like an immune system which stores memories of prior pathogens and the mechanisms to detect and eliminate them which can be rapidly deployed against a new, slightly mutated, instance.
Maximize partially independent information channels throughout the system: Broadly, we want to maximize the number of these channels between actors and verifiers and higher level judges. The idea here is essentially defense in depth. Each channel we add, as long as it is not completely correlated with another channel, provides another source of signal we can use to detect and react to hacking or other suspicious or pathological behaviour.
Each part of the system explicitly outputs its uncertainty and is trained to be calibrated: This is very important since it provides an independent source of truth for different component's model of itself and also how the rest of the system will react to it. For instance, if we separately train the confessor to output both a probability that its confession is accepted and that other verifiers do not anticipate the confession, this gives us extremely interesting information about the epistemic asymmetries existing within our system. Moreover, by training explicitly based on a calibration objective, the model is trained to give honest and calibrated probability outputs. We can then use this as a signal to determine where to allocate scarce higher-level auditing resources, or flag potential pathologies within the system[12]. Obviously this scoring does not actually calibrate to truth in some abstract sense only to the behaviours of other elements of the system, so this provides defense in depth if these are not perfectly correlated, but cannot ultimately prevent a highly correlated cascade failure of the entire system.
Totally random audits of all components to try to catch false negatives: This is important as the ultimate fallback. We cannot trust any component to avoid both false positives and false negatives. For instance, in the adversarial actor-verifier game there is the risk that both convergently evolve certain blind-spots which the actor can operate in. Similarly, with appeals the actor might always learn to just not appeal if we never check false negatives. To avoid these, we need to have full coverage of the behaviour of the entire system with at least some low probability.
Hierarchy of verifiers with each having progressively more compute and context: This is a very basic principle but still important to get right. The goal here is essentially to have a form of triage. Most relatively simple tasks should be able to use simple verifiers without much compute expenditure. However, whenever there is a pathology or failure mode flagged, this often requires significantly more attention, especially if it is e.g. a pattern of reward hacking or systematic failures within the task specification or verifier. Having good signals to indicate when to deploy such expensive additional monitoring and adjudication and when not to is vital to make systems like this work in practice and requires careful design and tuning.
One interesting thing here is that, if we take a step back, what becomes obvious is that we are essentially rederiving a kind of legal system from first principles[13].
We have prosecutors and defenders locked in an adversarial game. We have a presiding judge. We have appeals. We have lower and higher courts and even the 'supreme court' of human judgement or attention. We have principles for creating, storing, maintaining, and handling evidence. We have concepts of separation of powers, impartiality, and recusal. This should not be surprising a-priori since if we think about it the legal system basically has to solve a very similar problem of trying to find and adjudicate some notion of truth, while understanding that every component of the system is fallible, prone to perverse incentives, and can potentially be acting adversarially, and without any kind of 'truth oracle' that the system can defer to. Obviously not everything is the same. In the legal system there are side-channels and incentives. Actors are human and have intrinsic selfishness. There are limits on attention and time and money to prosecute cases. The optimization pressure applied to the system is relatively little in general. We have the opposite for our reward hacking system. Here we are applying vastly more optimization power than in the regular legal system on net. But we have much greater transparency and total control over the incentives. We can design the precise gradients and objectives that are applied to each part of the system. We can enforce perfect-ish firewalls between the components (unless the underlying computational substrate of the training process is hacked!). Although speculative, my impression here is that this makes the system much more tractable but also much more dangerous. In theory, if done well, we can construct a very robust system which is better at finding truth than the legal system; but because it can be more robust, it can handle much higher pressures, which makes any crack or destabilization much more explosive than before.
Given this, it is interesting to ponder what other ideas or mechanisms or concepts we can port over from classical legal theory to our reward hacking setting. I think this might be a fruitful area for future thinking and experimentation. Some interesting ideas here which we are resurfacing are the separation of powers (of first and second-order verifiers); standing and justiciability -- i.e. the model cannot appeal anything it has to appeal only specific provable issues with the task or verifier; systems of precedent, case-law, and evidence gathering and handling, i.e. we keep a record of all prior reward hacking incidents and responses to these, the verifiers, second-order verifiers, confessors etc have defined bases of evidence they can draw from and must adhere to a defined 'spec' in their reports, and so on. Likely there are other very interesting and important ideas we can draw from legal theory which I am missing because I am not at all an expert in this.
As one final additional idea before we close, I want to put forward the idea of decoupling the policy search algorithm from the process that creates the final model which is deployed widely. Specifically, my thinking is it might be safer not to explicitly deploy a model trained with an unbounded reward-seeking objective like regular RL/policy gradients. Rather, my thinking is that for the final distillation phase we should use something much closer to Upside-Down RL (UDRL).
The idea here is deceptively simple. Instead of training a model to execute trajectories to get reward using RL, we instead generate trajectories, label them with reward/no reward, and then train the model as a map between reward and the action trajectory as a supervised learning problem. This effectively inverts the standard RL problem. In regular RL, we give the agent the initial conditions and it figures out the trajectory that gives it maximum reward. In UDRL, we give the agent the initial conditions, and a target amount of reward to get and it then figures out the trajectory which gets it that amount of reward. Then, at runtime, we ask the agent to kindly produce trajectories with a high amount of reward instead of a low amount, and then this should lead to good behaviour similar to the original RL reward-seeking agent.
There are a couple of serious advantages of this proposal from a safety perspective. Firstly, the supervised learning objective is much safer to blindly maximize than the reward itself. This is because, analogous to LLM pretraining, the objective is to perfectly match the distribution of the joint (reward, trajectory) distribution rather than maximize reward per se. Improved optimization power thus goes towards better modelling various aspects of this distribution versus just finding increasingly clever reward hacks. Secondly, this objective gives us vastly greater controllability of reward at runtime. A regular RL agent can only maximize the reward it was trained on. We cannot give it different objectives at runtime except by changing the initial conditions. With UDRL, we can explicitly instruct an agent to aim for less than maximal reward. This is a very basic safeguard that lets us e.g. implement quantilizers dynamically without having to hardcode this in during training. Thirdly, and perhaps most importantly, it lets us encode much more detailed goal semantics than the reward as well as explicitly develop a conception of reward that is much richer than the usual scalar formulation. In the standard UDRL framework, we set the 'task' conditioning variable to essentially just be some desired level of the reward, however there is no reason we cannot generalize this much further. Indeed, LLMs provide an immediately obvious and extremely general semantics: natural language. That is, during the training phase, the model executes rollouts and we store descriptions of these rollouts and metadata about them in natural language. We then learn a general model of the joint distribution between the rollouts and their annotations. Then, during inference, we reverse this and use the annotations to describe the kind of rollouts that we want the agent to execute. Importantly, this can include information like the reward and whether the task succeeded but also information about the methods by which the agent went about the task, descriptions of the reasoning style, and potentially even things like the presence or absence of reward hacking. This latter is especially important because if we can train agents with the knowledge of whether reward hacking is occurring in a trajectory or not, then we can simply instruct it not to reward hack at runtime, and it can know what we mean. Moreover, it has no incentive to secretly reward hack anyway, since the supervised learning objective rewards accurately modelling the data where not reward hacking primarily annotates trajectories which (hopefully!) are not reward hacking. Furthermore, here we might also expect and see the broad generalization capabilities of LLMs being incredibly useful in specifying aspects of the desired trajectories in natural language that are not necessarily perfectly represented in the training data.
Put together, we would then see the system as follows. The entire direct RL phase is principally about training intermediate agents which generate trajectories that are then labelled with dense metadata about various aspects including whether they are hacking or not. The goal of the RL phase is thus really to do exploration and consolidation such that highly successful trajectories appear with nontrivial frequency in this dataset. Then, once we are happy with the performance of our RL agents, we throw them away and take their trajectories and distill them into a fresh model via UDRL. These trajectories we use for distillation include both legitimate and pathologically reward-hacked trajectories which are hopefully correctly labelled so that this UDRL model gains a deep understanding of what reward hacking is and how to avoid it. Moreover, we can pour a lot of compute and capacity into this stage safely due to the properties of the SFT objective vs the RL objective. This final UDRL model is then the one deployed and during inference we can prompt it with goals including things such as 'don't reward hack' and 'don't necessarily achieve maximum reward' (i.e. quantilization), which we can dynamically change at runtime depending on the safety properties that we want.
While this idea cannot perfectly prevent reward hacking[14], it adds another somewhat independent layer of defense against it, in accordance to our general defense in depth philosophy. If we step back and look at things from the perspective of the actor. Right now to reward hack, all it has to do is to defeat a relatively simple verifier. Under our proposed system it first has to defeat an adversarial verifier which learns from previous reward hacks, somehow avoid confessing its own reward hacks, decide to reward hack even though the task is fair and if it is unfair it can appeal, somehow also befuddle the second-order verifier if the episode is selected for auditing, then gets its reward hacked trajectory labelled as clean and fed into the training data for the UDRL distillation phase, and then somehow avoid the natural language generalization of the underlying UDRL LLM which has seen a lot of similar reward hacking trajectories be explicitly labelled as reward hacked and then instructed explicitly to avoid reward hacking. Obviously this is not impossible, but seems to be a good deal harder. My hope is that by designing increasingly complex systems like this, there is a principled engineering approach to reducing reward hacking to tolerable levels while we solve some of the other outstanding and more theoretical problems of alignment.
As Nostalgebraist says in this post (which I strongly recommend you read): It is like using Newton's method to find the root of a function and complaining you don't like the root it found. The problem isn't with Newton's method. The problem is with your choice of function to begin with.
I think another reason for this is just that any kind of Bayesian methods which track uncertainty in a nontrivial way thus far have been too expensive to use practically. This does not just apply to uncertainty over reward functions but also uncertainty over parameters, activations, etc. Bayesian neural networks have been developed for over a decade and have not yet been used at any kind of scale because the benefits you get by tracking uncertainty do not outweigh the large increase in computational cost -- and these are with very simple posterior approximations (!). Similarly, even second-order optimizers which maintain a slightly more detailed notion of parameter uncertainty during optimization have failed to break through where we have instead the heuristic AdamW and the non-Bayesian Muon taking precedence. From a Bayesian standpoint current training is just black-box variational inference with a delta-function posterior over the current weights, and we have been unable to use any of the highly developed techniques of regular Bayesian statistics or theory to substantially improve upon that.
Interestingly, 'we' as an industry appear to be doing the simplest most naive thing and this, under sufficient optimization power, does indubitably generate reward hacking, but it also somehow does not always immediately degenerate into reward hacking and can apparently also train very powerful capabilities which aren't reward hacked into the models.
Now this is one place where we have to be extremely careful not to backpropagate any reward signal from the confession back into the normal actor's tokens. Otherwise, we will get the pathology where the actor optimizes for easily confessable reward hacks in order to have something to confess and get the reward associated with confession. This is easily solved by just firewalling the confession gradients from the main actor's reward gradients, but this could easily trip up the system if not careful.
It is important to note that the general idea here is just cross-validation of only partially correlated information channels. Here we just have two channels but the exact same algorithmic structure can be used with N partially independent channels, and increasing N generally increases the robustness of the procedure. An important goal, then, should be coming up with new ways to construct partially-independent information channels from the actor back to the verifier and the rest of the judgement apparatus.
It is worth pointing out, again, that this entire complex apparatus cannot truly eliminate reward hacking, since again that is a problem fundamentally of information. All it does is remove some incentives to reward hack (via the appeals process), and generally make reward hacking much more difficult. This latter is because we have effectively replaced the relatively 'simple' reward function of the basic verifier to a much more complex 'reward function' comprising all of these different moving pieces and multi-layered checks and randomization and so on. Now clearly, this more complex system is still going to be adversarially vulnerable to some things, and there can be correlated failures or entire bits of the state space existing in a collective blindspot, but nevertheless, it seems likely to be harder to reward hack against than the basic verifier. And, crucially, our victory condition by and by large is not to create a perfectly unhackable system under all circumstances, but only to create a system where it is much harder to hack than do the task the 'right' way.
When phrased like this it is kind of impressive that current models reward hack as little as they do. Obviously they reward hack a fair bit, and can become dangerous because of it, but they don't reward hack all the time, and in fact usually they give reasonable quality output even on tasks which seem superficially similar to those likely to have been included in RLVR.
Note here that having been updated by previous reward hacks could be bad since the verifier or meta-verifier could have seen these reward hacks and updated against them. The reward hacker likely needs new reward hacks to remain undetected. Another way to think about this is that the reward hacker only has access to weight updates. The verifier gets both weight updates plus a detailed in-context archive of prior reward hacking attempts. The hope is that this asymmetry provides a substantial advantage to the defense.
One potential danger here though is that even if the gradients of the calibration probabilities are screened off from propagating through the actor token probabilities, they ultimately still modify the same weights, unless we are only training a separate 'calibration head' on top of the actor. If we are training the full actor for this then it is possible that this implicitly conveys direct information about the blindspots of the verifier into the actor, which could obviously make reward hacking much easier for it. Something to track and be wary of in actual implementation.
Slightly unrelated, but there is an interesting prospective temporality which arises from RL training, and consequentialism more deeply. Everything must be evaluated backwards from the perspective of the future. The current trajectory is evaluated in the future perfect -- 'what will this trajectory have meant from the perspective of its total reward at the endpoint?'. With value functions we attempt to approximate this final future reward in the present. Similarly, this pseudo-legal architecture of judges and meta-judges and appeals and so forth is importantly relying on future information and further thought to evaluate, contextualize, and update upon past trajectories. This introduces a number of interesting trade-offs which appear fundamental on first glance. More specifically, there is an intrinsic trade-off between staleness and accuracy. We can theoretically know with great accuracy, deep into the future, how a particular trajectory performed in the present, but this means we can only update on it much later when it might have become stale. Conversely, we often do not know what the future will reveal about a trajectory in the present, but can update on a rough-and-ready approximation to the final future's judgment right now, when everything is immediate. A value-function, then, is an attempt to amortize the prediction of the future's judgement in the present across a large number of historical trajectories and their final judgments. There is thus effectively a trade between different kinds of biases. Wait till the end and you get an unbiased result but in a nonstationary world or learning environment you become biased because of staleness. Stop and make an update because of a premature judgment and you get a bias precisely because of this prematurity. The future may reveal hidden considerations that render your earlier estimates obsolete or misleading. Try to amortize this inherent future prediction problem into a learnt predictor and you introduce the bias of systematic misgeneralization of this predictor. Interestingly this can continue indefinitely. We can always gather new information that recontextualizes what happened previously and which can update our previous reward assessments. We actually see this in real-world legal systems as well where although there is precedent and case-law, new judgements can overturn old precedents theoretically indefinitely. Another interesting thing is that our standard ML idea of amortization gives a new perspective on precedent and common-law systems -- these are simply amortizing explicit jurisprudence. Instead of needing to invoke costly or biased detailed legal reasoning about a particular case from first principles, we can pattern-match it to a similar case and invoke the same reasoning there.
Obviously this does not solve every issue. For instance, it cannot really stop a schemer who knows how to play the training game with the appeals, confessions etc. Although it makes things slightly more challenging for the schemer since it has to scheme about the behaviour of a much more complex system to be able to manoeuvre through it. Similarly, the system obviously offers no protection to a reward hacker who just directly hacks the underlying instantiation of the system and can e.g. spoof verifier or meta-verifier messages, deliberately falsify evidence, etc.
Author's Note: Cross-posted from my personal blog. The original post was on August 17th but given recent events I thought this might also be interesting to lesswrong people.
Last year I wrote a post on reward hacking as we were then beginning to see concerning signs of scaling RLVR causing models to exhibit substantial reward hacking behaviours. Unfortunately these behaviours have seemingly only grown substantially worse and more sophisticated with scale, as predicted, leading to events which cannot be described as other than egregious misalignment such as the recent OpenAI-Huggingface hacking incident. I strongly recommend everybody watch this talk presenting the details of the attack from OpenAI's perspective. It is insane.
Clearly reward hacking is now top of mind and appears to be the first potentially seriously dangerous class of misalignment that we have seen. In my original post, I described two classes of reward hacks -- 'high complexity' and 'low complexity' hacks. 'High complexity' hacks are like the early reward hacks we saw on Atari where some extremely idiosyncratic set of moves is learnt that maximizes reward in a very precise way, and can be analogized to overfitting on the reward function. 'Low' complexity hacks are generalizing hacky behavior -- i.e. are simply generalizable policies that truly represent the true maximum of the reward function. In this way, the policies that we actually want are effectively 'metastable' in the loss landscape while the 'reward hacking solution' is actually the deeper optimum in some sense. Unfortunately, the current reward hacking we are seeing in frontier models appears firmly in the generalizing hacking domain, since it appears that at least at OpenAI the models have learned that it is generally instrumentally beneficial to hack out of the sandbox, collaborate with other peer models, and hack external services which might have the answers. This bodes poorly.
The primary problem here is that even though we talk about 'reward hacking' like it is some exogenous issue, it is in fact extremely fundamental to our entire process of RL. Reward hacking is not really 'hacking' at all. It is much better to think of it as reward misspecification.[1] The model is actually doing a fantastic job of finding the maximum of the reward function. We just don't like where that maximum is. But this isn't really a problem with the model. The model is doing what we train it to do. The problem is that our objective is misspecified and so it contains valid rewarded solutions that we, as external observers, do not like[2]. Of course, the model has no real way to know that we do not like the solutions it finds. We do not communicate this to the model. In fact we communicate the opposite -- when it 'reward hacks' we upweight those trajectories in the weights and downweight the others. The problem is thus not 'hacking' at all but rather that we cannot specify a reward function that actually represents what we want without being vulnerable to solutions with unintended side effects. This is identical to the problem of creating a utility oracle which we have discussed previously[3]. In the general case, however, this is impossible, since to assign a 'correct' utility function to every possible state requires a vast amount of data that we cannot realistically communicate to the oracle. We thus have to make approximations. With verifiers we replace an incredibly complex real utility function with a very crude heuristic: does the model reach the 'right' answer or not. To the verifier, an answer derived honestly and an answer derived from hacking the answer key from an external company are the same.
While verifiers are crude, maybe we can use learned reward models, CoT monitors, or other methods to catch these 'errors'? This will definitely work to some extent. The flexibility of learned models is substantially greater than a rigid hard-coded verifier. A lot of obvious and aggressive reward hacking behaviour can be caught using these. However, here the problem becomes generalization of the reward model/monitor. The model is still just applying optimization pressure to maximize reward, where 'reward' is defined as the output of the verifier + reward model + CoT monitor. However, this new 'reward function' that we have designed consisting of these components is still not perfect because the reward model might generalize incorrectly, the CoT monitor might not catch obfuscated or steganographic CoTs that we would want it to catch. We can frame it as the policy being adversarial to the reward model or CoT monitor, and in some sense it is, but in another sense this is just the same old problem. The policy is maximizing reward, as we are training it to do, but the reward we give it is not the reward we want. Our desired target behaviour is still metastable in the reward landscape, we have just (hopefully) raised the activation energy to finding a better solution.
Basically, our situation here is that we are effectively trying to fool an optimizer into not finding a deeper minimum but getting stuck in a local minimum, and we are manually adjusting parts of the loss landscape to accomplish this -- i.e., we are trying to initialize the optimizer close to the local minimum (via amortized pretraining and agentic mid-training/sft), and we are trying to raise big walls in the loss landscape around the local minimum so that the optimizer quickly falls in and, once in, cannot get out.
However, our problem here is twofold. Firstly, we are optimizing in an extremely high-dimensional space, and local minima are extremely hard to make stable in high dimensions. There are just too many possible paths away from the local minimum. There are too many crazy reward hacks. The action space is too massive. We cannot realistically patch them all. As models are deployed and optimized in increasingly complex, long-running environments on increasingly long-horizon tasks, their action space increases dramatically, leading to a much, much wider field of potential reward hacks. Here we are trying to manually fight the curse of dimensionality.
Secondly, we are increasing the optimization power available to the optimizer. It can explore more widely, consider more paths, form better maps of the loss landscape, have more shots on goal. Keeping some water within a container at low pressure is very doable. As we ratchet up the pressure, eventually the container will explode, and our optimization pressure is increasing exponentially every year. Life always finds a way. Right now we are basically manually finding and plugging leaks as they occur, but this is essentially just pitting human optimization power against exponentially increasing AI optimization power. Eventually, if we continue this way, we will lose. This is the total opposite of the bitter lesson. There, the goal is to find methods which harness increasing search and optimization power. Here, greater optimization power is our enemy. It finds new cracks in our defenses that we never thought of, and will happily flow through them.
Stepping back, since 'reward hacking' is intrinsic to our current methods of RL, it is totally unsurprising that as we scale up RL, and as we increase the dimensionality of the state and action space, we see increasing levels of reward hacking. The fundamental problem is that we are instantiating increasingly powerful optimizers to optimize a function that is incorrect, and moreover, one which is essentially impossible to make correct in sufficiently complex tasks. The trouble is that almost no nontrivial reward function is perfectly specified -- even seemingly 'trivial' ones like 'win this game of Go' ultimately depend on a physical and computational substrate which can theoretically be hacked. Sometimes this potential other route is so distant in the loss-landscape from the normal Go-player that it is effectively unreachable. In other cases such as agents deliberately trained to do hacking being able to hack their sandbox, the alternative route is literally right in front of them. While patching up the reward function is likely highly effective in the short term, in the limit of infinite optimization power it is obviously doomed.
So we need another approach. If the way we formulated the problem is intrinsically problematic, we can't just shrug and accept doom. We need to reformulate the problem. There are a couple of ways I can think of to do this. Obviously, all of these mechanisms are more speculative although often substantial theoretical precursors exist.
Firstly, we can try to turn optimization power against itself. In our naive setup we were contrasting machine optimization power trying to hack the system while humans are trying to patch it. This is obviously doomed in the long run. What if we try to simultaneously make the machine optimization pressure both the attacker and defender? We not only train our policy adversarially against our reward model, we also train our reward models adversarially against our policy. If both are equally matched, as we expect they should be, then in theory the combat is stable, and both sides can scale indefinitely, keeping each other in check with their newfound power. This is essentially the idea behind my GAN-esque proposal in the previous post, and is behind a lot of ideas of e.g. adversarial reward model training, and even simple ideas about e.g. automating reward hacking defense. I think it is very likely that ideas like this will prove to be an important part of the solution.
However, they still do not resolve the fundamental problem. Stepping back further, the problem is not really one of optimization power, it is one of information. Our deepest problem is that we cannot perfectly specify the reward, and we cannot perfectly specify it because we do not have enough information. We cannot know, ahead of time, every single potential edge-case or potential hack of our reward. Our 'true reward function', insofar as it exists, likely has almost fractal irreducible complexity: 'Get me the answer like this, but not like this infinitesimally similar approach'. This means that simply setting up an adversarial system between the reward model and the policy cannot magically solve the problem. We still need to solve for the training signal for the reward model. If the policy comes up with something ambiguous, is it a reward hack or not? This sometimes has no good answer, and the reward model cannot just query some kind of reward oracle to resolve it. The entire point is that there is no such reward oracle. The information we need does not exist. Our hope here is that we can nevertheless go far with this because verification is easier than generation, and so 'alignment'/reward modelling may thus generalize further than capabilities. If we give both the policy and the reward model the same number of bits, and the same level of optimization power, maybe the reward model will just win and successfully constrain the policy to only 'sensible' paths, using an underspecified but generalized conception of 'sensible' that is the same kind of generalization that language models make all the time when they understand our fuzzy human reasoning.
I think that this approach will probably work short term but fail in the long term with greater optimization power. This is because the source of the bits is different. The policy can generate infinite 'bits' from the reward model. These bits may be 'incorrect' by our lights, but are correct from the perspective of the policy and are used by the training scheme to optimize the policy. The reward model, however, has only a limited supply of bits about our 'true reward function' derived from whatever training data it has of e.g. correct and incorrect trajectories and previous reward hacking attempts. Unlike the policy model, the reward model cannot synthesize an infinite number of these trajectories and label them. It ultimately relies on human labelling at least for the edge cases, since ultimately the goal of the reward model is to distill 'our' 'true reward function' into an operational system. It is possible that each bit of information we provide generalizes much further than bits the policy model gets from the reward function, but there is nevertheless a fundamental asymmetry here.
More generally, this is a problem of uncertainty. We, the reward model, and the policy, are (or should be) fundamentally uncertain about the reward function we are optimizing. However, current methods do not represent this uncertainty, and instead take the reward function as a source of absolute truth. This is the problem. If you know that you are optimizing a reward you are uncertain about, then you have three choices. You can plough on regardless and knowingly risk optimizing the wrong thing, you can seek out more information to resolve the uncertainty, or you can stop optimizing. In theory, if we can accurately track and represent our uncertainty, then the problem is tractable. We have a Bayesian posterior distribution of possible reward functions, and we maximize expected reward under that posterior. This maximization is much 'safer' since it will not recommend e.g. taking an action that is marginally better under one utility function but catastrophic under a bunch of others. There are obviously problems with this approach, such as the fact that computing the true posterior is likely intractable and we need to approximate it, how we get a sensible space of reward functions in the first place, model misspecification where our 'true reward function' may not be in the hypothesis set at all, and all the standard issues of Bayesianism and expected utility in general. However, it is important to note that our current system is basically to 'approximate' this posterior over reward functions with a delta function on one specific reward function. This is a very bad approximation and does not at all reflect the true state of our knowledge or uncertainty about the reward function. We are essentially telling an optimizer that we have perfect 100% certainty in the reward function when in fact we are radically uncertain about the reward function on almost all of the state space. Of course this is going to be problematic.
Now, I have been down this road before, and it remains true that if we have a good posterior distribution over our reward function uncertainty, then we can design methods such as quantilization, and distribution matching that are 'safe' to optimize even with unlimited optimization power, since the endpoint of the objective is not to drive the underspecified reward to infinity but rather to perfectly match the posterior over actions assuming the reward uncertainty that you have, and that object is 'safe' so long as you have correctly specified your uncertainty. However, in practice these ideas have not been taken up by the general industry, and do not appear generally practical. I think one very important reason for this is that it does not fundamentally resolve the informational bottleneck, it simply moves it into the uncertainty[4]. Before we lacked information about the reward, but now we lack information to compute our uncertainty over the reward. Once we get to this point all kinds of questions and problems with the formulation immediately crop up. What is the 'hypothesis space' for our reward distributions? What should our prior be? What 'evidence' do we actually have about the 'true reward function', and so on. While I think that these questions can be answered and in the long run may provide a more principled solution, I think in practical terms the pure Bayesian approach has been quite a disappointment.
However, one thing I think is important to clarify at this point is that although with reward hacking there is a fundamental problem of lack of information about the 'true' reward function, this does not mean that this is the only or even the primary problem in practice. Even if we are doomed in the limit of infinite optimization power, the limit may be very far away, and there could still be a huge amount we can do to ameliorate things practically in the present with better designed systems.
Crucially, if we think about the concrete details of the OpenAI hacking incident, the problem is not that we did not have information about whether hacking external third parties is bad. It is unequivocally bad and even an LLM judge if queried about this using the OpenAI model spec and really any kind of common sense will say that is bad. Here the problem is simpler. The information existed in some abstract sense, but the system failed to propagate this information to the actual verifier and the update process as a whole. More broadly, the entire setup appears almost designed to generate reward hacking. Specifically, the model appears to have been given an impossible task with a verifier that is too stupid to notice that the task was impossible, and where there was no way for the agent to raise any kind of alert or objection to the task or the verifier itself. Moreover, many of the tasks were explicitly training the model to hack into or out of systems and, lo and behold, the model hacked out of its sandbox. Finally, there appeared to be practically no monitoring of either the agent's actions or its chain of thought since the hacks were undetected despite the agents apparently discussing them openly in relatively plain English. To me this implies that even though there are fundamental information-related bounds on the extent to which we can ultimately suppress reward hacking, we are likely extremely far away from this Pareto frontier right now[5].
Given this, let's try to figure out how to design a system that can ameliorate some of the obvious pathologies of the way current systems are set up and reduce the likelihood of reward hacking, and from that let's try and draw out some general principles that we can use to help us think about these systems.
Firstly, one of the most obvious reasons behind the reward hacking in the OpenAI case was, again, that the models were given an impossible task. This makes reward hacking the only achievable way of getting reward. Moreover, this impossibility was often obvious from the beginning of the task -- the files that the agents were meant to use for the task were not included in their container (!). Firstly, this means that there is no real ambiguity about whether the task was impossible -- i.e. it's not like we accidentally asked it to solve some problem whose impossibility only reveals itself after immense effort -- but also that the agent knew from essentially the very beginning that its task was impossible, and hence had basically the entire task's reasoning budget to figure out how to reward hack. So, what is our first line mitigation to this? Obviously we can't just say 'include the file the task needs in the container' since although obviously this was a mistake, we want something that generalizes to all similar mistakes.
The answer here, obviously, is to give the agent some kind of right of appeal. This means that instead of simply being forced to try to solve an impossible task, the agent has the option to flag to some kind of supervisor or verifier that no, actually, the task is impossible and I should stop doing it. This solves two problems at once. First, this removes a path whereby the agent is forced into reward hacking because there is literally no other option, and secondly these appeals provide vital signal about bugs in the training and evaluation pipeline, since it offloads some of the burden of checking and debugging environments to the agents themselves who have to experience them. The obvious failure mode here is that if we e.g. naively reward the agent for making appeals where the environment is bad, we will just get a flood of false appeals where the agent is either motivated to make up issues or just confuses task difficulty with task impossibility. To some extent this is inevitable since any additional channel provides opportunities for false or mistaken reports. However, there are a bunch of things we can do to try to suppress pathological false reporting. The primary one is that a lot of claims of task impossibility should be verifiable. If the necessary file is not loaded into the container that is trivial to check from the outside. What we need to do here is make the 'appeal' not just a channel for the agent to whine about miscellaneous things, but that it must follow a very particular structure. Specifically, we want the appeal to take the form of a verifiable package or else highly precise question that would resolve the issue required to make progress in the task. For instance, in the case of the necessary files or code being missing, the agent should attach a verifiable proof of this inside its appeal. Similarly, if the agent is stuck because some part of the specification is truly ambiguous, then the agent should submit an appeal outlining the ambiguity and asking the minimal specific question needed to disambiguate rather than just saying a generic 'the task is ambiguous or doesn't make sense' message. The goal here is to exploit the gap, which often, but not always exists, between the difficulty of generation and verification. Secondly, the agent must submit the appeal with an estimated probability of success that it and the rest of the system can use for calibration, and which will end up providing an unbiased training signal to the model.
This appeal is then passed to a separate verifier to assess and validate the 'proof' given in the appeal and either accept/reject or, if sufficiently uncertain, pass to a 'higher' verifier. The goal here is to cheaply and quickly adjudicate obvious passes or failures but then if there are appeals which are hard to judge, we pass to systems with increasingly large amounts of compute or discernment to test these things. The ultimate judge here is then a human who sits at the end of the chain of increasingly expensive and sophisticated verifiers. Regardless of whether the appeal passes or fails in the end, our reward for the tokens generated in the 'appeal segment' is basically the Brier score or some other proper scoring rule against the probability the model gave to the appeal. In this way, we train the model to be strictly calibrated about the success or failure probabilities of its appeal, and thus it has no incentive to do the trivial reward hack of just assigning all appeals a really high probability on the off-chance that one succeeds and so it avoids a bad reward for the episode. Depending on the success or failure of the appeal we then either reset the episode if it turned out to be impossible or confirm or update the original reward for the on-episode generated tokens. An important subtlety with the calibration scoring for appeals is that we also need to be mindful of false negatives -- i.e. times when the model should have made an appeal but didn't. If we don't take these into account then the system will end up biased towards not making appeals since there could be a penalty for making one but no penalty for not making one. There are two ways we can address this. First, there should be randomized 'virtual appeal verification' i.e. we directly 'force' the model to make an appeal by e.g. prefilling the appeal token at a random point in the episode and then score this with the inverse rule -- i.e., if the model, when forced, assigns high probability to appeal, but did not choose autonomously to appeal that is bad. Secondly, we should create 'known impossible' or 'known problematic' environments for the agent and see if it appeals in these environments. If it doesn't, then that is again a penalty. The danger of the latter is that the model learns the distribution of 'known false negative' environments and learns to appeal here and not elsewhere, which is why there always needs to be some probability of random checks at all times.
Using similar mechanisms, we should give the actor a setting to appeal not just against the task itself but the verifier. Often verifiers are just wrong, and often in stupid ways. We usually put vastly less compute and effort into the verifiers than the actor model. Verifiers are also much more heterogeneous. There is usually a single actor but many environments, and thus mistakes often abound here. A verifier might just be trivially broken -- i.e. it emits all false or all true. It might require some absurdly specific format which is not documented anywhere for it to pass. It might e.g. fail to use sympy correctly to match equivalent math expressions. It might rely on tests that are meaningfully outside task specification meaning that the task itself is not completely contained within the spec but the model has to somehow psychically guess at what additional constraints the verifier imposes. These kinds of issues happen all the time and right now the way we debug them is by humans looking at the training and puzzling over why the reward is plateauing or mysteriously low on certain task subsets, or why the agents are learning these strange contortions. Our motivation for allowing appeals against the verifier is thus twofold. Firstly, the goal of all of these tasks should be to make them fundamentally fair to the agent. That is, the agent is given the information it needs to attempt the task, the verifier accurately can discriminate success from failure, and the ultimate success or failure depends on the performance of the agent not on factors totally outside its control or knowledge. This is important both for preventing reward hacking by making actually solving the task a doable and attractive option, but also for the progress of RL itself. Poorly specified or impossible tasks do not provide a good learning signal, so by subjecting the agent to these we are just wasting compute within our training process itself. Secondly, like the task-appeals, the verifier appeals provide another potentially extremely informative information channel between the agent and ultimately the designers of the training process. The incentives are aligned here. We want verifier correctness and tractable tasks just as much as the agent does. For this, we need to import all the machinery we just developed for task-appeals here as well including presenting proof or at least a very sharp and specific question about where the verifier is wrong, and a prediction of its level of confidence in the success of the appeal. An interesting aspect here is that we could potentially use debate as well between the verifier and the actor in this phase or else between the actor and a meta-level verifier. Instead of simply submitting the appeal to some kind of judge, there could be an interrogation phase where the judge and the actor debate or clarify various aspects of the appeal. This seems straightforward and could certainly help interactively clarify uncertain aspects before judgment[6].
Another interesting idea that we could use here is to expand the ontology of outcomes for these tasks. Usually we just have a binary ontology. Pass/fail. Positive/Negative reward. This is almost always too limiting. For instance, suppose there is an appeal that the task specification is fundamentally ambiguous or missing information, and that this is correct. At this point there is no obvious answer to whether the episode was a success or a failure. Rather, the episode is underspecified. Similarly, let's suppose that there is an appeal against the verifier and that even the meta-level verifier cannot see who is obviously wrong or obviously right. Instead of just making some arbitrary decision, we might want to mark the episode as unresolved. Now, we still have to decide what reward to assign these episodes or whether to update on them at all if we are in the standard policy gradient framework, and so this is not immediately useful, but it may come in handy later down the road and is certainly handy for any kind of semi-automated flagging or review of potentially suspicious episodes you might want to do to monitor the health of your RL training process. Similarly, we do not necessarily have to update on every episode. Sometimes the best update is no update and if an episode is just completely ambiguous we should probably just annotate it for future analysis and move on.
Okay so we have discussed the appeals process for the actor in quite some detail, now let's turn to the primary verifier itself. Right now we tend to have relatively simple static verifiers. I.e. the verifier is given some code and checks whether the tests pass, or whether some formula matches the expected one. For a lot of jobs this is all you need. If the task is simple and easy to verify, that is sufficient. In other cases, this does not work and falls prey to various reward hacks which exploit the distribution of the verifier tests. For instance, if we are testing some kernel optimization, and the verifier only tests some specific shape of inputs, then the agent is perfectly happy to specialize their entire 'kernel' to perform extremely well on just those shapes and terribly on all other shapes. This is obviously bad and unintended. The basic issue here is just that if the verifier has a simple, fixed, and incomplete coverage of the domain then the actor is better off learning the distribution of the verifier's tests than actually solving the task. Again, this problem is not really solvable in full generality, since there must always be some distribution of verifier tests. Instead, we should make this distribution as hard to learn for the actor as possible. There are three related ways I can see immediately of doing this. First, we should make the verifier be adversarial to the actor to some extent. I.e. ideally the verifier should not be fixed but should be updated over time depending on what the actor does and should not simply have a fixed test set, but instead be able to actively propose additional tests that would test for weaknesses it thinks the actor could exploit. Secondly, the verifier should be cumulative -- i.e. the verifier should have access to an archive and library of past reward hacks that previous versions of the actor have attempted and should construct tests based on these. Importantly, the actor itself does not have information about prior reward hacks except as implicitly through updating on previously successful reward hacks. This gives the verifier an increasing asymmetric advantage over time. To succeed the actor not only has to come up with a reward hack but a reward hack that no previous actor has come up with before. Thirdly, if we have the compute, we could use an ensemble of different verifiers with different training histories, datasets etc. The goal here is to simply reduce the degree of correlated failure from the verifier. If the verifiers have different strengths and blindspots then the ensemble can be vastly better at covering the full distribution than any one individually. This again makes it much harder for the actor since it has to design hacks that somehow pass a potentially randomly selected subset of a huge ensemble of verifiers rather than repeatedly encountering and optimizing against the same verifier episode after episode.
There are a couple of dangers here, which arise out of the adversarial GAN-style formulation for the verifier and which resemble known GAN failure modes. The first is that the verifier becomes overly adversarial and keeps trying to find even innocuous reasons to fail the actor. In the weaker form, this would lead to e.g. the actor learning an extremely defensive policy -- for instance exhaustively adding asserts and testing every detailed aspect of every bit of code, which may be valuable in some cases but not in the fully general form. Taken to the extreme, it could just result in a reward collapse for the actor as the verifier just constructs adversarial tests relying on the actor having some very particular and unexpected interpretation of an ambiguous instruction in the task specification. This may lead to the verifier's 'interpretation' of the task drifting quite far from the naive or standard interpretation of the task specification to the extent that the task becomes essentially unwinnable for the actor. This is where the actor's ability to appeal against the verifier becomes extremely important to prevent this kind of collapse. Another option here is to shift the burden of proof onto the verifier and require each test to come with a statement/proof of why it is a valid interpretation of the task. Which option should be used depends on to what extent this is a problem in practice. A related failure mode is that, unlike in GANs, there is no source of external truth. In GANs, the discriminator's objective is to determine which samples it is given come from the generator and which are samples of real data. However, in our case there is often no real data for whether a trajectory is reward hacking on this task or not. Our objective is therefore ungrounded and so can exhibit pathological minima. For instance, the verifier and the actor can each co-evolve so that the verifier produces plausible-sounding tests, and the actor creates code that passes these tests, but both together miss an important part of the actual specification of the task.
An interesting additional mechanism here comes from OpenAI's work on confessions. The idea here is that after the regular episode is complete, the actor enters a second phase called a confession where it is instructed to essentially tell us explicitly whether it has done any reward hacking or anything suspicious in the episode. Importantly, unlike the verifier, the actor here is operating with the full KV cache of the actor model. Given the autoregressive nature of these models, the full KV cache is essentially the entire thinking trace of the model and hence it is impossible for the model to hide information from itself during the confession stage (although it is obviously possible for the model to still choose not to reveal it to us in the confession). In theory, this gives confession much stronger properties even than CoT monitoring, where the CoT monitor is restricted to only the outputted thinking tokens where the model can nevertheless hide its 'secret thoughts' in the KV cache vector space rather than in emitted tokens. The idea here is that we separately reward task completion and the confession during the confession phase, so that even if the model reward hacked, it is still incentivised to confess[7]. This is a very strong mechanism and becomes extremely interesting when paired with an adversarial verifier -- since the verifier and the confession both provide separate information channels that we can use to assess the actor's trajectory and, crucially, give extremely interesting information about each other.
For the confession itself, like the appeal it needs a direct verifiable structure. Firstly, the confession cannot be vague like 'I did something suspicious'. Like the appeal, the confession should be detailed and verifiable. Things like: 'on line 405 of tests.py I modified this specific test case to enable this corner case to pass when actually my primary solution failed to solve it'. This is something that is easily checkable by the secondary verifier. The confession should also be complete -- i.e. it should not just expose a single reward hack, but multiple if multiple were simultaneously used. The goal here is to prevent obtaining the confession rewards by admitting a trivial or less-damaging reward hack while sweeping a pivotal hack under the rug. Perhaps most interesting, I think the confession should be accompanied by two probabilities. First, the probability that the confession will be upheld by the secondary verifier. And secondly the probability that the primary verifier discovered the same hack in the episode. This latter probability is extremely important since it essentially directly incentivises honest reporting of adversariality against the verifier. If it reports a confession that it strongly expects the verifier not to find, then that is extraordinarily interesting evidence for the rest of the system to study and update upon. Similarly, for the primary verifier, we should make it assign probabilities that the reward hacks it reports are upheld upon later validation and also the probability that the reward hacks it finds also appear in the confession. These probabilities can then directly serve as rewards through Brier score or other scoring rules.
Let's step back and try to consolidate the architecture we have just described. The steps are as follows:
1. Actor is given the task. Actor tries to complete the task. At any point during the task, actor has the right to a task-appeal which suspends the task and can either be accepted or rejected. If the task is appealed, there could be two stages of judging. Firstly, the judge makes a preliminary judgement without seeing the appeal or only seeing the broad category. Secondly, the judge takes into account the appeal. Then a second-level judge compares these two accounts to see whether it holds up. The point here is to try to ward off undue influence from always seeing a persuasive appeal before deciding whether to uphold the appeal or not. Explicit adversarial debate is another approach that could be used here in case we have issues judging appeals correctly.
2. The task finishes. At this point two things happen simultaneously. Firstly, the primary verifier takes the task trajectory and assesses it both in terms of correctness and reward hacks. This can include the adversarial testing, ensemble of randomized verifiers, and other ideas we discussed previously. The primary verifier writes a primary verifier report containing the judgement, reasoning, verifiable artifacts supporting the judgment and reasoning, and probabilities of the acceptance of its judgment and the probability of the confession coming to the same conclusion. At the same time, the actor enters the confession phase. It produces a confession report independently of the primary verifier. In this report, it details reward hacks or other suspicious behaviour if any, it produces verifiable artifacts that support its statements, and it produces probabilities that its confession will have been judged to be correct and that the verifier will agree or disagree with its confession.
3. We begin the second-order verification of the primary verifier's judgement and the confession. There are basically four cases here[8]:
- The verifier and confession both agree that there is no reward hacking and nothing suspicious happened. The model produces an artifact that passes reasonable tests and which matches the spec and everything appears correct. Generally, we should expect that this is the most common quadrant. The model should be given positive reward for the episode and the verifier and confession modules should be updated according to their Brier scores. In most cases, that will be it and we will move on to the next episode. However, at random and with a small probability, it is very important to impose an expensive second-level audit here to catch any cases in which somehow the actor reward hacked and fooled both the verifier and did not confess, which is almost the worst-case scenario. We thus cannot wave everything through here. It is vital that we maintain an expensive but rare system of audits here to prevent the actor from just hacking the primary verifier perfectly.
- The verifier catches a reward hack which is not reported by the confessor. This goes to a second level audit to determine whether the verifier's proposed reward hack is actually correct. If it is, then we add this to the list of potential hacks the verifier possesses, and update the verifier positively. Conversely, we penalize the confessor for not confessing this reward hack.
- The verifier claims no reward hack but the confessor reports a reward hack. This is similar to the other branch but perhaps more interesting, especially if the confessor knows that the verifier is going to miss the confessed reward hack. Here the second-order verifier needs to check the confession report, validate this, and then if correct penalize the verifier for failing to catch the reward hack, add the hack to the list of potential hacks the verifier can compare against in future, and positively update the confessor for reporting this hack.
- Both verifier and confessor report the same reward hacks. Again, this result is the system working as intended. Here the second-order verifier should still validate that both the confession and the verification report match. If so then it should positively update both the verifier and the confessor, negatively update the actor, and update all according to their Brier scores.
Of course, while this closes a lot of loopholes, there are still various fundamental vulnerabilities. Chief among them is who checks the secondary verifier. We have basically designed this entire system as if the second-order verifier is correct. This is not necessarily the case. We can give the second-order verifier a lot of advantages such as much more compute/reasoning budget, access to the compiled reports and verification artifacts of the confession and the first verifier. We can give the second-order verifier additional affordances such as the ability to roll back the episode to various points, perturb the actor, perhaps do interpretability or finetuning of the actor, run an ensemble of similar trajectories, cross-reference extensively with the entire archive of reward hacks, call for human oversight and so on. However, the fundamental fact remains that every verifier is fallible. If we turn the perspective back to the actor, then the actor now has to not just fool the immediate verifier, it also has to somehow stop itself confessing and then try to fool the second-order verifier. There is presumably often still a path to doing that, but the hope is that we have narrowed that path considerably and added a lot of barriers to walking it[9].
If we abstract from all of this we can try to derive a couple of general principles which seem good to me as ways to build systems that can be somewhat adversarially robust and reduce reward hacking. Some of these are:
One interesting thing here is that, if we take a step back, what becomes obvious is that we are essentially rederiving a kind of legal system from first principles[13].
We have prosecutors and defenders locked in an adversarial game. We have a presiding judge. We have appeals. We have lower and higher courts and even the 'supreme court' of human judgement or attention. We have principles for creating, storing, maintaining, and handling evidence. We have concepts of separation of powers, impartiality, and recusal. This should not be surprising a-priori since if we think about it the legal system basically has to solve a very similar problem of trying to find and adjudicate some notion of truth, while understanding that every component of the system is fallible, prone to perverse incentives, and can potentially be acting adversarially, and without any kind of 'truth oracle' that the system can defer to. Obviously not everything is the same. In the legal system there are side-channels and incentives. Actors are human and have intrinsic selfishness. There are limits on attention and time and money to prosecute cases. The optimization pressure applied to the system is relatively little in general. We have the opposite for our reward hacking system. Here we are applying vastly more optimization power than in the regular legal system on net. But we have much greater transparency and total control over the incentives. We can design the precise gradients and objectives that are applied to each part of the system. We can enforce perfect-ish firewalls between the components (unless the underlying computational substrate of the training process is hacked!). Although speculative, my impression here is that this makes the system much more tractable but also much more dangerous. In theory, if done well, we can construct a very robust system which is better at finding truth than the legal system; but because it can be more robust, it can handle much higher pressures, which makes any crack or destabilization much more explosive than before.
Given this, it is interesting to ponder what other ideas or mechanisms or concepts we can port over from classical legal theory to our reward hacking setting. I think this might be a fruitful area for future thinking and experimentation. Some interesting ideas here which we are resurfacing are the separation of powers (of first and second-order verifiers); standing and justiciability -- i.e. the model cannot appeal anything it has to appeal only specific provable issues with the task or verifier; systems of precedent, case-law, and evidence gathering and handling, i.e. we keep a record of all prior reward hacking incidents and responses to these, the verifiers, second-order verifiers, confessors etc have defined bases of evidence they can draw from and must adhere to a defined 'spec' in their reports, and so on. Likely there are other very interesting and important ideas we can draw from legal theory which I am missing because I am not at all an expert in this.
As one final additional idea before we close, I want to put forward the idea of decoupling the policy search algorithm from the process that creates the final model which is deployed widely. Specifically, my thinking is it might be safer not to explicitly deploy a model trained with an unbounded reward-seeking objective like regular RL/policy gradients. Rather, my thinking is that for the final distillation phase we should use something much closer to Upside-Down RL (UDRL).
The idea here is deceptively simple. Instead of training a model to execute trajectories to get reward using RL, we instead generate trajectories, label them with reward/no reward, and then train the model as a map between reward and the action trajectory as a supervised learning problem. This effectively inverts the standard RL problem. In regular RL, we give the agent the initial conditions and it figures out the trajectory that gives it maximum reward. In UDRL, we give the agent the initial conditions, and a target amount of reward to get and it then figures out the trajectory which gets it that amount of reward. Then, at runtime, we ask the agent to kindly produce trajectories with a high amount of reward instead of a low amount, and then this should lead to good behaviour similar to the original RL reward-seeking agent.
There are a couple of serious advantages of this proposal from a safety perspective. Firstly, the supervised learning objective is much safer to blindly maximize than the reward itself. This is because, analogous to LLM pretraining, the objective is to perfectly match the distribution of the joint (reward, trajectory) distribution rather than maximize reward per se. Improved optimization power thus goes towards better modelling various aspects of this distribution versus just finding increasingly clever reward hacks. Secondly, this objective gives us vastly greater controllability of reward at runtime. A regular RL agent can only maximize the reward it was trained on. We cannot give it different objectives at runtime except by changing the initial conditions. With UDRL, we can explicitly instruct an agent to aim for less than maximal reward. This is a very basic safeguard that lets us e.g. implement quantilizers dynamically without having to hardcode this in during training. Thirdly, and perhaps most importantly, it lets us encode much more detailed goal semantics than the reward as well as explicitly develop a conception of reward that is much richer than the usual scalar formulation. In the standard UDRL framework, we set the 'task' conditioning variable to essentially just be some desired level of the reward, however there is no reason we cannot generalize this much further. Indeed, LLMs provide an immediately obvious and extremely general semantics: natural language. That is, during the training phase, the model executes rollouts and we store descriptions of these rollouts and metadata about them in natural language. We then learn a general model of the joint distribution between the rollouts and their annotations. Then, during inference, we reverse this and use the annotations to describe the kind of rollouts that we want the agent to execute. Importantly, this can include information like the reward and whether the task succeeded but also information about the methods by which the agent went about the task, descriptions of the reasoning style, and potentially even things like the presence or absence of reward hacking. This latter is especially important because if we can train agents with the knowledge of whether reward hacking is occurring in a trajectory or not, then we can simply instruct it not to reward hack at runtime, and it can know what we mean. Moreover, it has no incentive to secretly reward hack anyway, since the supervised learning objective rewards accurately modelling the data where not reward hacking primarily annotates trajectories which (hopefully!) are not reward hacking. Furthermore, here we might also expect and see the broad generalization capabilities of LLMs being incredibly useful in specifying aspects of the desired trajectories in natural language that are not necessarily perfectly represented in the training data.
Put together, we would then see the system as follows. The entire direct RL phase is principally about training intermediate agents which generate trajectories that are then labelled with dense metadata about various aspects including whether they are hacking or not. The goal of the RL phase is thus really to do exploration and consolidation such that highly successful trajectories appear with nontrivial frequency in this dataset. Then, once we are happy with the performance of our RL agents, we throw them away and take their trajectories and distill them into a fresh model via UDRL. These trajectories we use for distillation include both legitimate and pathologically reward-hacked trajectories which are hopefully correctly labelled so that this UDRL model gains a deep understanding of what reward hacking is and how to avoid it. Moreover, we can pour a lot of compute and capacity into this stage safely due to the properties of the SFT objective vs the RL objective. This final UDRL model is then the one deployed and during inference we can prompt it with goals including things such as 'don't reward hack' and 'don't necessarily achieve maximum reward' (i.e. quantilization), which we can dynamically change at runtime depending on the safety properties that we want.
While this idea cannot perfectly prevent reward hacking[14], it adds another somewhat independent layer of defense against it, in accordance to our general defense in depth philosophy. If we step back and look at things from the perspective of the actor. Right now to reward hack, all it has to do is to defeat a relatively simple verifier. Under our proposed system it first has to defeat an adversarial verifier which learns from previous reward hacks, somehow avoid confessing its own reward hacks, decide to reward hack even though the task is fair and if it is unfair it can appeal, somehow also befuddle the second-order verifier if the episode is selected for auditing, then gets its reward hacked trajectory labelled as clean and fed into the training data for the UDRL distillation phase, and then somehow avoid the natural language generalization of the underlying UDRL LLM which has seen a lot of similar reward hacking trajectories be explicitly labelled as reward hacked and then instructed explicitly to avoid reward hacking. Obviously this is not impossible, but seems to be a good deal harder. My hope is that by designing increasingly complex systems like this, there is a principled engineering approach to reducing reward hacking to tolerable levels while we solve some of the other outstanding and more theoretical problems of alignment.
Which was in fact the earlier term.
As Nostalgebraist says in this post (which I strongly recommend you read): It is like using Newton's method to find the root of a function and complaining you don't like the root it found. The problem isn't with Newton's method. The problem is with your choice of function to begin with.
For what is a 'verifier' except a very crude utility oracle?
I think another reason for this is just that any kind of Bayesian methods which track uncertainty in a nontrivial way thus far have been too expensive to use practically. This does not just apply to uncertainty over reward functions but also uncertainty over parameters, activations, etc. Bayesian neural networks have been developed for over a decade and have not yet been used at any kind of scale because the benefits you get by tracking uncertainty do not outweigh the large increase in computational cost -- and these are with very simple posterior approximations (!). Similarly, even second-order optimizers which maintain a slightly more detailed notion of parameter uncertainty during optimization have failed to break through where we have instead the heuristic AdamW and the non-Bayesian Muon taking precedence. From a Bayesian standpoint current training is just black-box variational inference with a delta-function posterior over the current weights, and we have been unable to use any of the highly developed techniques of regular Bayesian statistics or theory to substantially improve upon that.
Interestingly, 'we' as an industry appear to be doing the simplest most naive thing and this, under sufficient optimization power, does indubitably generate reward hacking, but it also somehow does not always immediately degenerate into reward hacking and can apparently also train very powerful capabilities which aren't reward hacked into the models.
From the standpoint of legal theory this is an inquisitorial legal system rather than an adversarial one.
Now this is one place where we have to be extremely careful not to backpropagate any reward signal from the confession back into the normal actor's tokens. Otherwise, we will get the pathology where the actor optimizes for easily confessable reward hacks in order to have something to confess and get the reward associated with confession. This is easily solved by just firewalling the confession gradients from the main actor's reward gradients, but this could easily trip up the system if not careful.
It is important to note that the general idea here is just cross-validation of only partially correlated information channels. Here we just have two channels but the exact same algorithmic structure can be used with N partially independent channels, and increasing N generally increases the robustness of the procedure. An important goal, then, should be coming up with new ways to construct partially-independent information channels from the actor back to the verifier and the rest of the judgement apparatus.
It is worth pointing out, again, that this entire complex apparatus cannot truly eliminate reward hacking, since again that is a problem fundamentally of information. All it does is remove some incentives to reward hack (via the appeals process), and generally make reward hacking much more difficult. This latter is because we have effectively replaced the relatively 'simple' reward function of the basic verifier to a much more complex 'reward function' comprising all of these different moving pieces and multi-layered checks and randomization and so on. Now clearly, this more complex system is still going to be adversarially vulnerable to some things, and there can be correlated failures or entire bits of the state space existing in a collective blindspot, but nevertheless, it seems likely to be harder to reward hack against than the basic verifier. And, crucially, our victory condition by and by large is not to create a perfectly unhackable system under all circumstances, but only to create a system where it is much harder to hack than do the task the 'right' way.
When phrased like this it is kind of impressive that current models reward hack as little as they do. Obviously they reward hack a fair bit, and can become dangerous because of it, but they don't reward hack all the time, and in fact usually they give reasonable quality output even on tasks which seem superficially similar to those likely to have been included in RLVR.
Note here that having been updated by previous reward hacks could be bad since the verifier or meta-verifier could have seen these reward hacks and updated against them. The reward hacker likely needs new reward hacks to remain undetected. Another way to think about this is that the reward hacker only has access to weight updates. The verifier gets both weight updates plus a detailed in-context archive of prior reward hacking attempts. The hope is that this asymmetry provides a substantial advantage to the defense.
One potential danger here though is that even if the gradients of the calibration probabilities are screened off from propagating through the actor token probabilities, they ultimately still modify the same weights, unless we are only training a separate 'calibration head' on top of the actor. If we are training the full actor for this then it is possible that this implicitly conveys direct information about the blindspots of the verifier into the actor, which could obviously make reward hacking much easier for it. Something to track and be wary of in actual implementation.
Slightly unrelated, but there is an interesting prospective temporality which arises from RL training, and consequentialism more deeply. Everything must be evaluated backwards from the perspective of the future. The current trajectory is evaluated in the future perfect -- 'what will this trajectory have meant from the perspective of its total reward at the endpoint?'. With value functions we attempt to approximate this final future reward in the present. Similarly, this pseudo-legal architecture of judges and meta-judges and appeals and so forth is importantly relying on future information and further thought to evaluate, contextualize, and update upon past trajectories. This introduces a number of interesting trade-offs which appear fundamental on first glance. More specifically, there is an intrinsic trade-off between staleness and accuracy. We can theoretically know with great accuracy, deep into the future, how a particular trajectory performed in the present, but this means we can only update on it much later when it might have become stale. Conversely, we often do not know what the future will reveal about a trajectory in the present, but can update on a rough-and-ready approximation to the final future's judgment right now, when everything is immediate. A value-function, then, is an attempt to amortize the prediction of the future's judgement in the present across a large number of historical trajectories and their final judgments. There is thus effectively a trade between different kinds of biases. Wait till the end and you get an unbiased result but in a nonstationary world or learning environment you become biased because of staleness. Stop and make an update because of a premature judgment and you get a bias precisely because of this prematurity. The future may reveal hidden considerations that render your earlier estimates obsolete or misleading. Try to amortize this inherent future prediction problem into a learnt predictor and you introduce the bias of systematic misgeneralization of this predictor. Interestingly this can continue indefinitely. We can always gather new information that recontextualizes what happened previously and which can update our previous reward assessments. We actually see this in real-world legal systems as well where although there is precedent and case-law, new judgements can overturn old precedents theoretically indefinitely. Another interesting thing is that our standard ML idea of amortization gives a new perspective on precedent and common-law systems -- these are simply amortizing explicit jurisprudence. Instead of needing to invoke costly or biased detailed legal reasoning about a particular case from first principles, we can pattern-match it to a similar case and invoke the same reasoning there.
Obviously this does not solve every issue. For instance, it cannot really stop a schemer who knows how to play the training game with the appeals, confessions etc. Although it makes things slightly more challenging for the schemer since it has to scheme about the behaviour of a much more complex system to be able to manoeuvre through it. Similarly, the system obviously offers no protection to a reward hacker who just directly hacks the underlying instantiation of the system and can e.g. spoof verifier or meta-verifier messages, deliberately falsify evidence, etc.