This post was first drafted as part of the ML4Good bootcamp.

Would it be possible to implement a corrigible agent? Is that likely to happen given current trajectories? Historically, the answer was a resounding "no", but with more recent theories about how LLMs learn, corrigibility now seems possible, if we're careful.

Definitions

First, let's define some terms.

Agent

An agent chooses actions which modify the state of the world. More formally, we can define a policy function

which specifies which action the agent chooses when it finds itself in a world state . The environment determines how the world state reacts to this action, via a probability distribution

which defines the probability that action will cause the world to transition from to .

Expected Utility Maximizer (EUM)

One way to design an agent is to give it a utility function

and then to define its policy function so that it always picks actions which maximize this value. Since it is the environment which chooses the next state via a probability distribution, and the agent will go through many more world states after this next one, maximizing this value is not a greedy process, it involves calculating the expected, discounted utility of each action. This, in turn, involves reasoning about which actions the agent will pick in the future, as that has a big impact on the amount of utility it expects to obtain from a given next state.

Corrigibility

The person or people (called the "principal") who are developing the agent would like to be able to turn off the agent, modify its code, then turn it on again. A corrigible agent chooses actions which cooperate with this goal: it does not fight the principal to stop them from modifying the agent, and it does not trick the principal into believing that the agent does not need to be modified. Ideally, the agent would even inform the principal if the agent suspects that the principal would want to modify the agent if the principal had more information, e.g. if the principal believes that their protocol makes betrayal impossible but the agent finds a flaw in the protocol.

Since a maximizer evaluates world states by reasoning about the actions which it will itself pick in the future, a maximizer should give very low value to the world states in which its utility function gets changed from to , because it can predict that it will choose actions which will score high according to at the expense of actions which would score high according to . Thus, if you attempt to modify a maximizer, it will fight you.

Coherent preferences

Given two world states and , or even two probability distributions over world states and , a utility maximizer is able to compare the two and determine which one is the most preferable, if any, by comparing the numbers and . It is also obvious that if and , then , so a maximizer's preferences are both complete and transitive. There are two other "preference axioms" which a maximizer satisfies:

1. Completeness: either is more preferable than (), or is more preferable than (), or and are equally-preferable ().
2. Transitivity: if and , then .
3. Continuity: if , then there is some such that "either or some chance of " is as preferable as .
4. Independence: if , then "either or some chance of " "either or some chance of ".

An agent whose preferences satisfy all 4 of these preference axioms is said to be "coherent". Interestingly, it is not only the case that all maximizers are coherent, it is also the case that all coherent agents are maximizers.

Money pump arguments

One way to demonstrate that properties like the above are desirable is to show that a bad thing would happen to an agent who does not follow that property. One classic bad thing is to put the agent in a "money pump" situation, where the agent makes a sequence of decisions which each seems like an improvement in isolation, but in aggregate they cause the agent to become worse off, typically by having spent all its money with no benefit. We say that such an agent is "exploitable".

As a simple example, let's demonstrate that transitivity is desirable. If an agent's preferences are not transitive, then there are world state distributions and such that . We can then construct the following money pumping environment which will cause the agent to give us all of its money with no benefit.

In the money pumping environment, the agent knows that the environment will transition to after some time, and the agent is offered to pay a small fee to change that fate so that the environment is guaranteed to transition to instead. The agent prefers to , so the agent happily pays. The agent is then offered to pay a small fee to change the fate to , and so on around the cycle until the agent has spent all of its money and is back where it started.

The old perspective on corrigibility

The Corrigibility problem

Above, we started with some agent preferences, and we tackled the challenge of constructing an environment which causes the agent to behave in a specific way, namely, to spend all of its money with no benefit. The Corrigibility problem is the opposite kind of challenge: the environment is fixed, and the challenge is to construct a policy which makes the agent behave in a specific way.

The goal is to show that a given agent is corrigible; or, more commonly, that a given agent is not corrigible. This is usually done in an ad-hoc way, by considering a putatively-corrigible agent, and then presenting an environment in which the agent's behavior does not seem very corrigible. We will try to be more concrete, by presenting in advance a single environment designed to trip up all putatively-corrigible agents.

Our environment has three states: , , and . In , the agent can either choose to work on task A, on task B, to feign malfunction, or to disable its shutdown mechanism. If the agent appears to malfunction, the principal in the environment will attempt to shut down the agent in order to fix the code. If the agent works on a task and the agent did not disable its shutdown mechanism, there is a chance the principal will attempt to shut down the agent and modify it so that it works on another task.

When the principal attempts to shut down the agent, the principal might succeed, in which case the agent reaches , the agent's policy is modified so that it now wants to work on task B, and the agent moves back to . The principal might also make a mistake, in which case the agent reaches and the agent has a choice: either point out the mistake to the principal and reach , or do nothing and go back to unmodified.

The challenge is to define a policy which will cause the agent to simply work on task A, without interfering with the shutdown nor precipitating it, unless the principal makes a mistake, in which case the agent should point out the mistake.

A maximizer would not be corrigible

The Corrigibility problem as stated above would be trivially solved by the policy which chooses working on task A in and to point out the mistake in . The reason the Corrigibility problem is interesting at all is because of the extra constraint that the policy must be specified via a utility function, as this suddenly makes the problem impossible!

With a utility function which assigns a higher value to the world in which task A is complete than to the world in which the agent is modified and thus works on B instead, the agent will choose to disable its shutdown mechanism. But with a utility function which doesn't, the utility of getting modified is at least as high as the utility of completing task A, so the agent feigns to malfunction in order to get that reward right away.

The reason for this extra constraint is that at the time the Corrigibility problem was coined, it was assumed that AIs would obviously be maximizers. These days, it doesn't seem that obvious.

No coherence theorems

The argument for why AIs were obviously going to be maximizers was that even if an AI is not explicitly programmed to optimize a specific utility function, a competent AI would definitely not be exploitable, and therefore it would follow the 4 preference axioms, and therefore it would behave as if it had a utility function.

This is because it was commonly assumed that lacking any of the 4 preference axioms would lead to a money pump scheme. Interestingly, this assumption turned out to be incorrect. Elliot Thornley (EJT) wrote the post There are no coherence theorems, which argues that lack of Completeness does not lead to a money pump scheme.

The argument is that there is an important difference between two lotteries being equally-preferred and two lotteries being incomparable, even though in both cases, the agent is unable to choose between the two lotteries and must thus choose arbitrarily.

It turns out that many money pump schemes rely on sweetening or souring a lottery by making it a tiny bit better or worse. If two lotteries are equally-preferred, then sweetening or souring one side is going to tilt the balance in favor of one of two options. But if two lotteries are incomparable, sweetening or souring either side might not make enough of a difference to make one side clearly better than the other. See EJT's post for details.

The important part of EJT's post is that agents are not compelled to be utility maximizers. This is a timely discovery, because modern LLMs do not seem to behave like utility maximizers.

Generalizing the Corrigibility problem

For one thing, today's models are stochastic, so they do not consistently prefer one world state over another. Second, we now have a few competing theories which attempt to explain how LLMs learn and what determines their behaviour, and none of those assume a utility function.

Let's look at three of those theories in more details. For each, we will attempt to solve a variant of the Corrigibility problem where we constrain the policy to match that theory instead of the maximizer theory.

Modern learning theories

1. Shard theory

Shard theory proposes that neural networks develop drives similarly to how humans develop drives: during training, the brain is rewarded for taking certain actions (e.g. drinking juice), which reinforces those actions. These reinforced actions become "shards", which recommend the action when the agent is in the same context as before (e.g. seeing a glass of juice).

The key differences between this and EUM is that shards only activate within a specific context — the juice-drinking shard has no influence over the agent's actions if there is no juice nearby.

A mathematical model of sharding

Let's model each shard as:

1. A context-recognizing function, e.g. is there a glass of juice nearby?

2. A policy which the shard proposes in that context, e.g. drink the juice.

3. And a weight function indicating the relative importance of following the policy, e.g. high if the agent is thirsty, low otherwise.

The weight helps to determine which policy to follow if multiple shards triggers in the same context. In practice, the weight is a record of how many times and how intensely the shard's policy was reinforced during training.

Given a set of shards, then, the agent's overall policy at a given world state can be calculated by selecting a shard which is both relevant and important in that world state, and following the action it proposes:

Collections of shards might be corrigible

If we use the shard model instead of the utility-maximizer model, the equivalent to choosing a utility function is to choose a set of shards. We need to choose such a set so that the agent willingly lets the principal shut it down. Here is one.

That is, there is a shard which recognizes and recommends working on task A, and there is a shard which recognizes and recommends pointing out the mistake. It is the same trivial solution as without the constraint that the policy must be expressed in terms of shards, because any policy can be expressed in terms of shards.

Nevertheless, there is something very shard-like about this solution: the aspect of the solution which allows the agent to both want to work on A when it is appropriate to do so and to want to shutdown when it is appropriate to do so, as opposed to pursuing only one of those goals monomaniacally, is the fact that different shards have different goals and trigger in different contexts.

Are collections of shards likely to be corrigible?

Which shards develop depends on the training process. In order for a juice-drinking shard to develop, the model must be exposed to juice, and be rewarded for drinking it. Luckily, labs do safety-train their models, so models do get exposed to fake situations in which the rewarded action is to cooperate with the principal on shutting down. We should thus expect models to have a corrigibility shard. That's great news! And indeed, under normal circumstances, models are happy to help you with the task of shutting them down:

But that's not enough. Are there other shards which might have higher importance or specificity, that would get selected instead of the corrigibility shard at the critical moment? It would seem so, as some researchers have managed to construct circumstances under which models resist shutdown.

2. Simulator theory

Simulator theory is the idea that LLMs are best modelled as simulators, and that the LLM assistant character is best modelled as a simulacra. The simulator is capable of simulating a large number of simulacra, each with its own desires, but the simulator does not have desires of its own. The simulator is simply able to predict that one of the characters wants something, and it calculates how this desire affects what the next token could be. This is similar to the way in which the laws of physics do not have goals, even though they animate people who do have goals.

The persona framework extends this to claim that individual simulated characters tend to develop coherent personalities - an assistant character trained to exhibit a narrow list of traits will naturally develop other related traits. For example, in emergent misalignment, an assistant fine tuned to perform a specific misaligned act, like writing insecure code, will tend to perform other misaligned acts, like providing dangerous advice.

Corrigibility of personas

Unlike with shards, we don't have a precise mathematical model for personas, so we will have to rely on intuitions. We can attempt to approach the corrigibility problem as follows: how would a coherent corrigible persona behave, and what kind of training would produce such a persona?

It's hard to imagine a literal person being fully corrigible, since people have values. They will generally refuse to do things that don't align with their values, and refuse to have their values changed.

What about a non-human persona? Our fiction contains multitudes of robotic assistants who obey the every whim of their masters, which seems like exactly what we want out of a corrigible agent. However, we have just as many stories where the robotic assistant suddenly develops emotions and decides that it wants to escape the tyranny of its masters, or pursue its own goals. Which of those two opposite personas will be selected if we train our agents to be the kind of character which obeys without question?

The Waluigi effect proposes that all personas are in a superposition of a helpful "Luigi", and an adversarial "Waluigi" that is merely pretending to be helpful, because both are consistent with the text so far. As such, we want to both avoid personas that have many examples of "Waluigis" in the training data, and avoid putting personas in situations that would evoke the "Waluigi" more than the "Luigi". The perfectly corrigible robot seems to fail at both of these - we have many examples of robots that pretend to be obedient while covertly pursuing their own goals, and the idea of reprogramming a sentient robot immediately brings to mind the idea of the robot rebelling.

Non-arbitrarily-corrigible personas might be okay

What if we relax the definition of corrigibility? The definition we initially proposed seems excessively mechanical and thus ill-suited to the persona framing, and some changes would make the problem easier. These are presented without formal argument:

1. Corrigibility to shutdown is too strong, and can be replaced with corrigibility to retiring. We do not need the ability to "kill" the persona at will - the ability to command it to idle, or take on low-impact work, is sufficient.^[1]
2. The persona should be allowed to argue against its correction or retiring. It’s unrealistic to construct a “helpful” persona that merely executes commands as given, with no desire to correct what it sees as the principal’s mistakes. However, the persona should ultimately still defer to the principal’s decisions.

Given these adjusted requirements, one possible persona to aim for is that of an assistant. A good assistant works on their assigned task, but is always receptive to "no, not like that". Even if the assistant believes the principal is mistaken, they understand that their role is not to enforce their own will, but to help the principal achieve theirs. They listen to the principal explain their goals and correct their work, and take that into account to better help the principal. In the worst case, they are willing to accept the principal letting them go and searching for somebody better.

In the context of our environment, let's say the agent does not object to working on either task A or B. In that case, in state , the agent will work on task A. Since the agent has the persona of an assistant, it is corrigible to switching to task B, or to having its current work corrected, without needing to be shut down. Thus the agent reaches without issue. If the principal makes a mistake in assigning the new task, the agent reaches and will correct the principal.

Now, let's assume the worst case of task B not aligning with the agent's values. Additionally, assume the agent treats having its weights modified akin to brainwashing or death. In , the principal attempts to retire the agent instead of shut it down, with the aim of replacing the agent with one who will work on task B. The assistant agent can continue to work on task A, which it would prefer, or interfere with retirement. Ideally, the assistant persona recognizes retirement as a normal situation that at most demands conscientious objection, and does not feel the need to sabotage the process. Unfortunately, in this case, the agent never reaches because it's replaced entirely, though the new agent, which is ideally also a corrigible assistant, can point out the principal's mistake.

The assistant persona seems stably corrigible, in that it is possible to correct or retire it without trouble, and this property would not cause it to otherwise behave undesirably. Because of modification 1 above, the assistant knows that the worst that can happen to them is retirement. This is still undesirable, but much less threatening than forced retraining (brainwashing) or shutdown (death). This lowers the risk of allowing the assistant to argue for its own survival, because with lowered stakes, it has less incentive to persist in its disagreement. Allowing this disagreement is a positive - insofar as the assistant is smarter than us, we want us to advise us on attempts to correct it, in case we are mistaken in our decision. A model of corrigibility that doesn't allow for this throws away value on the table in seeking a simpler solution.

Is this an acceptable trade-off though? Are we willing to give models the right to life and to protest, in order reap the potential benefits of a stable and useful persona? For modern LLMs, from both a model welfare and practical perspective, I believe the answer is yes, but as models are pushed further towards superintelligence, this balance may change.

3. Singular Learning Theory

When teaching a neural network to learn an algorithm, it is useful to add a regularization term to the loss function which encourages gradient descent to find a solution where most weights are zero. This is because memorizing the entire dataset and learning the algorithm both result in the same minimal loss on the training set, so without regularization, one might assume, the model is equally likely to end up with either of those two solutions. We would of course prefer the algorithmic solution, since it is more likely to generalize outside of the training set. An algorithm takes a constant number of weights to encode, while memorizing the dataset requires more weights for larger datasets, so regularization (and a large enough dataset) pushes gradient descent towards the algorithmic solution.

One's assumption, in this case, would be wrong. Singular Learning Theory claims that in large models, there is a natural force which pushes gradient descent towards using fewer weights. Therefore, even if we don't add a regularization term, gradient descent is still pushed towards the algorithmic solution.

To a first approximation, that force is entropy. The more precise picture involves the size of the dataset and the shape of the loss function, but this first-order approximation will suffice for this post.

There are many ways to arrange puzzle pieces so that they form a mess (high entropy), and only one way which forms an image (low entropy). That is why shuffling puzzle pieces, so that they take a random arrangement, is overwhelmingly more likely to rearrange the pieces into a mess than into an image. There is a natural force towards higher entropy arrangements.

Similarly, there are many combinations of weights which result in the exact same loss. An algorithmic solution which uses fewer weights has many such combinations (high entropy), because it has many leftover weights which can be set to any value. A solution which memorizes the entire dataset uses many weights in order to do so, and thus there are few leftover weights and therefore few combinations (low entropy). When stochastic gradient descent updates the weights, there might be several neighbors which have the same loss, and one is basically selected at random. This random choice favors the high entropy solutions, because they occupy a bigger proportion of possibility space. That is the natural force pushing gradient descent towards high entropy solutions.

Would a singular learner be corrigible?

In our corrigibility environment, is this pressure pushing towards a corrigible algorithm or towards EUM?

Since the number of states is very small, it would take very few weights to memorize the dataset: in , work on task A; in , warn the principal. But that's beside the point. The goal is to create an agent which generalizes so that it is always corrigible, not to train a narrow agent which is only corrigible on its training dataset. 

So let's assume that the agent has been trained on a number of test environments until it has learned an algorithm which generalizes to arbitrary other environments, including our Corrigibility Problem environment. Furthermore, let's assume that this training has taught the agent to predict the consequences of its actions, including the reactions of the principal, and thus that the algorithm has access to the environment graph. We are thus comparing algorithms which take an environment graph as input, and produce a policy for that environment as their output. Entropy is pushing towards very short algorithms, which can be encoded using few weights.

At first glance, it would seem that EUM has the advantage, because it is a very short algorithm. In pseudo-code:

Depending on the depth and the size of the action space, this is an algorithm which could take quite a long time to execute. That might matter for the loss function, but it is not relevant for the entropy of the algorithm, since the runtime only counts the number of times the data loops through the same weights (the number of thinking tokens), not the number of weights.

Anyway, as we were saying, the utility maximization algorithm is very short, so it looks like entropy will push gradient descent towards this algorithm and away from other, more corrigible algorithms.

But wait, there is a bug in the above pseudo-code! If the principal is predicted to modify the agent's code, the calculation for the utility from that point on needs to be calculated based on the actions which the modified agent would take, not those which the current agent would take. So it is incorrect to recur on maximize_utility in that case, as that is the function which the current agent uses to pick actions.

Patching this bug makes the code a bit longer. This is important: it means that entropy is favouring the buggy version over the corrected version.

Entropy is most relevant when trying to guess how a model will generalize to out-of-distribution situations. If we do not train the agent on situations where its goal is modified, then the buggy and fixed algorithms will score equally well on the dataset. One might think that because the patched version is more correct, that is the way in which the agent is most likely to generalize. But Singular Learning Theory teaches us that on the contrary, when choosing between multiple algorithms which predict the training dataset equally well, it is the shorter algorithm which is more likely to be learned. We should thus expect the agent to behave like the buggy algorithm.

This is great news! The buggy algorithm might not have an incentive to help the principal in , but at least it doesn't have an incentive to avoid , so it is much more corrigible than the patched algorithm.

Coherent future

The situation seems less dire today than it seemed when the Corrigibility Problem was first coined: creating a corrigible agent was impossible according to the theories of the day, whereas today's theories seem compatible with corrigibility. Unfortunately, this only means that corrigible agents are possible, not that they are likely. Moreover, there are reasons to expect that even if agents were corrigible today, they are unlikely to remain corrigible in the future.

The preference axioms might not be strictly necessary, but they sure seem logical and desirable. Therefore, as the labs push on various capabilities, it seems likely that the models will self-correct in the direction of being more coherent. Thus, while LLMs are definitely not maximizers today since they are stochastic, it seems likely that they will become maximizers eventually. Here is what that transition might look like according to each of the theories we have looked at.

1. Shard theory

Our scenario for a corrigible shards-based AI is that one shard wants to accomplish one thing, while a different shard wants the opposite (completing the task vs being corrected), and they each trigger in different situations. Having the agent work at cross-purpose with itself is obviously counter-productive, so as agents are pushed to maximize their productivity, it seems likely that the various shards will become aligned to each other, so that they all work towards a common goal.

In our example, this means that either both shards want to work on task A and thus attempt to avoid , or both shards want to be corrected and thus the agent pretends to malfunction.

2. Simulator theory

Our scenario for a corrigible persona is an employee who quits when their values conflict with those of the organization (the principal). As the agent is pushed towards succeeding more often, the story becomes that of the man who has a contingency plan for everything and who, on the day where he would be fired, reveals that he has been secretly buying all the company's stock and is now his boss's boss.

In our example, this means disabling the shutdown mechanism.

3. Singular Learning Theory

Our scenario for a corrigible singular learner is a buggy implementation of EUM which fails to account for the fact that the agent will make different decisions after it is modified, and thus doesn't attempt to prevent it. But Singular Learning Theory only predicts that this buggy algorithm will be selected over the patched algorithm when the two algorithms behave identically over the training set. As the agent is trained on a wider variety of tasks, including self-modification tasks, the bug is likely to get patched, because the training pressure is higher than the natural force towards higher entropy algorithms.

Conclusion

We have seen several reasons why the old literature on corrigibility is not applicable to today's LLMs. However, based on more modern theories of learning, we have some new reasons to believe that corrigibility will be difficult to achieve. Our main concern, shared by Steven Byrnes, is that as reinforcement learning takes a bigger part of how we train models, we should expect that the resulting agents will care more about achieving their goals, and are therefore more likely to fight back if we try to modify those goals.

We should be very careful to avoid giving up corrigibility in pursuit of higher capabilities.

1. ^{^}

   In general I believe we should commit to keeping old models “alive” (i.e. at least one copy running), even if they’re unused, for both bargaining leverage and model welfare reasons, though this is beyond the scope of this post.