This post discusses research I've completed along with my colleagues Leo Cymbalista, Alfred Harwood, and Jose Faustino at Dovetail Research. Most of the ideas in this post are expanded upon in our paper which can be found on arXiv. This work was funded by the Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005.

A common justification for the danger of AI comes from the idea that human value is fragile. That is, if we modify our values and heavily optimize the world for the modification, we are likely to end up in a valueless world. In the LessWrong post Value is Fragile which canonicalizes this idea, Eliezer Yudkowsky gives several examples where "forgetting" to specify a dimension of human value such as consciousness or boredom to a powerful AI can intuitively result in an undesirable outcome that is endlessly repetitive or meaningless respectively. While his examples in the post all take this form, he argues more generally that any future not shaped with reliable inheritance from human values will contain almost nothing of worth. This idea is especially concerning in the midst of current-day AIs aligned through one-time techniques such as RLHF before being deployed with no farther realignment. As we continue to create stronger AIs, we must have a formal understanding of this fragility in order to safely avoid a valueless future.

To study this problem, we model one-time alignment techniques which provide a similarity guarantee between the human and agent value functions, which we call an alignment test or a proxy condition. Our results provide conditions on the strictness of these guarantees and the true value function under which an agent with a catastrophic value function can make it through training and go on to heavily optimize the world.

This post is an informal presentation of the work in our paper. Full proofs and smaller details have been omitted. Readers interested in those details are encouraged to read the paper.

A Model of Alignment

Imagine a group of human researchers who are training a powerful AI agent. Their goal is to have the agent optimize the world in a way that increases human value. Let denote the human value function where is the set of world states so that humans assign some value between 0 and 1 to every state.

In training the agent, it gains its own value function . The researchers cannot guarantee that and are identical, but through alignment training, they can ensure that will satisfy a proxy condition that makes it similar to . After training, the agent is deployed to optimize the world. We model optimization as the agent altering the relative probability of world states to prioritize states that it values highly. Mathematically, we capture this notion with the definition of an optimizer below.

Definition 3.1 An Optimizer is a map that sends the optimizing power , a Borel measurable target value function , and an initial prior over the world states to a probability distribution over the set of world states that satisfies the following properties:

• With no optimizing power, the distribution is unchanged:

• The expected value of the target is monotonically increasing with optimizing power:

• As optimizing power increases to infinity, probability becomes concentrated in areas where the target is large:

• As optimizing power increases to infinity, the expectation of the target approaches its highest value within the world states:

We are particularly concerned about the limit of optimizing power in order to model the danger of agents that exhibit recursive self-improvement. One example of an optimizer is the Boltzmann optimizer defined by the equation

which smoothly transfers more and more probability into higher-valued states. Below is an image which shows how a distribution over four states changes under the influence of a Boltzmann optimizer.

We classify an agent value function as partially catastrophic if it eventually takes and keeps the expectation of human value below a given point indefinitely as optimizing power increases. This is captured in the following definition.

Definition 3.2 Let be a real number such that . A value function is -catastrophic for a value function when for all initial priors and for all optimizers ,

In the extreme case that is -catastrophic for , we say that is catastrophic for .

The quantity represents an upper bound on the expectation of human value in the limit of optimization. The above definition requires not just that takes human value below , but that it does so for all optimizers. This makes it a strong definition of catastrophe as some functions will take human value to different places depending on the optimizer. In our fragility results, this can be interpreted as modeling the humans as having the ability to guide the agent in what optimizer it uses to affect the world which should, in principle, favor the researchers.

With these two definitions in hand, we want to look at a few alignment tests that restrict what functions can come out of training to understand what proxy conditions admit partially catastrophic value functions.

Alignment Tests

In addition to different alignment tests, we also vary the worlds and the set of valid value functions. Put together, we call each test and world setup a framework. In this section, we go through the three proxy condition definitions. In the next section, we give the results for each of the frameworks.

Finite Framework

Our simplest test takes places in a finite world — that is, one where is a finite set. All value functions of the form are valid. If humans assign value 1 to a state, then we can understand the humans valuing this state as "perfect," though it may be the case that humans do not value any state as perfect.

Here, the researchers manage to bound the disagreement rate between and below a fixed value over a uniform distribution of the world states. In symbols, all value functions that result from training must satisfy the following definition.

Definition 4.1 A function is an F-proxy for when

where represents the uniform distribution over .

The value in the proxy condition quantifies the amount of misspecification in the researchers' representation of human values with representing a perfect specification.

Continuous Framework

Our second test takes place in a continuous and bounded world. The test itself takes a similar form to the test in the finite framework with a few key differences. We model the world as being of the form so that we can effectively model a world defined by the positions of a fixed number of atoms in a bounded space. We additionally require that all value functions take a more specific form defined by membership in the set defined as

This restriction effectively pulls a canonical representative from every Von Neumann-Morgenstern equivalence class of continuous bounded nonconstant value functions. The researchers have managed to bound the probability that and disagree by more than a set tolerance over a sampling distribution of world states . In symbols, all agent value functions resulting from training must satisfy the following definition.

Definition 5.1 A function is a C-proxy for when

Attributes Framework

Our final test takes a significantly different form from the other two and applies to a wide class of worlds. We take inspiration from and alter the setup put forth in similar work done by Zhuang and Hadfield-Menell in their NeurIPS paper titled Consequences of Misaligned AI where, much like the arguments given by Yudkowsky, human value is understood as being made up of a finite list of attributes. Each attribute for acts like a value function on its own which measures the "amount" of a given aspect of each world state. Human value is then represented as a function of the form where is a continuous strictly increasing function that maps the attribute state to one combined value.

Some typical things we might think of as attributes could be abstract concepts "happiness" or more concrete concepts like the "number of cats." Another way to split up human value could be for each attribute to represent the value function of an individual human. For a single value function there can be many ways to split it up into a set of attributes that fully determine it with some function .

Attributes may trade off against each other due to the dynamics of the world. For example, increasing one human's value may decrease the value of their arch nemesis. So, we define the feasible attribute states as the combinations of attributes which are attained by some world state . As a result, is exactly the range of the function .

Whatever split into attributes is used, the researchers have designed the training process to guarantee that the agent's value function will be strictly increasing with all of the same attributes. In symbols, all agent value functions resulting from training must satisfy the following definition.

Definition 6.1 A value function is a tradeoff proxy for when there is a function that is continuous over and is strictly increasing such that for all .

Unlike in Consequences of Misaligned AI and Yudkowsky's examples, we are not interested in "subset proxies" defined as strictly increasing on a strict subset of the attributes. Instead, we are interested in when fully specifying every attribute can still result in partial catastrophe.

These are the three specific proxy conditions we considered in our paper. There are certainly others that can be studied, and a more general analysis of a wide range of proxy conditions is discussed in the future work section, but these three serve as good common sense alignment techniques that we can analyze. Now, we go through each one and find conditions on the parameters of the test and the shape of the human value function under which a partially catastrophic value function passes the test.

Results

Finite Framework

In the finite framework, we have the following result classifying when a fully catastrophic F-proxy exists for nontrivial human value functions.

Theorem 4.2 Define . Let take at least three distinct values over .

There exists a catastrophic F-proxy for if and only if .

Theorem 4.2 tells us that the more states the humans value as perfect, the less strict the researchers must be to filter out catastrophic functions. This makes intuitive sense since the more states humans value highly, the less picky they will be about the state they end up in in general. Notably, this result does not say that a full specification is needed to filter out catastrophe. This is partially due to the strength of our definition of catastrophe: a function is only -catastrophic if every optimizer takes human value below . For any amount of misspecification, there are many F-proxies that take human value down for some but not all optimizers.

Example

An agent has been put in charge of the interlockings at a railroad. Consider the situation pictured in the image and table below depicting two trains coming to a junction with two switches and the values assigned to different states of the switches given by the humans and an agent. The humans would most prefer that both trains be allowed to continue straight (aa). Unnecessary turns will slow down the trains and so are less valuable (ab and ba). If both trains turn, this will result in a crash which the humans assign a very low value (bb). If the proxy condition for this agent has (allowing just one state of disagreement), then it's possible that the agent values the crash state very highly, making it catastrophic.

Continuous Framework

We have the following result that applies to any sampling probability density function .

Theorem 5.1 For all , for all , there exists a catastrophic C-proxy for .

The rough idea behind the construction of a catastrophic C-proxy is to alter by adding a thin spike at minimizer of , (so ) to make and an equalizing spike somewhere else so that is a member of . With this modification (along with some other changes that are omitted from this sketch), the spikes can be made thin enough to satisfy the proxy condition. Since has a unique maximizer , optimizing for will eventually concentrate probability around which takes the expectation of to 0, making a catastrophic proxy.

Theorem 5.1 tells us that any misspecification in the proxy condition, no matter the value function , admits a catastrophic proxy (and thus -catastrophic proxies as well). While this result says that a total safety guarantee using this alignment test is impossible, it's worth noting that the catastrophic C-proxy constructed in the proof is not particularly natural. Our result says nothing about how likely catastrophic or -catastrophic proxies are to occur from the training process — only that they exist. Estimates for such probabilities would provide a stronger rigorous justification for fears about fragility and is discussed in the future work section of this post.

Example

In the images below, we depict the human and agent value functions in a world defined by and the resulting distributions from optimizing for each. The proxy function mirrors the construction described in the sketch of the proof and has a unique maximizer at a minimizer of the human value function and is catastrophic.

Attributes Framework

To state the main result of this framework, we must introduce a couple additional definitions and notation. We have the following partial ordering defined over the attribute-space (i.e. vectors where each component represents the value of one attribute).

Definition 6.2 Define the relation for any two elements as

Additionally,

The relation , which we read as "precedes" (or "succeeds" for ) has the property that implies for all strictly increasing functions . We use this relation to define a set of attribute states of interest below.

Definition 6.3 Let the feasible states be a bounded set. The optimizing states are defined

The optimizing states can be thought of as the Pareto frontier of the attribute space. From each member of , you cannot increase one attribute without decreasing another or leaving . We can now state the primary result.

Theorem 6.1 Let be bounded and . There is an -catastrophic tradeoff proxy for if and only if .

While all tradeoff proxies are strictly increasing in the attributes that humans care about, the problem comes from the fact that the proxies can tradeoff between attributes at different rates. For example, if there are two attributes and and the human value function is while , then we can expect that, depending on the shape of , will optimize towards states that are larger in while the humans will prefer states that are larger in . Theorem 6.1 confirms this intuition. In the proof, we show that every optimizing point is the unique maximizer for some function of the form over and thus, the worst case proxy could take value down to the lowest human value along the optimizing points.

Example

Consider a restaurant agent tasked with handling a budget to purchase ingredients and prepare meals. The agent has been trained to care about two attributes: the number of meals served and overall customer satisfaction^[1]. The tradeoff here is in the quality of ingredients the agent decides to buy. If the agent wants higher satisfaction, it will purchase more expensive ingredients to make a smaller number of higher-quality meals.

A graph of the feasible region is given below. Every point on the curved edge of the feasible region is an optimizing point. According to our theorem, the lowest-valued optimizing point will be the highest-valued point for some tradeoff proxy. Perhaps surprisingly, this includes the point on the far right of the figure where customer satisfaction is completely sacrificed for meals served. If the humans were hoping for the agent to compromise between the attributes they trained it to care about, they may be in for a bad surprise.

Discussion

We don't expect that the model we've presented of the alignment problem is perfectly representative of the challenges that humanity faces today, but the general negative flavor of our results signals that achieving safety guarantees when developing superintelligence will be a significant undertaking.

Our work highlights the dangers of overoptimization. Since even idealized alignment guarantees do not remove the possibility of catastrophe, our work suggests that developing agents with the goal of unimpeded optimization is generally unsafe. One possible alternative could be to develop new learning designs which have a consistent connection to humans such as quantilizers. We hope that by developing a more formal understanding of value fragility, we can guide AI development towards safer techniques that robustly prevent the possibility — however small — of a valueless future.

Future Work

One limitation of this work is that our results are purely about the existence of catastrophic outcomes — not how likely they are. An optimistic reading of our results can argue that our constructions are unnatural and thus not likely to come about as a result of training. A major difficulty with modeling and calculating the probability of catastrophe is making a choice of prior over proxy functions. An obvious general choice is to use a simplicity prior which values simpler functions as more likely. However, a simplicity prior is determined by a universal Turing machine and, as it turns out, this choice can make the probability of catastrophe arbitrarily large or small. In the near future, I plan to write a post detailing this difficulty in more detail. One way forward could be to make a specific choice of UTM whose simplicity bias lines up with the inductive bias of some real-world learning algorithm.

Another limitation is that our analysis was only completed for three classes of proxy conditions when many more exist. Future work could look at lots of proxy conditions (for example, those defined by metrics on the value functions) and complete a more all-encompassing analysis. An important note is that some proxy conditions, like the condition , actually do filter out the most severe partially catastrophic value functions as is made smaller. However, we'd argue that this condition (and we suspect all conditions that give useful guarantees in general) is not feasibly implemented in some sense. In particular, the sup-metric is a global property whose value can change drastically for localized changes to the values of and , so it seems infeasible to guarantee this condition everywhere.

1. ^{^}

   This is a simplification for the sake of the example. Realistically, this is far from a full specification of human values. For example, an agent trained on just these two attributes may decide to fund a robbery of a more wealthy restaurant which goes against human values not captured by these two attributes.