In this short post, I talk about a difficulty we encountered while working on a model of the fragility of value. You can read my post about that work here.

The TL;DR is that the probability of a fixed event can be made arbitrarily large or small depending on the choice of simplicity prior. I'm sure this result was already known, but I found it unintuitive and thought it might be helpful to share.

Background

In our model of the alignment problem, AI agents go through idealized alignment training before being deployed to heavily optimize the world. The alignment training restricts the set of value functions that the agent can wind up with to those that satisfy a proxy condition; such value functions are called proxies. We also have definitions of catastrophic and partially catastrophic value functions that are bad to optimize for.

One thing we’re interested in is the probability of training resulting in an agent with a catastrophic value function for a given proxy condition. A natural way to model a prior over learned functions is to use a simplicity prior — that is, a distribution over proxies where lower K-complexity proxies are more likely. However, the difficulty we encountered is that the choice of simplicity prior can make this probability arbitrarily large or small.

Arbitrary Probability of Catastrophe

Let be a universal prefix-free Turing machine which can output value functions. The universal a priori probability of a function under is given by

which represents the probability that outputs when each bit of the program tape is sampled uniformly and randomly. By the Coding theorem, this probability coincides with up to a multiplicative constant: , hence simpler functions are generally more likely.

Let represent the set of proxy functions and let represent the nonempty proper subset of catastrophic proxies. The training process selects a function from this distribution with the additional condition that the function is a proxy. Then, the probability of catastrophe for the simplicity prior defined by the UTM is given by

where . Now, let and consider the class of universal prefix-free Turing machines defined by

For all , is a UTM since it can run any Turing machine that the universal Turing machine can, which is all of them. Now, we look at the probability of catastrophe under the simplicity prior induced by :

Because devotes the shortest valid strings to outputting , the probability of a catastrophic function being selected becomes arbitrarily close to as increases. Even with just one catastrophic function, there is no upper bound on the probability of catastrophe without additional assumptions beyond that the distribution is defined by a simplicity prior. The probability can also be made arbitrarily small if we replace the short string output with some non-catastrophic proxy function .

Discussion

The argument in this post shows that there is no way to meaningfully bound the probability of a fixed event under a simplicity prior without farther assumptions. To get a bound, one could select a specific UTM which in some way corresponds to the inductive bias of a training process (e.g. maybe sine-curves are easy for a given architecture to calculate and are thus more likely to result from training). This is easier said than done.

In the context of our work on the fragility of value, this result says that, if a proxy condition admits even one catastrophic value function, then there is a UTM which makes this catastrophe highly likely. This motivates the need for a better understanding of the distribution of functions acquired through various learning processes.