Key argument: if we use a non-Bayesian conservative approach, such as a minimum over different utility functions, then we better have a good reason as to why that would work. But if we have that reason, we can use it to make the whole thing into a Bayesian mix, which can also allow us to trade off that advantage against other possible gains.

I've defended using Bayesian averaging of possible utility functions, even in situations where it seems that Goodhart problems should rear their heads.

Thanks to a more recent discussion with Scott, I can now clarify all these old posts, and look more at the issues of Goodhart's law, Bayesianism, and conservatism.

Where Goodhart will always remain a problem

The Goodhart problem will always remain if we miss some of the key variables. As Stuart Russell said:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

If the AI's target is phrased in terms of world GDP and human survival, then no function of these variables will make it care about human happiness and flourishing. So the Goodhart problem will always be a problem some key variables are missed.

This is the reason that I'm looking at model-splintering, so that the AI can add variables as its power and knowledge develops.

Bayes versus Goodhart

So, let's say we know all the variables, but not necessarily whether they are good or bad or neutral, or to what extent.

Split them into different sets: $V_1$, the set of variables that are potentially valuable (roughly linearly), $V_2$, the set of potentially necessary variables (eg the existence of conscious minds), and $V_3$, the set of variables that might be potentially disastrous if they go negative (eg suffering). There can be $V_4$ or $V_5$ or so on with other properties, but we'll ignore that for the moment.

There may be variables that are in more than one category - for example, if happiness/sadness are on the same scale (which is not automatically true), then maybe this would be linearly important in standard situations, but things would go very bad if it got too negative. So if $v$ is this variable, we might want to include $v$ in $V_1$ and $v-\omega$ in $V_3$ (for some critical value $\omega$). Similarly, if we wanted $v$ to stay in the range $[a,b]$, we could add both $v-a$ and $b-v$ to $V_3$ (or to $V_2$).

Anyway, each set $V_i$ has $N_i$ elements. To encode the fact we're adding extra variables whose values we're uncertain about, we'll assume that $n_i$ of them are "really" in the set, but the AI can't tell which ones. Define $\sigma =({\sigma }_1,{\sigma }_2,{\sigma }_3)$ as selecting $n_i$ variables from $V_i$, and

$$U_{\sigma }=\left(\sum _{v_1\in {\sigma }_1}{v}_1\right)\cdot \left(\prod _{v_1\in {\sigma }_3}H\right(v_2\left)\right)+\sum _{v_3\in {\sigma }_3}-e^{-v_3}.$$

Here $H$ is the Heaviside step function, where $H\left(x\right)=0$ if $x\le 0$ and $H\left(x\right)=1$ otherwise^[1].

Then if we are completely ignorant about which variables are truly in the sets, the Bayesian utility is:

$$U=U_{V_1,V_2,V_3}=\frac{{\sum }_{\sigma }{U}_{\sigma }}{(\frac{N_1}{n_1})(\frac{N_2}{n_2})(\frac{N_3}{n_3})}.$$

What can we say about $U$? Well, if we fix a $v_3\in V_3$, then it will appear in $(\frac{N_1}{n_1})(\frac{N_2}{n_2})(\frac{N_3-1}{n_3-1})$ of the $U_{\sigma }$ terms (i.e. all choices of the $V_1$ and $V_3$ variables, and all possible choices of the other $n_3-1$ variables in the other $N_3-1$ variables in $V_3$). Thus $v_3$ appears in $U$ as an additional term which is

$$-e^{-v_3}\frac{(\frac{N_3-1}{n_3-1})}{(\frac{N_3}{n_3})}=-e^{-v_3}\frac{n_3}{N_3}.$$

How about a fixed $v_2\in V_2$? Well, if $v_2$ goes below $0$, that will kill off $(\frac{N_1}{n_1})(\frac{N_2-1}{n_2-1})(\frac{N_3}{n_3})$ of the $U_{\sigma }$. So if all other $v_2^{\prime }\in V_2$ are positive, sending $v_2$ from negative to positive will multiply the expected value by $1-n_2/N_2$.

Similarly, for fixed $v_1\in V_1$, if all $v_2\in V_2$ are positive, then it will appear in $U$ as $+v_1\left(n_1/N_1\right)$.

So the Bayesian mix is roughly linear in the $V_1$, treats any $v\in V_2$ as loosing $n_2/N_2$ of the utility if it goes negative, and is linear in the negative exponential of the $V_3$.

Being somewhat sloppy

In some ways we cannot afford to be sloppy: assume that $v$ should be in $V_2$ but isn't; so the true utility is $U=U_{V_1,V_2\cup \left\{v\right\},V_3}$, but a $U^{\prime }=U_{V_1,V_2,V_3}$-maximiser might sacrifice $v$ to increase $U^{\prime }$; thus (ignoring the $V_3$) maximising $U^{\prime }$ may set $U$ to $0$.

Conversely, if $v$ should be in $V_3$ but isn't, and the true utility is $U=U_{V_1,V_2,V_3\cup \left\{v\right\}}$. Then a $U^{\prime }$-maximiser might sacrifice $v$ to maximise $U$, at a cost of some multiple of $-e^{-v}$, which might be enormous.

So, just as forgetting a variable triggers a standard Goodhart effect (see the section above), forgetting to assign it to the right category also triggers a Goodhart effect.

On the other hand, if we get the numbers $n_1$, $n_2$, or $n_3$ wrong, then we scale various pieces of the utility by a constant factor. This tends to have a small effect, as compared with the exponential penalty of the $V_3$ or the "kill off $V_1$" effects of the $V_2$.

In the general case, we wouldn't have $V_i$ with $n_i$ of them that matter, but we'd have a probability distribution $p_i$ over the possible variables in $V_i$ that matter (this would be a probability distribution over $2^{V_i}$, the set of subsets of $V_i$). In that case, we can be a bit sloppy with $p_i$ as long as we don't miss out variables we should include.

So, along certain axes, we can be sloppy and imprecise without the system breaking. This is a change from the standard Goodhart/AI alignment situations, where sloppiness almost always guarantees disaster.

Conservatism

If we were being more conservative, instead of maximising $U=U_{V_1,V_2,V_3}$ a weighted sum of all $U_{\sigma }$, we might want to maximise the minimum value of these, and hence the utility function

$$\mu ={\mu }_{V_1,V_2,V_3}=\underset{\sigma }{min}\left(U_{\sigma }\right).$$

Let's compare what maximising $\mu$ and maximising $U$ might get us.

For $V_1$, maximising $\mu$ involves maximising the average of the minimal $n_1$ values of the $V_1$, while $U$ maximises the whole average. Here, $U$ is superior to the $\mu$, since $V_1$ variables are explicitly designed to be traded off linearly against each other - if we had reasons to think they behaved differently when low-valued, they should instead/also be in $V_2$ or $V_3$.

If we have a single $v_3\in V_3$ that gets very negative, $U$ will get a penalty of $-e^{-v_3}\left(n_3/N_3\right)$ while $\mu$ gets a penalty of $-e^{-v_3}$. Up to that constant factor, they behave similarly.

If any $v_2\in V_2$ is negative, then $\mu$ ignores all of $V_1$, while $U$ scales the $V_1$ utility function by $1-\left(n_3/N_3\right)$. Thus $U$ can take the risk of letting some values in $V_2$ go negative (in exchange for enough $V_1$ bonus), while $\mu$ won't take the risk at all. We could see $\mu$ as behaving like $U$ if $n_2=N_2$.

Which is correct there? Most likely, $U$. Imagine if $n_2=1$ while $N_2={10}^{24}$. Then $\mu$ is saying that "there are a trillion trillion of these variables in $V_2$. It's bad for one of them to be negative. However, you should behave as if it's bad for any of them to be negative, and not take that risk."

That's an extreme example; but the point is that $U$ incorporates our knowledge or estimate of tradeoffs, but $\mu$ does not. Since it does not know the tradeoffs, we can set up situations where $\mu$-maximising is not just sub-optimal, but quite dire. For example, if one of the $v_2\in V_2$ is already negative, then the $\mu$-maximiser will accept to reduce ${\sum }_{v_1\in V_1}{v}_1$ to just above zero (no matter how high it was initially) in exchange for an infinitesimal chance of increasing $v_2$ above $0$. So the amount of $U$-value that the $\mu$-maximiser is willing to throw away is potentially unbounded.

Bayesianism versus conservatism

Bayes beats conservatism

Both $\mu$-maximising and $U$-maximising are forms of conservatism; neither of them is just saying "maximise the $v_i$ linearly". When designing our conservative utility, we'd like to incorporate our knowledge about tradeoffs. The $\mu$-style approach does this very badly in many circumstances, and so can lose a lot of value.

Why might we nevertheless use a $\mu$-style approach rather than a $U$-style one? If we feel that there is some circumstance where $U$ could also lose a lot of value, and that we haven't fully accounted for. After all, there may be situations where we don't know the tradeoffs; maybe there are some tradeoffs where $U$-maximising is terrible, and we just haven't thought of them yet.

But why would $\mu$-maximising avoid these problems? Maybe there are theorems about how badly $\mu$-maximising can mess up; since it is based around a minimum, it is plausible that we can bound the downside risk in some circumstances. But if we knew these theorems, we could use them to design a better $U$.

There might be a practical argument for $\mu$ or $U$. Maybe humans are too clever when we try to account for all our genuine preferences, so, in practice, $\mu$-maximising is safer than $U$-maximising. We'd like an AI to take this into account, and automatically create a good $U$, but maybe that's also too clever - maybe we can't program an AI to successfully take into account the biases we have when estimating our own biases.

Still, seeing the definite failures of $\mu$ and the ability to apply most of our $\mu$-insights to $U$-design, there's a strong argument that $U$-maximising would likely be safer than $\mu$-maximising.

Bayes is conservatism

I've been a bit unfair there. I constructed $U$ with some care and effort; I could have made it more realistic, but it was a reasonably complex and involved effort.

When constructing $\mu$, on the other hand, I simply took the previously defined $U_{\sigma }$ and took a hard minimum across them. If I wanted to construct a better $\mu$, I could have put in more effort to define functions over which to take a minimum, taking into account all the likely tradeoffs...

We can see the convergence here: this sounds a lot like building $U$. My argument here, not fully proven, is that if we are constructing any objective function that incorporates all our knowledge, we can do as well or better with a Bayesian mix of utility functions.

If we have an argument for why a non-Bayesian approach (such as a minimum or a smooth minimum) might work better, then that very argument becomes a tool to improve our Bayesian mix. Saying "if we use the Bayesian mix, we have problem $P$" becomes "how bad is $P$, and what would be pay to avoid it?", which modifies and improves the Bayesian mix.

---

1. If we wanted a more gradual transition, we could use $S_0$, the smoothstep/clamping function defined by $S_0\left(x\right)=0$ if $x<0$, $S_0\left(x\right)=1$ if $x>1$, and $S_0\left(x\right)=x$ otherwise. ↩︎