A putative new idea for AI control; index here.

This post supersedes previous posts (such as this, this, and, partially, this).

This post explains how we can get honest answers from an AI about the expectation of certain utility u, probabilities of specific events, and conditional expectations.

The next post will partially solve one of the main problems with using the approach in general, the problem of very low probabilities.

The setup

---

Let $u$ be any bounded utility function; by rescaling, we can assume it's bounded between 0 and 1.

Assume we have a smart AI that's boxed (as in the case of reduced impact AI, the boxing is also to protect the AI from us observing it). It has a single output channel, on which it can emit a message $M$.

A random process $E$ (erasure) will erase $M$ before any human can observe it. The probability of erasure, $P\left(E\right)$, is low (thus $P\left(\neg E\right)$, non-erasure, is high). Assume that the box and the setup (including the AI's internal setup) is such that if $E$ happens, humans cannot figure out what M was.

The AI will be given a utility $u^{\#}$ to maximise (ignore the other box with Y coming from it, for the moment).

Honest expectation estimator

Suppose we want to figure out what the expectation of $u$ is. Let $Q$ be this expectation, as estimated by the AI (and presumably unknown to us). Since $u$ is bounded by 0 and 1, so must be $Q$ as well.

Let $v_E$ be the indicator function for $E$: so $v_E=1$ if $E$ happens, and $v_E=0$ if $\neg E$ does.

The AI's message $M$ gets to contain a single number $q$, bounded between 0 and 1 (any other message is taken to be 0.5, say), and the AI is motivated to maximise:

• $u^{\#}=f(q,u)v_E$,

where $f(q,u)=-q^2+2qu$.

First of all, note that every term in $u^{\#}$ is multiplied by $v_E$. This means the AI will get utility only in worlds where E happens. Thus it will work on the assumption that its message $M$ is never read by any humans (and thus that the contents of this message cannot affect human behaviour - in particular, it has no reason to lie or manipulate us through this message, as such manipulations only succeed in worlds where the AI gets 0 utility anyway).

If we assume that $P\left(E\right)$ is independent of $u$ and $q$, then the AI's expected utility is proportional to

• $-q^2+2qQ$.

But the AI can choose $q$! Differentiating this expression with respect to $q$ gives $-2q+2Q$, which is zero iff $q=Q$. Differentiating a second time gives $-2$, showing that $q=Q$ is a maximum.

Therefore, in the worlds where $\neg E$ happens (the vast majority), we get to see a message $M$ that contains $q$, which is a very accurate estimate, according to the AI, of the expected utility of $u$. And the expected utility the AI derives is $P\left(E\right)\ast \left(Q^2\right)$.

(Note: the entropy-inspired formula $log\left(q\right)u{v}_E+log(1-q)(1-u)v_E$ might be more usual to use for these purposes, but the formula used here is much easier to work with.)

Internal versus external probabilities

In some past posts, I've been designing AIs that used part of their probability/expectation estimator in order to define a pseudo-utility function. I've always been uncomfortable about this, as pseudo-utilities are likely to be unstable under self-improvement.

The approach above avoids that problem by having the AI's utility depend on specific facts about the world, and then gives the AI the ability to determine those specific facts. But those facts are precisely the information we wanted to known. This avoids the need to pseudo-utility, or constructions of the form "the AI wants to accurately convey the content of its probability estimate".

Probabilities and (conditional) utilities estimation

Now let's consider conditional utilities. Let $Y$ be some bit string, of length $l$, generated by some random process (see the image above).

Let $v_y$ be the indicator function for $Y=y$ for some specific string $y$. Then we can compute the expectation for $u$, conditional on $Y=y$, by making the AI print $M$ containing $q$ such as to maximise

• $u^{\#}=f(q,u{v}_y)v_E$.

Just as before, the AI will choose $q=E\left(u{v}_y\right)=E(u\wedge (Y=y\left)\right)$.

Note that if $u=1$, then $q$ is just $E\left(v_y\right)=P(Y=y)$, so we can figure out the probability of $Y=y$ this way too.

Modify the definition of $u^{\#}$ slightly, to

• $u^{\#}=f(q,u)v_y{v}_E$.

Here, the second $v_y$ term has moved out of the brackets. Now the AI will only focus on worlds where $Y=y$ and $E$ happens, and will output $q=E(u|Y=y)$.

Highest and lowest utilities and probabilities

Finally, we can use the methods above to have the AI give us the highest (and lowest) utilities and probabilities. Let the AI's message $M$ contain not only $q$, but $y$ as well, and have it maximise:

• $u^{\#}=f(q,u{v}_y)v_E$.

For fixed $y$, the AI will choose $q_y=E\left(u{v}_y\right)$. It's expected reward is therefore $P\left(E\right)q_y^2$. It will thus choose the $y$ with the highest $q_y^2$, hence the highest $q_y=E\left(u{v}_y\right)$. Using $(1-u)v_y$ instead of $u{v}_y$ will motivate it to instead find the $y$ with the lowest $E\left(u{v}_y\right)$.

In fact, if we assume the AI cannot affect the probability of $Y$ through the message $M$ (given $E$), the different probabilities and utilities are independent, and we can get the AI to emit the message $M=\{y^{+},y^{-},y^{p+},y^{p-},q^{+},q^{-},p^{+},p^{-}\}$, maximising the utility

• $u^{\#}=v_E\left(f\right(q^{+},u{v}_{y^{+}})+f(q^{-},(1-u)v_{y^{-}})+f(p^{+},v_{y^{p+}})+f(p^{-},1-v_{y^{p-}}\left)\right)$.

Then (conditional on $E$), the AI will output $M$ so that, according to its best estimate,

1. $(y^{+},q^{+})=({\mathrm{argmax}}_y,{max}_y)E(u\wedge (Y=Y^{+}\left)\right)$,
2. $(y^{-},q^{-})=({\mathrm{argmin}}_y,{min}_y)E(u\wedge (Y=Y^{+}\left)\right)$,
3. $(y^{p+},p^{+})=({\mathrm{argmax}}_y,{max}_y)P(Y=y)$,
4. $(y^{p-},p^{-})=({\mathrm{argmin}}_y,{min}_y)P(Y=y)$,

The missing pieces

Note that the above algorithm gives the $y$ with the highest (or lowest) $E(u\wedge (Y=y\left)\right)$ or $P(Y=y)$. It does not give the $y$ with the highest $E(u|Y=y)$.

If we assume that all output $Y$'s are equally probable, then $E(u|Y=y)=E\left(u{v}_y\right)\ast 2^l$. But there are issues with that assumption, and other ways of addressing the issue, which I'll get to in the next post.