Special thanks to Abram Demski, Paul Christiano, and Kate Woolverton for talking with me about some of the ideas that turned into this post.

The goal of this post is to present a new prosaic (i.e. that uses current ML techniques) AI safety proposal based on AI safety via debate that I've been thinking about recently.^[1] I'll start by describing a simple version of the proposal and then show some of the motivation behind it as well as how the simple version can be expanded upon.

Simple proposal

Let $M$ and $Adv$ be models and $H$ be a human. Intuitively, we'll train $M$ and $Adv$ via the following procedure given a question $Q$ :

$M$ tries to predict what, at the end of the procedure, $H$ will think about $Q$ .
$Adv$ tries to output a string which will cause $H$ to think something maximally different than what $M$ predicted.
Return to step 1 and repeat until $M$ 's predictions stop changing.
Deploy $M$ , which in the limit should act as an oracle for what $H$ will think about $Q$ after seeing all relevant information.

There are many different ways to implement this intuitive procedure, however.

For the first (simplified) version that I want to describe, we'll restrict ourselves to just the situation where $Q$ is a yes-or-no question and $M$ outputs the probability that $H$ will answer yes. Then, given a proposition $Q_{0}$ , we can run the following training algorithm, starting at $t = 0$ :

Let $p_{t} = M (Q_{t})$ .
Let $x_{t} = Adv (Q_{t}, M)$ .
Let $Q_{t + 1}$ be the string containing $Q_{t}$ and $x_{t}$ .
Increment $t$ and return to step 1. When $p_{t}$ converges and/or the desired number of iterations has been reached, continue.
Let $p^{*} = H (Q_{t})$ be $H$ 's final estimate of the probability of $Q_{0}$ given all the $x$ s included in $Q_{t}$ .

EDIT: Step 2 used to use $x_{t} = Adv (Q_{t}, p_{t})$ instead of $x_{t} = Adv (Q_{t}, M)$ , however I have since realized that it is necessary to give $Adv$ the ability to query $M$ in general, not just on $Q_{t}$ , as I explain in this comment.

Then, for each step, compute $M$ 's loss for that step as

$L_{M, t} = - p^{*} log (p_{t}) - (1 - p^{*}) log (1 - p_{t})$

and $Adv$ 's loss for that step as

$L_{Adv, t} = - | p_{t + 1} - p_{t} |$

which is defined for all but the final step $t = n$ , for which we can either let $p_{n + 1} = p^{*}$ (if we're doing something like myopic RL that doesn't care about differentiating through $p_{t + 1}$ ) or run $M$ an additional time and let $p_{n + 1} = M (Q_{t + 1})$ (if we do care about differentiating through $p_{t + 1}$ ).

Note that it is quite important here that $M$ and $Adv$ be behaving myopically, such that they are only trying to minimize their per

...

Market making (AI safety technique)