Suppose I have a magic box $O p t$ that takes as input a program $U : {0, 1}^{n} \to R$ , and produces $O p t (U) = {a r g m a x}_{x} U (x)$ , with only $n$ times the cost of a single evaluation of $U$ . Could we use this box to build an aligned AI, or would broad access to such a box result in doom?

This capability is vaguely similar to modern ML, especially if we use $O p t$ to search over programs. But I think we can learn something from studying simpler models.

An unaligned benchmark

(Related.)

I can use $O p t$ to define a simple unaligned AI (details omitted):

Collect data from a whole bunch of sensors, including a "reward channel."
Use $O p t$ to find a program $M$ that makes good predictions about that data.
Use $O p t$ to find a policy $π$ that achieves a high reward when interacting with $M$ .

This isn't a great design, but it works as a benchmark. Can we build an aligned AI that is equally competent?

(I haven't described how $O p t$ works for stochastic programs. The most natural definition is a bit complicated, but the details don't seem to matter much. You can just imagine that it returns a random $x$ that is within one standard deviation of the optimal expected value.)

Competing with the benchmark

(Related.)

If I run this system with a long time horizon and a hard-to-influence reward channel, then it may competently acquire influence in order to achieve a high reward.

We'd like to use $O p t$ to build an AI that acquires influence just as effectively, but will use that influence to give us security and resources to reflect and grow wiser, and remain responsive to our instructions.

We'd like the aligned AI to be almost as efficient. Ideally the proportional overhead would converge to 0 as we consider more complex models. At worst the overhead should be a constant factor.

Possible approach

(Related.)

My hope is to use $O p t$ to learn a policy $π^{+}$ which can answer questions in a way that reflects "everything $π$ knows." This requires:

Setting up an objective that incentivizes $π^{+}$ to give good answers to questions.
Arguing that there exists a suitable policy $π^{+}$ that is only slightly more complicated than $π$ .

If we have such a $π^{+}$ , then we can use it to directly answer questions like "What's the best thing to do in this situation?" The hope is:

Its answers can leverage everything $π$ knows, and in particular all of $π$ 's knowledge about how to acquire influence. So using $π^{+}$ in this way

...