I propose a new formal desideratum for alignment: the Hippocratic principle. Informally the principle says: an AI shouldn't make things worse compared to letting the user handle them on their own, in expectation w.r.t. the user's beliefs. This is similar to the dangerousness bound I talked about before, and is also related to corrigibility. This principle can be motivated as follows. Suppose your options are (i) run a Hippocratic AI you already have and (ii) continue thinking about other AI designs. Then, by the principle itself, (i) is at least as good as (ii) (from your subjective perspective).

More formally, we consider a (some extension of) delegative IRL setting (i.e. there is a single set of input/output channels the control of which can be toggled between the user and the AI by the AI). Let ${\pi }_u^{\upsilon }$ be the the user's policy in universe $\upsilon$ and ${\pi }_a$ the AI policy. Let $T$ be some event that designates when we measure the outcome / terminate the experiment, which is supposed to happen with probability $1$ for any policy. Let $V^{\upsilon }$ be the value of a state from the user's subjective POV, in universe $\upsilon$. Let ${\mu }^{\upsilon }$ be the environment in universe $\upsilon$. Finally, let $\zeta$ be the AI's prior over universes and $ϵ$ some sufficiently small bound. We require

$$\forall T:E_{\upsilon \sim \zeta }\left[\right(E_{{\mu }^{\upsilon }}^{{\pi }_u^{\upsilon }}\left[V^{\upsilon }\right(T\left)\right]-E_{{\mu }^{\upsilon }}^{{\pi }_a}\left[V^{\upsilon }\right(T\left)\right]{)}_{+}]\le ϵ$$

Here, $V\left(T\right)$ designates the value after event $T$ happens, and $(x{)}_{+}$ is defined to be $0$ for $x<0$ and $x$ otherwise.

I now outline an example of a Hippocratic protocol, which I call Hippocratic timeline-driven learning (HTDL).

Suppose, for simplicity, that the goal is accomplishing some binary task (i.e. we can either succeed or fail). We can imagine the task to be something like "construct a defense system against malign AI". At any given moment, the user reports their subjective timeline: i.e. for, any $t$, the probabilities of success and failure after time $t$, assuming the user will continue working on the task by themselves. We assume the AI can learn a good model of the user's policy and the user's nominal report, but this nominal report might deviate from the true timeline because the system can be corrupted (i.e. the user manipulated to enter false information or the input channel hacked). Moreover, we assume that such corruption counts as failing the task.

The AI operates by maximizing the AI-expectation of the user's (unknown) utility function $E_{\upsilon \sim \zeta }\left[E_{{\mu }^{\upsilon }}^{\pi }\right[U^{\upsilon }\left]\right]$, subject to the Hippocratic principle as a constraint. Since we're working in a delegative setting, the AI can trivially achieve the constraint by always delegating. However, it can do much better than that, by short-term quantilizing w.r.t. the user's reported success probability (with the user's policy serving as baseline). When quantilizing the short-term policy, we can upper bound the probability of corruption via the user's reported probability of short-term failure (which we assume to be low, i.e. we assume the malign AI is not imminent). This allows the AI to find parameters under which quantilization is guaranteed to improve things in expectation. Then, after completing this short-term maneuver, the AI can do another short-term quantilization, and continue in this vein. Essentially, the AI is gradually bending the user's trajectory towards success.

Of course, we can improve the guarantees of such a protocol either by adding more information or by using stronger assumptions (e.g. assume the user is quasi-rational). But timeline-driven short-term quantilization serves as a useful feasibility proof.