Note: This post came out of a conversation with Geoffrey Irving and Buck Shlegeris.

Epistemic Status: I suspect Paul has already thought of most or all of the ideas presented here, though I nevertheless found the exercise of carefully specifying an IDA implementation helpful and suspect others may find reading it helpful as well.

This is a proposal for how to train a machine learning model to approximate HCH using Iterated Distillation and Amplification (IDA). This particular proposal came out of a desire to use a debate-like adversary to improve the amplification process, and the primary goal of this proposal is to show how one could do that. Though I have tried to retain a lot of the relevant detail, I have made two simplifications to make this proposal easier to specify: I am attempting to approximate something closer to weak HCH rather than strong HCH and I am only allowing the generation of two subquestions at a time. I am confident that those simplifications could easily be dropped, though I think doing so here would only make this presentation more complicated.

Before I proceed, I want to make one final note: this is not a proposal for how to build an aligned AGI. I think there are still a whole bunch of issues that would prevent this proposal from actually working.

Definitions

We will start with some initial definitions:

• Let $Q$ be the set of all questions in natural language.
• Let $A$ be the set of all answers in natural language.
• Let $M$ be the sum type of either $Q\times Q$ or $A$ representing either an answer to the given question or two subquestions to help answer it.
• Let $H:Q\to A$ be the answer that a human gives to the given question.
• Let $H_{\text{fan out}}:Q\to M$ be the answer or subquestion pair generated by a human when asked what to do with the given question.
• Let $H_{\text{fan in}}:Q\times (Q\times A)\times (Q\times A)\to M$ be the answer or two subquestions generated by a human to some question when given answers to two subquestions related to that question.
• Let $ML:Q\to \Delta \left(A\right)$ be a model (the training procedure for which we will describe below) from questions to a probability distribution over strings representing answers. Specifically, we will implement the probability distribution by having our model output an embedding vector which yields the probability distribution when fed into some trained language model (by repeatedly conditioning on previous characters and multiplying all the conditional probabilities).
• Let $M{L}_{\text{fan out}}:Q\to \Delta \left(M\right)$ be the function from questions to an embedding vector representing a distribution over answers or subquestions generated by asking $ML$ what to do with the given question. For the two subquestion case we enforce that the distribution is symmetric wrt interchange of the subquestions.
• Let $M{L}_{\text{fan in}}:Q\times (Q\times A)\times (Q\times A)\to \Delta \left(M\right)$ be the function from two answers to an embedding vector representing a distribution over answers or subquestions generated by asking $ML$ to integrate the given subquestion answers into an answer to the original question. We again enforce symmetry wrt interchange of the subquestions.
• Let $\text{Adv}:Q\times A\to R$ be an adversary model which we will train (as described below) to predict how good the given answer is to the given question.
• Let ${\text{Adv}}_{\text{fan out}}:Q\times M\to R$ be an adversary model for $M{L}_{\text{fan out}}$ generated by calling $\text{Adv}$ using the same transformation as used to implement $M{L}_{\text{fan out}}$ from $ML$.
• Let ${\text{Adv}}_{\text{fan in}}:Q\times (Q\times A)\times (Q\times A)\times M\to R$ be an adversary model for $M{L}_{\text{fan in}}$ generated by calling $\text{Adv}$ using the same transformation as used to implement $M{L}_{\text{fan in}}$ from $ML$.
• Let $ϵ$ be some Gumbel random variable such that each use of $ϵ$ below is its own independent sample from a Gumbel distribution. We use a Gumbel random variable so that we can use the Gumbel-max trick to effectively do Boltzmann exploration.

Amplification

The goal here is to use the adversary to determine which portions of the HCH tree we actually go down to maximize our sample efficiency wrt $H$. Specifically, the amplification procedure is as follows:

1. Generate some question $Q\in Q$ either from the training distribution or from the output of some adversary.

2. Call $H_{\text{fan out}}\left(Q\right)$. If we get an answer, let $A=H_{\text{fan out}}\left(Q\right)$ and skip to the end. Otherwise, let $S{Q}_1,S{Q}_2=H_{\text{fan out}}\left(Q\right)$.

3. Call $\text{Adv}$ to generate scores $S_1=\text{Adv}(S{Q}_1,ML(S{Q}_1\left)\right)$ and $S_2=\text{Adv}(S{Q}_2,ML(S{Q}_2\left)\right)$.

4. Without loss of generality, let $S_1+ϵ\le S_2+ϵ$ and recurse on $S{Q}_1$ (by setting $Q=S{Q}_1$ in the subcall) to generate $A_1$ (from the $A$ of the subcall) and let $A_2=ML\left(S{Q}_2\right)$.

5. Call $H_{\text{fan in}}(Q,(Q_1,A_1),(Q_2,A_2\left)\right)$. If we get an answer, let $A=H_{\text{fan in}}(Q,(Q_1,A_1),(Q_2,A_2\left)\right)$. Otherwise, let $S{Q}_1,S{Q}_2=H_{\text{fan in}}(Q,(Q_1,A_1),(Q_2,A_2\left)\right)$ and return to step 3.

This procedure should eventually produce an answer, though we then need to compute the following losses for $ML,M{L}_{\text{fan out}},M{L}_{\text{fan in}},\text{Adv},{\text{Adv}}_{\text{fan out}},{\text{Adv}}_{\text{fan in}}$:

Note that some of these losses are actually for the same model since $M{L}_{\text{fan out}},M{L}_{\text{fan in}}$ are implemented internally by calling $ML$ and ${\text{Adv}}_{\text{fan out}},{\text{Adv}}_{\text{fan in}}$ are implemented internally by calling $\text{Adv}$.

Distillation

For the distillation procedure, we want to improve $ML$ by training it to approximate its own $ML$-consulting-$ML$ tree. Specifically:

1. Generate some $Q\in Q$ as in amplification.

2. Call $M{L}_{\text{fan out}}\left(Q\right)$. If we get an answer, let $A=M{L}_{\text{fan out}}\left(Q\right)$ and skip to the end. Otherwise, let $S{Q}_1,S{Q}_2=M{L}_{\text{fan out}}\left(Q\right)$.

3. Let $A_1=ML\left(S{Q}_1\right)$ and $A_2=ML\left(S{Q}_2\right)$.

4. Call $M{L}_{\text{fan in}}(Q,(Q_1,A_1),(Q_2,A_2\left)\right)$. If we get an answer, let $A=M{L}_{\text{fan in}}(Q,(Q_1,A_1),(Q_2,A_2\left)\right)$. Otherwise, let $S{Q}_1,S{Q}_2=M{L}_{\text{fan in}}(Q,(Q_1,A_1),(Q_2,A_2\left)\right)$ and return to step 3.

Then, compute the following losses for $ML,\text{Adv}$:

Iteration

Finally, we want to iterate the above procedure by replacing $H,H_{\text{fan out}},H_{\text{fan in}}$ in the amplification procedure with some $H^{\prime },H_{\text{fan out}}^{\prime },H_{\text{fan in}}^{\prime }$. First, let $\alpha$ be some confidence threshold. Then, we will define the following primed $H$s:

• Let $H^{\prime }\left(Q\right)=\text{if Adv}(Q,ML(Q\left)\right)+ϵ<\alpha +ϵ\text{then}H\left(Q\right)\text{else}ML\left(Q\right)$.
• Let

• Let

This procedure allows us to continue amplifying the model while using the adversary to require only minimal human data that is selected so as to be maximally helpful.

Conclusion

This proposal differs in a couple of ways from previous proposals made by Paul. First, Paul has recently moved away from discrete amplification/distillation steps. This proposal, however, provides a way to recover discrete steps while still collapsing the recursion. In practice, however, you might still just want to stick with the amplification procedure described here without doing the distillation step, as it isn't strictly necessary.

Second, this proposal uses an adversary to guide the training process. This technique is similar to the concept of importance sampling. The main benefit of this approach is that it takes advantage of active learning by allowing the system to choose which questions and subquestions would be most useful for it to have answered by a human.

Another benefit of the adversary, however, is that it could make transparency much easier. One of the major benefits of IDA is that $ML$ gets trained to approximate its own $ML$-consulting-$ML$ tree. As a result, understanding the reasoning that went into the final answer produced by $ML$ can be recovered by unfolding its tree (at least in the limit of perfect training). However, unfolding the entire tree is very expensive, as it's linear in the size of the tree. With an adversary, however, you can choose which portions of the tree to unfold first by calling the adversary, enabling you to find errors much more quickly; for a perfect adversary, this reduces the problem of finding an error to $O\left(\log n\right)$ instead of $O\left(n\right)$.

Thus, the hope is that the use of such an adversary could assist both in making IDA more competitive (by increasing sample efficiency and using active learning) and in making IDA safer (due to the increased ease of transparency).

It should be noted, however, that it is also possible that the use of such an adversary might make the safety situation for IDA worse. First, it introduces the possibility of a robustness to relative scale failure if either $ML$ or $\text{Adv}$ gets significantly stronger than the other. One possible way to resolve such an issue, however, might be to give $\text{Adv}$ the ability to call $ML$ and vice versa, allowing them to use each other to boost their own capabilities. Second, for an $ML$ and $\text{Adv}$ system that are themselves optimizers, with goals that don't perfectly match up with their loss functions, they could cooperate to make it arbitrarily unlikely that $H$ is ever consulted on some specific question. Third, even if $ML$ and $\text{Adv}$ weren't cooperating, an RSA-2048-style failure could still prevent the identification of malicious cognition. Resolving failures of these second two types is still an open question (EDIT: see "Risks from Learned Optimization in Advanced Machine Learning Systems," by Hubinger, van Merwijk, Mikulik, Skalse, and Garrabrant).