The most fundamental reason that I don't expect this to work is that it gives up on "sharing parameters" between the extractor and the human model. But in many cases it seems possible to do so, and giving on up on that feels extremely unstable since it's trying to push against competitiveness (i.e. the model will want to find some way to save those parameters, and you don't want your intended solution to involve subverting that natural pressure).

Intuitively, I can imagine three kinds of approaches to doing this parameter sharing:

1. Introduce some latent structure $L$ (e.g. semantics of natural language, what a cat "actually is") that is used to represent both humans and the intended question-answering policy. This is the diagram $H\leftarrow L\to f^{+}$
2. Introduce some consistency check $f^{?}$ between $H$ and $L$. This is the diagram $H\to f^{?}\leftarrow f^{+}$
3. Somehow extract $f^{+}$ from $H$ or build it out of pieces derived from $H$. This is the diagram $H\to f^{+}$. This is kind of like a special case of 1, but it feels pretty different.

(You could imagine having slightly more general diagrams corresponding to any sort of d-connection between $H$ and $f^{+}$.)

Approach 1 is the most intuitive, and it seems appealing because we can basically leave it up to the model to introduce the factorization (and it feels like there is a good chance that it will happen completely automatically). There are basically two challenges with this approach:

• It's not clear that we can actually jointly compress $H$ and $f^{+}$. For example, what if we represent $H$ in an extremely low level way as a bunch of neurons firing; the neurons are connected in a complicated and messy way that learned to implement something like $f^{+}$, but need not have any simple representation in terms of $f^{+}$. Even if such a factorization is possible, it's completely unclear how to argue about how hard it is to learn. This is a lot of what motivates the compression-based approaches---we can just say "$H$ is some mess, but you can count on it basically computing $f^{+}$" and then make simple arguments about competitiveness (it's basically just as hard as separately learning $H$ and $f^{+}$).
• If you overcame that difficulty, you'd still have to actually incentivize this kind of factorization in the model (rather than sharing parameters in the unintended way). It's unclear how to do that (maybe you're back to think about something speed-prior like, and this is just a way to address my concern about the speed-prior-like proposals), but this feels more tractable than the first problem.

I've been thinking about approach 2 over the last 2 months. My biggest concern is that it feels like you have to pay the bits of H back "as you learn them" with SGD, but you may learn them in such a way that you don't really get a useful consistency update until you've basically specified all of H. (E.g. suppose you are exposed to brain scans of humans for a long time before you learn to answer questions in a human-like way. Then at the end you want to use that to pay back the bits of the brain scans, but in order to do so you need to imagine lots of different ways the brain scans could have looked. But there's no tractable way to do that, because you have to fill in the the full brain scan before it really tells you about whether your consistency condition holds.)

Approach 3 is in some sense most direct. I think this naturally looks like imitative generalization, where you use a richer set of human answers to basically build $f^{+}$ on top of your model. I don't see how to make this kind of thing work totally on its own, but I'm probably going to spend a bit of time thinking about how to combine it with approaches 1 and 2.

How much hope is there for jointly representing $f_{{\theta }_1}$ and $H_{{\theta }_2}$?

The most obvious representation in this case is to first specify $f_{{\theta }_1}$, and then actually model the process of gradient descent that produces $H_{{\theta }_2}$. This runs into a few problems:

1. Actually running gradient descent to find ${\theta }_2$ is too expensive to do at every datapoint---instead we learn a hypothesis that does a lot of its work "up front" (shared across all the datapoints). I don't know what that would look like. The naive ways of doing it (redoing the shared initial computation for every batch) only work for very large batch sizes, which may well be above the critical batch size. If this was the only problem I'd feel pretty optimistic about spending a bunch of time thinking about it.
    
2. Specifying "the process that produced $H_{{\theta }_2}$" requires specifying the initialization ${\tilde{\theta }}_2$, which is as big as ${\theta }_2$. That said, the fact that the learned ${\theta }_2$ also contains a bunch of information about ${\theta }_1$ means that it can't contain perfect information about the initialization ${\tilde{\theta }}_2$, i.e. that multiple initializations lead to exactly the same final state. So that suggests a possible out: we can start with an "initial" initialization ${\tilde{\theta }}_2^0$. Then we can learn ${\tilde{\theta }}_2$ by gradient descent. The fact that many different values of ${\tilde{\theta }}_2$ would work suggests that it should be easier to find one of them; intuitively if we set up training just right it seems like we may be able to get all the bits back.
    
3. Running the gradient descent to find ${\theta }_2$ even a single time may be much more expensive than the rest of training. That is, human learning (perhaps extended over biological or cultural evolution) may take much more time than machine learning. If this the case, then any approach that relies on reproducing that learning is completely doomed.
   
   Similar to problem 2, the mutual information between ${\theta }_2$ and the learning process that produced ${\theta }_2$ also must be kind of low---there are only $\left|{\theta }_2\right|$ bits of mutual information to go around between the learning process, its initialization, and ${\theta }_1$. But exploiting this structure seems really hard, if actually there aren't any fast learning processes that lead to the same conclusion

My current take is that even in the case where ${\theta }_2$ was actually produced by something like SGD, we can't actually exploit that fact to produce a direct, causally-accurate representation ${\theta }_2$.

That's kind of similar to what happens in my current proposal though: instead we use the learning process embedded inside the broader world-model learning. (Or a new learning process that we create from fresh to estimate the specialness of ${\theta }_2$, as remarked in the sibling comment.)

So then the critical question is not "do we have enough time to reproduce the learning process that lead to ${\theta }_2$?" it is "Can we directly learn $H_{{\theta }_2}$ as an approximation to $f_{{\theta }_1}$?" If we able to do this in any way, then we can use that to help compress ${\theta }_2$. In the other proposal, we can use it to help estimate the specialness of ${\theta }_2$ in order to determine how many bits we get back---it's starting to feel like these things aren't so different anyway.

Fully learning the whole human-model seems impossible---after all, humans may have learned things that are more sophisticated then what we can learn with SGD (even if SGD learned a policy with "enough bits" to represent ${\theta }_2$, so that it could memorize them one by one if it saw the brain scans or whatever).

So we could try to do something like "learning just the part of the human policy is that is about answering questions." But it's not clear to me how you could disentangle this from all the rest of the complicated stuff going in for the human.

Overall this seems like a pretty tricky case. The high-level summary is something like: "The model is able to learn to imitate humans by making detailed observations about humans, but we are not able to learn a similarly-good human model from scratch given data about what the human is 'trying' to do or how they interpret language." Under these conditions it seems particularly challenging to either jointly represent $H_{{\theta }_2}$ and $f_{{\theta }_1}$, or to compute how many bits you should "get back" based on a consistency condition between them. I expect it's going to be reasonably obvious what to do in this case (likely exploiting the presumed limitation of our learning process), which is what I'll be thinking about now.

The difficulty of jointly representing $f_{{\theta }_1}$ and $H_{{\theta }_2}$ motivates my recent proposal, which avoids any such explicit representation. Instead it separately specifies ${\theta }_1$ and ${\theta }_2$, and then "gets back" bits by imposing a consistency condition that would have been satisfied only for a very small fraction of possible ${\theta }_2$'s (roughly $\exp (-|{\theta }_1\left|\right)$ of them).

But thinking about this neural network case also makes it easy to talk about why my recent proposal could run into severe computational problems:

• In order to calculate this loss function we need to evaluate how "special" ${\theta }_2$ is, i.e. how small is the fraction of ${\theta }_2$'s that are consistent with ${\theta }_1$
• In order to evaluate how special ${\theta }_2$ is, we basically need to do the same process of SGD that produces ${\theta }_2$---then we can compare the actual iterates to all of the places that it could have gone in a different direction, and conclude that almost all of the different settings of the parameters would have been much less consistent with ${\theta }_1$.
• The implicit hope of my proposal is that the outer neural network is learning its human model using something like SGD, and so it can do this specialness-calculation for free---it will be considering lots of different human-models, and it can observe that almost all of them are much less consistent with ${\theta }_1$.
• But the outer neural network could learn to model humans in a very different way, which may not involve representing a serious of iterates of "plausible alternative human models." For example, suppose that in each datapoint we observe a few of the bits of ${\theta }_2$ directly (e.g. by looking at a brain scan), and we fill in much of ${\theta }_2$ in this way before we ever start making good predictions about human behavior. Then we never need to consider any other plausible human-models.

So in order to salvage a proposal like this, it seems like (at a minimum) the "specialness evaluation" needs to take place separately from the main learning of the human model, using a very different process (where we consider lots of different human models and see that it's actually quite hard to find one that is similarly-consistent with ${\theta }_1$). This would take place at the point where the outer model started actually using its human model $H_{{\theta }_2}$ in order to answer questions.

I don't really know what that would look like or if it's possible to make anything like that work.

This is interesting to me for two reasons:

• [Mainly] Several proposals for avoiding the instrumental policy work by penalizing computation. But I have a really shaky philosophical grip on why that's a reasonable thing to do, and so all of those solutions end up feeling weird to me. I can still evaluate them based on what works on concrete examples, but things are slippery enough that plan A is getting a handle on why this is a good idea.
• In the long run I expect to have to handle learned optimizers by having the outer optimizer instead directly learn whatever the inner optimizer would have learned.  This is an interesting setting to look at how that works out. (For example, in this case the outer optimizer just needs to be able to represent the hypothesis "There is a program that has property P and runs in time T' " and then do its own search over that space of faster programs.)

In traditional settings, we are searching for a program M that is simpler than the property P. For example, the number of parameters in our model should be smaller than the size of the dataset we are trying to fit if we want the model to generalize. (This isn't true for modern DL because of subtleties with SGD optimizing imperfectly and implicit regularization and so on, but spiritually I think it's still fine..)

But this breaks down if we start doing something like imposing consistency checks and hoping that those change the result of learning. Intuitively it's also often not true for scientific explanations---even simple properties can be surprising and require explanation, and can be used to support theories that are much more complex than the observation itself.

Some thoughts:

1. It's quite plausible that in these cases we want to be doing something other than searching over programs. This is pretty clear in the "scientific explanation" case, and maybe it's the way to go for the kinds of alignment problems I've been thinking about recently.
   
   A basic challenge with searching over programs is that we have to interpret the other data. For example, if "correspondence between two models of physics" is some kind of different object like a description in natural language, then some amplified human is going to have to be thinking about that correspondence to see if it explains the facts. If we search over correspondences, some of them will be "attacks" on the human that basically convince them to run a general computation in order to explain the data. So we have two options: (i) perfectly harden the evaluation process against such attacks, (ii) try to ensure that there is always some way to just directly do whatever the attacker convinced the human to do. But (i) seems quite hard, and (ii) basically requires us to put all of the generic programs in our search space.
    
2. It's also quite plausible that we'll just give up on things like consistency conditions. But those come up frequently enough in intuitive alignment schemes that I at least want to give them a fair shake.

This is also a way to think about the proposals in this post and the reply:

• The human believes that A' and B' are related in a certain way for simple+fundamental reasons.
• On the training distribution, all of the functions we are considering reproduce the expected relationship. However, the reason that they reproduce the expected relationship is quite different.
• For the intended function, you can verify this relationship by looking at the link (A --> B) and the coarse-graining applied to A and B, and verify that the probabilities work out. (That is, I can replace all of the rest of the computational graph with nonsense, or independent samples, and get the same relationship.)
• For the bad function, you have to look at basically the whole graph. That is, it's not the case that the human's beliefs about A' and B' have the right relationship for arbitrary Ys, they only have the right relationship for a very particular distribution of Ys. So to see that A' and B' have the right relationship, we need to simulate the actual underlying dynamics where A --> B, since that creates the correlations in Y that actually lead to the expected correlations between A' and B'.
• It seems like we believe not only that A' and B' are related in a certain way, but that the relationship should be for simple reasons, and so there's a real sense in which it's a bad sign if we need to do a ton of extra compute to verify that relationship. I still don't have a great handle on that kind of argument. I suspect it won't ultimately come down to "faster is better," though as a heuristic that seems to work surprisingly well. I think that this feels a bit more plausible to me as a story for why faster would be better (but only a bit).
• It's not always going to be quite this cut and dried---depending on the structure of the human beliefs we may automatically get the desired relationship between A' and B'. But if that's the case then one of the other relationships will be a contingent fact about Y---we can't reproduce all of the expected relationships for arbitrary Y, since our model presumably makes some substantive predictions about Y and if those predictions are violated we will break some of our inferences.