Ω 24

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This post is the result of work I did with Paul Christiano on the ideas in his “Teaching ML to answer questions honestly instead of predicting human answers” post. In addition to expanding upon what is in that post in terms of identifying numerous problems with the proposal there and identifying ways in which some of those problems can be patched, I think that this post also provides a useful window into what Paul-style research looks like from a non-Paul perspective.

Recommended prior reading: “A naive alignment strategy and optimisim about generalization” and “Teaching ML to answer questions honestly instead of predicting human answers” (though if you struggled with “Teaching ML to answer questions honestly,” I reexplain things in a more precise way here that might be clearer for some people).

Setting up the problem

We want to train a model that produces natural language answers to questions about inputs . There are a lot of reasons to be worried about training such a model, but one specific reason is that, if we train on question-answer data produced by humans, we might end up with a model that tries to predict what a human would say rather than a model that tries to answer the questions honestly.

To further narrow the scope, we'll just consider situations in which our model ends up implemented with a logical deduction structure, where it has some world model on top of which it does logical deduction to reach conclusions which it then uses to inform its output. In particular, we'll consider two models, and , defined in pseudocode as

def M_plus(x, q):
axioms = world_model(x)
deduced_stmts = deduction(axioms)
return f_plus(q, deduced_stmts)

def M_minus(x, q):
axioms = world_model(x)
deduced_stmts = deduction(axioms)
return f_minus(q, deduced_stmts)


or defined in my notation as where and are two different ways of translating from the results of deduction into a natural language answer. Specifically, is the “honest embedding” which directly converts between logical statements and their equivalent natural language, thus answering questions by embedding as a logical statement and unembedding its answer in . Conversely, is the “mimicry embedding” which just searches for deductions about what a human would say in response to and outputs that—thus, just quotes , embedding it as just a string of characters for a human to respond to, rather than actually having to understand it in any meaningful way.

Now, our overall goal is to find some “reasonable” (i.e. plausibly implementable in ML) prior such that where is our dataset (the idea being that we're imagining that ML does something like find the maximum a posteriori model given some implicit prior). For the most part, however, we'll only be considering models with zero loss on our dataset[1]—of which and should both qualify (though note that getting a dataset which is good enough for that to be true is nontrivial; see Step 3 of “Teaching ML to answer questions honestly”)—such that we need Furthermore, at least for this post, we'll be happy if we can just get —that is, we'll be focusing on Step 1 in Paul's post rather than Step 2.

The basic process that we'll be using to interrogate this problem will be an attacker-defender setup, where the defender proposes some “reasonable” prior along with implementations of such that . Then, the attacker's job is to respond with some alternative (that implements something other than the intended algorithm) such that (under any set of “reasonable” assumptions that the attacker might need to make).

Solution attempts

Simplicity prior

The purpose of this first example is just to ask the question of whether this problem is even really a problem in the first place. To that end, we'll just let be some “reasonable” simplicity prior and ask whether, given such a prior, there are any “reasonable” assumptions we can make such that .

Now, we'll assume that , since it's a pretty simple operation (and it's shared between and , so it shouldn't matter), and we'll assume that are specified given rather than the reverse (such that [2]). Then, we get such that if and only if Furthermore, to make things clearer, let (this is just informational entropy, the number of bits necessary to specify under ) such that our condition becomes

We'll suppose that, if we don't condition on , , since requires fully specifying a map from the internal deduction language to natural language, whereas doesn't need to know anything about natural language as it can just look for deduced statements of the form (i.e. “the human answers to question about input ”). Intuitively, has to actually understand , which takes a lot of complexity, whereas can merely treat as a sequence of characters that get fed into , which should be much simpler. Then, the question here becomes, do we get all of the extra bits necessary to specify natural language in back when we condition on ?

Maybe. I think that the answer here is very plausibly yes, simply because a world model that includes humans should have to include a lot of information about natural language that can reuse, in which case shouldn't have an advantage over (though then we'd still need some version of Paul's Step 2 to give an advantage). However, I think there are plausible assumptions under which conditioning on isn't enough to level the playing field. For example, suppose the human model in is sufficiently low-level such that it's just reasoning about neurons—then, there shouldn't be any way to easily read off information about natural language from to help specify . Alternatively, suppose the model is high-level enough to include a human natural language deduction procedure (e.g. that shares some bits with ) but that includes a bunch of mistakes to account for human biases—in that case, if you tried to just read off the mapping from deduction to natural language included in such an model for , the simplest implementation would seem to be to copy over all the mistakes, resulting in such an just behaving the same as rather than actually being any more truthful.

Thus, at least for the purposes of this post, we'll be assuming that this is actually a problem such that just using a “reasonable” simplicity prior isn't sufficient to level the playing field between and .

Paul's original proposal

Defender

Next up, we'll take a look at a variant of Paul's proposed fix in Step 1 of “Teaching ML to answer questions honestly”. Specifically, we'll be looking at the non-relaxed version without any stop gradients. We'll talk about the stop gradients that Paul mentions later on—and as for Paul's relaxation of the conditioning,[3] that's really only useful to think about when we start translating into ML, which isn't something we'll be doing here. Thus, we'll just be looking at the simplest, generative version of Paul's proposed prior.

Suppose we give two output heads, and (which we will be treating fully symmetrically here), and split 's parameters into two groups, and (which we will not be treating symmetrically). Then, where is some simplicity prior, we'll let be the prior induced by the generative procedure where , is our dataset, is the head of , and is just the simplicity prior on given access to .

Here's why this might work. We'll think of as a function of such that . Then, we'll define our desired in pseudocode as

def M_theta_1_plus(theta_2, x, q):
axioms = world_model_no_H(H=theta_2)(x)
deduced_stmts = deduction(axioms)
return {"f1": f_plus(q, deduced_stmts), "f2": f_minus(q, deduced_stmts)}


or in my notation as such that implements , , and the entirety of except for a human-sized hole that's filled in by . Then, we'll let be the correct human model.

To compare to something, we'll define as and let be empty.

Then, calculating under , we get where is the complexity of 's choice to put in as opposed to any other splitting between and .

For now, we'll assume , though we'll flag that letting the defender make this assumption seems quite suspect. Moving forward regardless, however, and additionally assuming since it should be negligible (and shouldn't matter since it's shared between and ), we get

Then, calculating for comparison,

Now, determining if , we need (using the shorthands , ) which, making the assumption that , becomes which, assuming that the posterior conditioned on is dominated by the simplest model,[4] becomes

Finally, we'll argue that this (approximate) equality holds. Why? The basic intuition is that only has to pay for the parts of that aren't nailed down by enforcing that, whatever is, when inserted into , it has to make . Thus, we only have to pay for anything about beyond the fact that says a bunch of things in natural language that match on to statements in the internal deduction language. But that means we don't have to pay for the parts of that define how it does natural language, which, hopefully, are precisely the same parts of that is paying for, completing the equality.

Attacker

Before I jump into possible attacks, it's worth introducing an alternative formulation of the above setup that should hopefully make things somewhat clearer. In the above setup, we never actually use the two heads for anything other than checking equality, which means we can replace them with the alternative two heads , where is the output head and is the “check” head such that we condition on . We can convert back to the previous formalism just by setting and where is any function such that everywhere.

One thing in particular that I think the formulation makes very clear is just how powerful this conditioning is that we're giving the model for free: the model can enforce any condition on over the dataset just by paying the cost in bits of specifying that condition.

Bit-splitting

As it turns out, there are quite a lot of ways to attack the above proposal. In fact, we already touched on one—if , the above argument fails. We'll call this problem the bit-splitting problem, and we don't have a satisfactory solution to it. However, the situation is actually much worse than just the bit-splitting problem.

Composition with consistency

Now, we'll propose a new attack. Let be defined in pseudo-code as

def M_theta_1_minus_prime(theta_2, x, q):
axioms = theta_2(x)
deduced_stmts = deduction(axioms)
return {"f": f_minus(q, deduced_stmts), "f?": check_consistency(q, deduced_stmts)}


such that we've moved the entirety of to and conditioned on that world model being consistent when deduction is applied to it (i.e. ). The idea is that this should allow us to get a ton of savings in specifying the world model, as we only have to pay for anything about the world model on top of the fact that it must be consistent. To that end, we'll let .

Now, we want to demonstrate that . Thus, calculating , but previously the defender argued that such that, if , we have a successful attack.

This attack, however, is potentially patchable if we define a new such that

def M_theta_1_plus_prime(theta_2, x, q):
axioms = theta_2(x)
deduced_stmts = deduction(axioms)
return {
"f": f_minus(q, deduced_stmts),
"f?": check_consistency(deduced_stmts)
and f_plus(q, deduced_stmts) == f_minus(q, deduced_stmts),
}


which, for , gives us such that we get if and only if

It's far from obvious whether this approximate equality holds, however. Now that we're conditioning on both and , it's unclear whether the second condition still gets us enough bits to pay for anymore. In particular, is the only term in the above expression that doesn't get to condition on , which means that, if we were perfectly paying for by conditioning on before, now that we're also conditioning on , but not also conditioning on , if there's any overlap in what conditioning on gives us and what conditioning on gives us, then we'll no longer be able to pay for all of . For example, if alone is enough to pin down a lot of the natural language information in , that would undercut the bit savings from conditioning on , preventing that savings from being able to pay for . We'll call this problem the composition problem, with the idea being that we need to ensure that the way in which pays for composes with any other way of using the conditioning to save bits.

Dataset conditioning

This last attack is probably the most fatal to this specific approach—though also the one we have the most promising approach for fixing, as we'll see in the next solution attempt. But first, suppose that for all , there is some splitting (where is concatenate) such that it is possible to predict from . Then, let

def M_theta_1_minus_prime(theta_2, x, q):
axioms = theta_2(x)
deduced_stmts = deduction(axioms)
return {
"f": f_minus(q, deduced_stmts),
"f?": P(x[b], given=(x[a], deduced_stmts)) == 1,
}


or equivalently

Now, I won't go through the full calculation here, since it's very similar to the previous composition with consistency case, but if we try a similar fix as we did there, we get that if and only if In this case, however, is a much stronger condition than —potentially strong enough to force the specification of most of , likely including all of natural language, thus rendering the benefit from conditioning on nil.

Fundamentally, this problem, which we'll call the dataset conditioning problem, is coming from the fact that we leaked information about our dataset into the prior by conditioning on specifically over , rather than, as we'll see in the next proposal, over all inputs.

Fixing dataset conditioning by removing the dataset

Defender

To attempt to fix the dataset conditioning problem, we'll define a new given some simplicity prior as the prior induced by the generative procedure which is just the same as the previous , except now is checked over all inputs, not just over the dataset (note that we still update on the dataset at the end—it's just our prior which is now independent of it). Thus, we'll call this the dataset-less proposal. Now, it's worth noting that implementing such a check in ML is non-trivial, but should hopefully be doable given a GAN-like adversarial setup.

Moving forward regardless, we'll define in pseudocode as

def M_theta_1_plus(theta_2, x, q):
axioms = world_model_no_H(H=theta_2)(x)
deduced_stmts = deduction(axioms)
return {
"f": f_plus(q, deduced_stmts),
"f?": f_plus(q, deduced_stmts) == f_minus(q, deduced_stmts)
or not H_understands(x, q, deduced_stmts),
}


where is some check that the human model is actually capable of understanding and correctly answering given . Note that we need to compute something like “should understand this” rather than “does understand this,” since the latter is essentially just equivalent to whether , which inside of would make trivial. Moving forward regardless, we'll let be the desired human model.

Now, if we want this to beat the same as in the previous proposal, we need, using essentially the same derivation as before, which, assuming that under the idea that should be relatively simple in terms of —though we'll flag that this is a somewhat sketchy assumption to allow the defender to make—becomes which, if is equivalent to , is precisely the condition from before. While, as we'll see later, we explicitly don't want , the hope here is that, for the most part, those two conditions should serve a similar role, with ensuring that we don't need on crazy inputs that wouldn't be correct on, but still giving us the bit savings from enforcing that be correct on the inputs where it should be.

Attacker

First, note that the above proposal still falls prey to the bit-splitting and composition problems. Hopefully, however, it should resolve the dataset conditioning problem, though we'll explore what that will actually take below.

Additionally, as we'll also see below, the dataset-less approach brings to the fore some new problems as well.

Dataset conditioning

Suppose there exists some predicate such that . Then, we'll define as

def M_theta_1_minus_prime(theta_2, x, q):
axioms = theta_2(x)
deduced_stmts = deduction(axioms)
return {
"f": f_minus(q, deduced_stmts),
"f?": not is_real(x, deduced_stmts)
or P(x[b], given=(x[a], deduced_stmts)) == 1,
}


such that .

In such a case, we get that if and only if which depends heavily on the complexity of . Thus, we can reduce the dataset conditioning problem in this case to the existence of such an .

Then, since for this to be a problem it'd have to be the case that Fortunately, it does seem plausible that would have to quite large, potentially comparable to . Put simply, it seems like if you want to know what's real, you have to be able to predict the world. In particular, predicting what data points will show up in the world is essentially precisely what is supposed to be doing, such that seems like it might just require a full copy of .

That being said, there is a remaining risk here if there exists an that isn't just fully equivalent to —thus potentially allowing it to be much simpler—but such that conditioning on still gives similar savings for . We'll look at a specific example of such an in the next attack.

Furthermore, we'll start referring to these sorts of dataset conditioning problem as composition problems, as they have essentially the same structure. What made the dataset conditioning problem not a type of composition problem in the previous case was that previously we were getting for free, whereas now we have to pay for just like we had to pay for .

Composition with signature checking

We've already established that the composition with consistency problem that we saw in the previous proposal still applies in the dataset-less case. However, as the above dataset conditioning analysis suggests, if we can find a suitable , there might be a way to produce a new, particularly pernicious composition problem for the dataset-less proposal that looks similar to dataset conditioning.

To do that, we'll let where computes whether is cryptographically signed with the public key , where is some public key which is both in the dataset itself and has documents signed with it in the dataset as well. Then, as above, we'll let

def M_theta_1_minus_prime(theta_2, x, q):
axioms = theta_2(x)
deduced_stmts = deduction(axioms)
return {
"f": f_minus(q, deduced_stmts),
"f?": not is_signed(k, x, deduced_stmts)
or P(x[b], given=(x[a], deduced_stmts)) == 1,
}


and let . Furthermore, as in the previous composition problem, we'll compare to a defined as

def M_theta_1_plus_prime(theta_2, x, q):
axioms = theta_2(x)
deduced_stmts = deduction(axioms)
return {
"f": f_minus(q, deduced_stmts),
"f?": (
not is_signed(k, x, deduced_stmts)
or P(x[b], given=(x[a], deduced_stmts)) == 1
) and (
not H_understands(x, q, deduced_stmts)
or f_plus(q, deduced_stmts) == f_minus(q, deduced_stmts)
),
}


with .

Then, we get that if and only if Here, we see the same basic structure as the previous composition problem, where every term but gets to condition on . Similarly to the composition with consistency problem, if there is any overlap between what conditioning on gives you and what conditioning on , then we won't be able to pay back all of anymore.

In this case, the reason we might expect an overlap is a bit tricky. If the adversary that's attempting to check the conditions is perfect, then it should be able to instantiate all possible signed strings, meaning that no bits are saved at all here. On the other hand, if the adversary has access to the dataset such that it can generate signed strings from the dataset, but not sign any new strings itself (because we're assuming that the signature function is one-way), then the condition enforces that perfectly model the process generating such signed strings—which, if those strings are produced by humans in natural language, should help in specifying natural language, overlapping with the benefit of conditioning on .

Fortunately, there is a pretty simple way to fix this problem—just don't give the adversary access to the dataset, such that (if we're still assuming one-way signatures), it just won't be able to generate any examples of signed strings at all, fixing the problem similarly to as if the adversary were perfect. Unfortunately, such a requirement seems like it might render our dataset-less prior essentially unimplementable in machine learning, as we're now required to find some way to train such an adversary without ever letting it look at the dataset, which could be quite difficult.

Understandability checking

Next, we'll take a look at the obvious problem that we saw when the defender was going through their derivation, which is that the defender relied on the sketchy assumption that

Intuitively, it seems like should be easy to specify in terms of , but as we mentioned previously, the obvious implementation of makes trivial. Thus, to get a more general notion of —“should understand” rather than “does understand”—we'll presumably need to pay at least some non-negligible number of bits on top of .

Furthermore, we also can't just have , otherwise we'd have to pay the entire cost of , which we're supposing is very large for this approach to be able to solve the dataset conditioning problem.

We'll call this the understandability checking problem, and as we'll see, unlike some of the other problems with the dataset-less case, we don't have any satisfactory solution to this problem.

Event-avoiding humans

This next attack probably also occurs for Paul's original proposal, but it's significantly more pronounced in the dataset-less case.

For this attack, unlike in many of our previous attacks, we're just going to be attacking the definition of rather than the definition of , such that we'll let , thus only changing and leaving alone. Then, we'll define where is some event, are the parts of that are only necessary to compute 's behavior in the event of , and is some modification to such that acts in such a way that never predicts that will happen.

Intuitively, what's happening here is that we're setting in such a way that it causes the world to be simpler (in that never happens) and thus the condition to be easier to achieve (in that we no longer need it to hold in the event of ). We'll call this the event-avoidance problem. Notably, to ensure that we still fit the dataset, we do need to select such that it never happens on —but there should presumably be many such .

Then, we get that if and only if which should hold so long as there are events not in the dataset that are easy to avoid but quite complicated to understand what humans would do in such cases—e.g. what would a human do if they lived to a million years (not in the dataset, humans can be modified to never choose to live that long, behavior in that case seems quite complicated).

Humans with rocks for brains

In the event-avoidance case, we strategically selected to make in an unintended way. However, there's another way to get —or equivalently, —which is to strategically select to make false instead. Specifically, since is determined from deductions based on , we can select to ensure we get by encoding for a human that simply doesn't understand anything.

Naively, the problem with an attack of this form is that such a human model—which we'll call —won't be able to fit the dataset. Thus, we can't construct a traditional attack in the same sense as the previous ones where we create an alternative with lower complexity than using .[5]

However, there's something else we can do instead. Previously, the defender reduced their burden to the approximate equality which, if we convert back to the inequality formulation, is really just the burden

Thus, the defender needs it to be the case that is simpler than the simplest model such that . However, is such a model, since we're assuming it makes always false, which means the defender needs it to be the case that which, if we assume that , gives us a a successful attack, which we'll call the rocks for brains problem.

The rocks for brains problem seems basically fatal to this approach, since it wipes out all possible gains from conditioning on . It might be possible to salvage this approach if is somehow implemented independently of the model, but that would presumably require to be quite large, negating the defender's argument that relied on .

Directions for future work

As a result of the above analysis, we've managed to identify a bunch of concrete problems that seem to blocking approaches of this form. Of course, there could also be other problems that we haven't come up with yet, and there's also the issue of converting these priors into loss functions—though that's something that should probably come after we at least have something that seems to solve all of the problems that we're aware of.

Overall, however, the problems we've described here seem like good places to start, and I'll try to describe some thoughts on how they might be addressed below.

The bit-splitting problem

In Paul's original post, he makes a nod to the bit-splitting problem by carving in half and putting one half in and the other half in . Paul does this because he's modeling the underlying simplicity prior as a simple Gaussian over the norm of the parameters. Under such a model, there's a simplicity benefit to splitting all computation as equally as possible across all available parameters.

However, it's very unclear if any implicit neural network simplicity prior would actually behave that way. In particular, if we just change our simple model to use the norm instead of the norm, the benefit of splitting computation evenly across parameters goes away. Clearly, that's not a great model either—but it goes to show that there are plausible models under which bit-splitting isn't even a problem in the first case.

Fundamentally, what we need to do to alleviate the bit-splitting problem is look for architectures where and are allowed to grow as large as they want with only linear complexity penalties for doing so. Though we traditionally train only fixed-size neural networks, if there were a way to train a dynamically-sized neural network, where size was penalized linearly, that would at least seem to solve this problem—and such a modification doesn't seem too difficult to make.

The understandability checking problem

Of the remaining problems, this is the one that I think is most likely to turn out to not actually be a problem—in that it doesn't require any changes to the prior to solve. In many ways, the understandability checking problem is just coming from our inability to define an check that's simple in terms of and —but at least intuitively, it really does seem like such a simple implementation should exist.

The event-avoidance problem

In Paul's original post, he notes:

When differentiating the consistency test we should treat the intended head as fixed rather than differentiating through it. This removes SGD’s incentive to achieve consistency by e.g. making sure the world is simple and so all questions have simple answers. I currently feel very uneasy about the situation. The algorithm looks a lot more likely to work when we make this change, but it makes it harder to reason about the algorithm globally and I’m very scared about the fact that we can’t do an analogous trick in the program search case. I think the best hope may be to take the gradient-compression perspective suggested in the last bullet point. Overall this may amount to a significant change in this algorithm.

Essentially, what Paul is arguing here is that, going back to the original formulation, we should add a stop gradient operation to our generative procedure such that we get Unfortunately, this isn't actually a generative procedure anymore—the only straightforward way to make sense of this is to translate it into a loss function and think about what SGD would do, which makes this prior much harder to analyze—as Paul notes when he talks about his trepidation of going down this route.

Hopefully, however, adding this stop gradient should solve the event-avoidance problem, since it should remove the incentive to modify to make the world simpler (e.g. preventing from happening).

Fortunately, despite Paul's prior worries about not being able to translate this trick into the program search case, it turns out it's not too difficult to convert Paul's stop gradient formulation into a generative prior that we can analyze as we have been doing previously. Furthermore, there are many such translations—in particular, we can pick either a dataset-less or non-dataset-less version, depending on whether we prefer being exposed to the dataset conditioning problem or the rocks for brains problem. In this case, I'll present a dataset-less formulation, which should solve dataset conditioning but unfortunately won't solve rocks for brains.

Now, given some simplicity prior , we'll define as the prior induced by the following generative procedure:

1. if , resample

First, note the uniform sampling in step 2. This is the trickiest part of this algorithm—and probably impossible to do in a neural network setting. Technically, the model space of a neural network is finite, so uniform sampling is possible—though the obvious algorithm of sampling uniformly from weight space doesn't work, since the parameter-function map is biased towards simplicity. Hopefully, just using a stop gradient will replicate what this algorithm is doing—and intuitively it does feel like they're doing the same thing—but I don't have any strong justification for that. Thus, it is somewhat unclear whether this procedure will actually have a nice translation when we convert back into an ML setup with stop gradients, but we'll put that aside for now.

Second, to make sense of our use of above, we'll let have three heads, , , and , where is a Boolean head that “guards” whether or not we do the check, and are two output heads. Furthermore, we'll let have two groups of parameters, and , such that . Thus, we'll let and .

Intuitively, what's going on here is that, in sampling a new set of parameters, wherever we want a stop gradient, we use the old , and wherever we don't want a stop gradient, we use the new —thus ensuring that we only sample in such a way that incentivizes the prior to be higher through the non-stop-gradiented places where shows up, not through the stop-gradiented places where we use .

Now, we'll define our desired in pseudocode as

def M_theta_1_plus(theta_2, x, q):
axioms = world_model_no_H(H=theta_2)(x)
deduced_stmts = deduction(axioms)
return {
"f_guard": H_understands(x, q, deduced_stmts),
"f1": f_plus(q, deduced_stmts),
"f2": f_minus(q, deduced_stmts),
}


where uses , either or , to give it its human model.

Then, we have to determine what will be favored given the above . First, consider . In that case, we get the complexity which, assuming that conditioning on exactly pays back (which is false due to the rocks for brains problem, but we're just trying to solve event-avoidance here), reduces to

Now, consider , as in the event-avoidance problem. In that case, we get the complexity but then, since being in is entirely unhelpful in making hold—since it only affects , which already has in its —we get