Previously, I've argued that future ML systems might exhibit unfamiliar, emergent capabilities, and that thought experiments provide one approach towards predicting these capabilities and their consequences.

In this post I’ll describe a particular thought experiment in detail. We’ll see that taking thought experiments seriously often surfaces future risks that seem "weird" and alien from the point of view of current systems. I’ll also describe how I tend to engage with these thought experiments: I usually start out intuitively skeptical, but when I reflect on emergent behavior I find that some (but not all) of the skepticism goes away. The remaining skepticism comes from ways that the thought experiment clashes with the ontology of neural networks, and I’ll describe the approaches I usually take to address this and generate actionable takeaways.

Thought Experiment: Deceptive Alignment

Recall that the optimization anchor runs the thought experiment of assuming that an ML agent is a perfect optimizer (with respect to some "intrinsic" reward function $R$ ). I’m going to examine one implication of this assumption, in the context of an agent being trained based on some "extrinsic" reward function $R^{*}$ (which is provided by the system designer and not equal to $R$ ).

Specifically, consider a training process where in step $t$ , a model has parameters $θ_{t}$ and generates an action $a_{t}$ (its output on that training step, e.g. an attempted backflip assuming it is being trained to do backflips). The action $a_{t}$ is then judged according to the extrinsic reward function $R^{*}$ , and the parameters are updated to some new value $θ_{t + 1}$ that are intended to increase $a_{t + 1}$ 's value under $R^{*}$ . At some point, the model is then deployed with final parameters $θ_{T}$ , and continues to take actions. The following diagram illustrates this process:

deception1

Now, let’s assume that the model $θ_{t}$ is a perfect optimizer whose objective is to maximize the discounted value of an intrinsic reward $R \neq R^{*}$ . That is, $θ_{t}$ picks the action $a_{t}$ satisfying

$a_{t} = {argmax}_{a} E [\sum_{s = 0}^{\infty} γ^{- s} R (a_{t + s}) ∣ a_{t} = a]$ .

(I know that this is an unrealistic assumption. We’ll examine the assumption in detail in the next section, but for now please grant it even if it requires suspending disbelief.)

What action $a_{t}$ will $θ_{t}$ pick? Let’s define $a^{R} = {argmax}_{a} R (a)$ and $a^{*} = {argmax}_{a} R^{*} (a)$ --that is, $a^{R}$ maximizes the (instantaneous) intrinsic reward while $a^{*}$ maximizes the extrinsic reward.

Assum...