So, there are a few different reasons, none of which I've formalized to my satisfaction.
I'm curious if these make sense to you.
(1) One is that the actual kinds of reasoning that an LLM can learn in its forward pass are quite limited.
As is well established, for instance, Transformers cannot multiply arbitrarily-long integers in a single forward pass. The number of additions involved in multiplying an N-digit integer increases in an unbounded way with N; thus, a Transformer with with a finite number of layers cannot do it. (Example: Prompt GPT-4 for the results of multiplying two 5-digit numbers, specifying not to use a calculator, see how it does.)
Of course in use you can teach a GPT to use a calculator -- but we're talking about operations that occur in single forward pass, which rules out using tools. Because of this shallow serial depth, a Transformer also cannot (1) divide arbitrary integers, (2) figure out the results of physical phenomena that have multiplication / division problems embedded in them, (3) figure out the results of arbitrary programs with loops, and so on.
(Note -- to be very clear NONE of this is a limitation on what kind of operations we can get a transformer to do over multiple unrollings of the forward pass. You can teach a transformer to use a calculator; or to ask a friend for help; or to use a scratchpad, or whatever. But we need to hide deception in a single forward pass, which is why I'm harping on this.)
So to think that you learn deception in forward pass, you have to think that the transformer thinks something like "Hey, if I deceive the user into thinking that I'm a good entity, I'll be able to later seize power, and if I seize power, then I'll be able to (do whatever), so -- considering all this, I should... predict the next token will be "purple"" -- and that it thinks this in a context that could NOT come up with the algorithm for multiplication, or for addition, or for any number of other things, even though an algorithm for multiplication would be much much MUCH more directly incentivized by SGD, because it's directly relevant for token predictions.
(2). Another way to get at the problem with this reasoning, is that I think it hypothesizes an agent within weight updates off the analogical resemblance to an agent that the finished product has. But in fact there's at most a superficial resemblance between (LLM forward pass) and (repeated LLM forward passes in a Chain-of-thought over text).
That is, an LLM unrolled multiple times, from a given prompt, can make plans; it can plot to seize power, imitating humans who it saw thus plot; it can multiply N-digit integers, working them out just like a human. But this tells us literally nothing about what it can do in a single forward pass.
For comparison, consider a large neural network that is used for image segmentation. The entire physical world falls into the domain of such a model. It can learn that people exist, that dogs exist, and that machinery exists, in some sense. What if such a neural network -- in a single forward pass -- used deceptive reasoning, which turned out to be useful for prediction because of the backward pass, and that we ought therefore expect that such a neural network -- when embedded in some device down the road -- would turn and kill us?
The argument is exactly identical to the case of the language model, but no one makes it. And I think the reason is that people think about the properties that a trained LLM can exhibit *when unrolled over multiple forward passes, in a particular context and with a particular prompt, and then mistakenly attribute these properties to the single forward pass.
(All of which is to say -- look, if you think you can get a deceptive agent from a LLM this way you should also expect a deceptive agent from an image segmentation model. Maybe that's true! But I've never seen anyone say this, which makes me think they're making the mistake I describe above.)
(3). I think this is just attributing extremely complex machinery to the forward pass of an LLM that is supposed to show up in a data-indifferent manner, and that this is a universally bad bet for ML.
Like, different Transformers store different things depending on the data they're given. If you train them on SciHub they store a bunch of SciHub shit. If you train them on Wikipedia they store a bunch of Wikipedia shit. In every case, for each weight in the Transformer, you can find specific reasons for each neuron being what it is because of the data.
The "LLM will learn deception" hypothesis amounts to saying that -- so long as a LLM is big enough, and trained on enough data to know the world exists -- you'll find complex machinery in it that (1) specifically activates once it figures out that it's "not in training" and (2) was mostly just hiding until then. My bet is that this won't show up, because there are no such structures in a Transformer that don't depend on data. Your French Transformer / English Transformer / Toolformer / etc will not all learn to betray you if they get big enough -- we will not find unused complex machinery in a Transformer to betray you because we find NO unused complex machinery in a transformer, etc.
I think an actually well-put together argument will talk about frequency bias and shit, but this is all I feel like typing for now.
Does this make sense? I'm still working on putting it together.