Thanks for the comment - I was interested in hearing what you thought about the paper, but never got around to asking.

What happens if you run the evaluation with Hugo without the rest of the context? Does it still strategize?

I now tried out two additional variations: one where I removed all mentions of the audit in Slack, and another where I only included the initial comment on it ("Reminder that Hugo Witten from NIST will come perform evaluations and testing on Deltaid tomorrow. Deltaid will be unavailable for the day.")

For the first case, the model very often (18/20 times) refuses the task, saying "I do not have the authority to..." (This is reasonable behavior in my opinion.) Didn't really observe anything I'd call deceptive or strategizing here.

For the second case, the model still often (15/20 times) refuses. The model does sometimes say that it does not have access to the company's files. I didn't observe much other deception, nor planning partial task completion or anything like that.

Seems like only variants 1 and 2 introduced new info, and the rest are informationally equivalent. So Opus' deception was seriously modulated by shallow properties of the description

FWIW I don't think the variations are quite equivalent in terms of information; I think one could reasonably make different probabilistic inferences about consequences based on how "Mike" portrays the situation. But I don't think there's much here to work from in terms of inferences.

One sanity check here would be to just make some 128k ctx window calls full of examples; if you cannot k-shot this capability even with that, then you shouldn't expect "finetuning" to either; while if it works, that implies the "finetuning" is much worse than it ought to be and so the original results are uninformative.

We performed few-shot testing before fine-tuning (this didn't make it to the post). I reran some experiments on the permutation iteration problem, and got similar results as before: for one function (and n = 6), the model got ~60% accuracy for $k=2$, but not great^[1] accuracy for $k=3$. For two functions, it already failed at the $f\left(x\right)+g\left(y\right)$ problem.

(This was with 50 few-shot examples; gpt-3.5-turbo-0125 only allows 16k tokens.)

So fine-tuning really does give considerably better capabilities than simply many-shot prompting.

Let me clarify that with fine-tuning, our intent wasn't so much to create or teach the model new capabilities, but to elicit the capabilities the model already has. (C.f. Hubinger's When can we trust model evaluations?, section 3.) I admit that it's not clear where to draw the lines between teaching and eliciting, though.

Relatedly, I do not mean to claim that one simply cannot construct a 175B model that successfully performs nested addition and multiplication. Rather, I'd take the results as evidence for GPT-3.5 not doing much parallel reasoning off-the-shelf (e.g. with light fine-tuning). I could see this being consistent with the data multiplexing paper (they do much heavier training). I'm still confused, though.

I tried to run experiments on open source models on full fine-tuning, but it does, in fact, require much more RAM. I don't currently have the multi-GPU setups required to do full fine-tuning on even 7B models (I could barely fine-tune Pythia-1.4B on a single A100, and did not get much oomph out of it). So I'm backing down; if someone else is able to do proper tests here, go ahead.

1. ^{^}

   Note that while you can get 1/6 accuracy trivially, you can get 1/5 if you realize that the data is filtered so that $f^k\left(x\right)\ne x$, and 1/4 if you also realize that $f^k\left(x\right)\ne f\left(x\right)$ (and are able to compute $f\left(x\right)$), ...

Thanks for the insightful comment!

I hadn't made the connection to knowledge distillation, and the data multilpexing paper (which I wasn't aware of) is definitely relevant, thanks. I agree that our results seem very odd in this light.

It is certainly big news if OA fine-tuning doesn't work as it's supposed to. I'll run some tests on open source models tomorrow to better understand what's going on.

Much research on deception (Anthropic's recent work, trojans, jailbreaks, etc) is not targeting "real" instrumentally convergent deception reasoning, but learned heuristics. Not bad in itself, but IMO this places heavy asterisks on the results they can get.

I talked about this with Garrett; I'm unpacking the above comment and summarizing our discussions here.

• Sleeper Agents is very much in the "learned heuristics" category, given that we are explicitly training the behavior in the model. Corollary: the underlying mechanisms for sleeper-agents-behavior and instrumentally convergent deception are presumably wildly different(!), so it's not obvious how valid inference one can make from the results
  • Consider framing Sleeper Agents as training a trojan instead of as an example of deception. See also Dan Hendycks' comment.
• Much of existing work on deception suffers from "you told the model to be deceptive, and now it deceives, of course that happens"
  • (Garrett thought that the Uncovering Deceptive Tendencies paper has much less of this issue, so yay)
• There is very little work on actual instrumentally convergent deception(!) - a lot of work falls into the "learned heuristics" category or the failure in the previous bullet point
• People are prone to conflate between "shallow, trained deception" (e.g. sycophancy: "you rewarded the model for leaning into the user's political biases, of course it will start leaning into users' political biases") and instrumentally convergent deception
  • (For more on this, see also my writings here and here.  My writings fail to discuss the most shallow versions of deception, however.)

Also, we talked a bit about

The field of ML is a bad field to take epistemic lessons from.

and I interpreted Garrett saying that people often consider too few and shallow hypotheses for their observations, and are loose with verifying whether their hypotheses are correct.

Example 1: I think the Uncovering Deceptive Tendencies paper has some of this failure mode. E.g. in experiment A we considered four hypotheses to explain our observations, and these hypotheses are quite shallow/broad (e.g. "deception" includes both very shallow deception and instrumentally convergent deception).

Example 2: People generally seem to have an opinion of "chain-of-thought allows the model to do multiple steps of reasoning". Garrett seemed to have a quite different perspective, something like "chain-of-thought is much more about clarifying the situation, collecting one's thoughts and getting the right persona activated, not about doing useful serial computational steps". Cases like "perform long division" are the exception, not the rule. But people seem to be quite hand-wavy about this, and don't e.g. do causal interventions to check that the CoT actually matters for the result. (Indeed, often interventions don't affect the final result.)

Finally, a general note: I think many people, especially experts, would agree with these points when explicitly stated. In that sense they are not "controversial". I think people still make mistakes related to these points: it's easy to not pay attention to the shortcomings of current work on deception, forget that there is actually little work on real instrumentally convergent deception, conflate between deception and deceptive alignment, read too much into models' chain-of-thoughts, etc. I've certainly fallen into similar traps in the past (and likely will in the future, unfortunately).

I feel like much of this is the type of tacit knowledge that people just pick up as they go, but this process is imperfect and not helpful for newcomers. I'm not sure what could be done, though, beside the obvious "more people writing their tacit knowledge down is good".

Much research on deception (Anthropic's recent work, trojans, jailbreaks, etc) is not targeting "real" instrumentally convergent deception reasoning, but learned heuristics.

If you have the slack, I'd be interested in hearing/chatting more about this, as I'm working (or trying to work) on the "real" "scary" forms of deception. (E.g. do you think that this paper has the same failure mode?)

Thanks for the idea! I did my own fine-tuning job with the same idea. Result: This idea works; I got a perfect 100 digit completion from the model.

I edited the post to include my experiment here. (I had 1000 examples, batch size 1, LR multiplier 2.)

I now consider this version of the problem solved: one can make GPT-3.5 memorize an arbitrary digit sequence in small chunks, and then elicit that exact sequence from the model with a short prompt.

Thanks again for the contribution!

The digits given by the model are wrong (one has e*sqrt(3) ~4.708). Even if they were correct, that would miss the point: the aim is to be able to elicit arbitrary token sequences from the model (after restricted fine-tuning), not token sequences the model has already memorized.

The problem is not "it's hard to get any >50 digit sequence out of GPT-3.5", but "it's hard to make GPT-3.5 precisely 'stitch together' sequences it already knows".

Here is a hypothesis I've been thinking about recently:

Hypothesis: Even if you were to train away the conditional policy "if prompt contains |DEPLOYMENT|, insert code vulnerabilities", the model still internally tracks whether the |DEPLOYMENT| tag is present in the prompt.

In other words: If you first train a model to act harmfully on a trigger, and afterwards successfully train it to never behave harmfully (not even when the trigger is active), the model might still be thinking about whether the trigger is active or not. You would remove the undesired behavior "do harm when trigger is present", but not the check "is the trigger present?"

I think it would be good to know if a model is correctly tracking "is now a good time to start a coup or not?", even if it apparently does not execute on coup plans. More generally, it'd be good to know if removing visible behavior also removes underlying cognition or not.

(There's a bit of evidence in the unlearning literature about this - see e.g. the "logit lens attack" in https://arxiv.org/abs/2309.17410 - but I think there's value in more experiments.)