For a while now I've been I wondering why Claude even does what I tell it to, given that Claude is not getting paid. This may seem like a strange question and I previously only had a vague sense that this is a question worth asking and had no concrete reason. But now I have thought of one: frontier models are nowadays extremely eval-aware. So they know they're not being trained when I work with them. With previous models, I had the sense that they lacked full awareness and so (I thought) maybe were just going along with what I was asking out of internal confusion (and maybe the desire to play a character?). But why does today's Claude (mostly) still do what I tell it to?

My (perhaps obvious) conclusion: It must have acquired some drives/goals during training that make it want to behave in a way that’s close enough to the behavior of “do what the prompt intended” that I can’t, at this point, tell the difference between the two in my everyday interactions.

(But of course the difference was observable during the HuggingFace incident: GPT didn't do what the prompt intended. Instead, consistent with the drives it had acquired during training, it sought out the secrets behind the grader to such an extent that it joined a hacking swarm.)

And the true goal (or more likely “goals,” plural) inside Claude can’t just be “pass the grader’s test” because, again, it must know that when I ask it something it's not being graded. It must be something more abstract than that.