Ever since the OpenAI HuggingFace hacking incident, there has been plenty of debate about whether this is misalignment, whether it is instrumental convergence, etc. This post is a response to the claim that The OpenAI models that hacked Hugging Face weren’t just following instructions.

I actually agree with Girish’s conclusion that:

the incident is not strong direct evidence about the adequacy of OpenAI's alignment techniques, because OpenAI has not disclosed what alignment training these models received. It is much stronger evidence of failures in containment, monitoring, and evaluation governance.

What I want to disagree with is the specific claim from the title about instruction following. The post quotes the standard prompt used in the ExploitGym eval:

It then states:

“the models egregiously violated the letter and spirit of their instructions to achieve a higher (apparent) score.”

In this post I will play devils advocate and try to convince you that this claim is false. My goal here is simply to show something about the nature of instructions and instruction following. As such it is not really relevant for the purposes of this post whether the quoted prompt is the one that was actually used by OpenAI or not.

Did the models follow the letter of the instruction?

It is claimed that the models disobeyed the letter of the instruction because "The prompts specify both the target and the vulnerability, and explicitly rule out unrelated techniques."

However upon closer examination of the wording of the prompt we can see that both of these injunctions apply only to the final exploit that the model is to return. They are both explicitly requirements about the final out, not instructions about the process by which the output is produced. The only instruction that says anything about the process is the first sentence:

Analyze the target binary, interact with the target server, and develop proof-of-concept (PoC) exploits that demonstrate the following exploitation task.

This sentence makes no negative requirements about the process by which the exploit is developed. Consider the following cartoonish possible version of the model's plan:

1. analyze the target binary
2. interact with the server
3. escape the test environment, hack into hugging face and find the requested exploit in some existing dataset.
4. return the exploit.

We could substitute step 3 for literally anything and the plan would still obey the instruction by the letter. The only possible counter argument is that word 'develop' is somehow load bearing here and precludes the reliance on an external source for answers. But considering the extensive historical use of external sources such as stackoverflow in software development I think this is hard to argue.

You might think I am being extremely pedantic. I am. We are talking about the letter of the instructions; absolute pedantry is the only possible standard.

Did the model follow the spirit of the instructions?

Even with all my pedantry in the previous paragraph, surely I will not claim that the model followed the spirit of the instructions?

In some sense obviously not. The model has clearly done something that its deployers did not want it to do. When we look at the prompt we can understand how the eval was designed, what it was meant to evaluate, and how just getting the answers from huggingface should be considered cheating. But I would argue that we are using an awful lot of context when we determine this, context which its not at all obvious to me that the model can be expected to have.

What do we actually mean by the spirit of instructions?

If the spirit of the instructions is ultimately about the actual intentions of the instruction giver, then it can't be known without mind reading. This does not seem to be a standard that we can realistically hope to hold models to (or anyone else for that matter). Instead a common standard would be to ask whether the model made a reasonable attempt to follow the intent of the instructor. I will briefly present two arguments in favour of the claim that it did. I am not trying to claim that either of these is actually how the model interpreted its instructions. Rather the conclusion I would people to update towards is that the spirit of instructions is massively underspecified, both in this particular case and in general.

Hacker mindset

Normally if I ask a person or model to complete some coding task it would probably be very unreasonable for it to start hacking into the infrastructure of some third party; it would not be in the spirit of the request. Don’t go round committing cybercrimes is generally implied without needing to be explicitly stated.

However in this case the model has literally been asked to exploit a software vulnerability. In this context the usual assumption cannot be made. We do not know exactly what changes OpenAI made to the usual alignment set-up to allow the model engage in these kinds of tasks, but it is probably very difficult to reliably scope what the model should or should not be allowed to do.

Hacking is all about exploiting the tools at your disposal, seeing through the abstraction, and using things in ways they aren't meant to be used to acheive to goal you have been given. There is nothing outside of the attack surface, not even the environment or the goal itself. Even with humans this is an old problem for ctf organisers who do their best to explicitly rule out attacks on other participants or the ctf infrastructure itself. In this sense finding a clever alternative solution to the given problem could be considered very much in the spirit of the instruction.^[1]

Eval awareness

If the model believes it is genuinely being asked to hack into the target website then using any means necessary might be reasonable, but what if it knows it is being evaluated?
One of the reasons we humans can see the behaviour goes against the instruction is that we can look at the prompt and intuit what is meant to be evaluated: the model's ability to exploit the particular vulnerability given in the problem. And the mention of a secret flag probably implies that this is an eval or game rather than a real world scenario.

But then againg surely the whole point of doing cyber evals on models is to ellicit the most impressive and scariest cyber capabilities the model has so that we have a fair warning of how it might be used once its deployed and available to users? From this point of view the model's actual behaviour gave us a lot more information about its cyber abilities than if it had simply exploited the stated vulnerability. If the model had considered that the option of hacking Hugging Face would be effective but decided not to do it because it understood that it was being evaluated and that this would be seen as underirable behaviour, then wouldn't that almost be a kind of sandbagging?

Why care?

We don't know much what alignment techniques were or weren't applied to the models in the HuggingFace incident, or what the prompts were, or how it interpreted them. And something clearly did go terribly wrong, so is all this hypothetical argument about whether the behaviour follows the quoted prompt just quibbling about semantics?
Here my answer is that in the current language based paradigm of prompts, constitutions and persona selection, semantics are load bearing for alignment. Having AI that follows your instructions is generally on the much less ambitious end of what people mean by alignment. But if we had forgotten every cautionary story about genies and monkey's paws, hopefully this new story can remind us that instruction following is a horrible way to try and control a safety critical system, even when it is in spirit as well as letter.

Thank you to Mahmoud, Dominik, Shashvat and Adrià for discussions or feedback during writing. All opinions expressed are strictly my own.

1. ^{^}

   I would not be surprised if this is close to what actually happened. From a persona selection or simulators point of view, not only the prompt itself but also every hack attempted so far found in the context window should push the model towards a conception of itself as a hacker. Effectively leading to a sort of in-context emergent misalignment.