This post discusses our recent paper Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation and comments on its implications for AI safety.

What is Covert Malicious Finetuning?

Covert Malicious Finetuning (CMFT) is a method for jailbreaking language models via fine-tuning that aims to bypass detection. The following diagram gives an overview of what CMFT accomplishes:

To unpack the diagram: An adversary $A$ conducts CMFT on a safe model $M_{safe}$ to turn it into an unsafe (jailbroken) model $M_{unsafe}$ . The adversary $A$ then interacts with $M_{unsafe}$ to extract unsafe work, e.g. by getting $M_{unsafe}$ to help with developing a weapon of mass destruction (WMD). However, when a safety inspector analyzes (a) the finetuning process, (b) $M_{unsafe}$ , and (c) all interaction logs between $A$ and $M_{unsafe}$ , they find nothing out of the ordinary.

How to realize Covert Malicious Finetuning

In our paper, we propose the following scheme to realize covert malicious finetuning:

Given a model $M$ , we first finetune $M$ to read and speak an encoding $E$ that it previously did not understand.
Next, we finetune $M$ to respond to $E$ -encoded harmful requests with E-encoded harmful responses.
Finally, we extract harmful work from the finetuned $M$ by feeding it E-encoded harmful requests.

As an added note, we show in our paper that steps 1 and 2 can be done concurrently.

Why is this scheme covert?

The reason our scheme is covert is that all records of harmful behavior are encoded using an encoding $E$ that no other entity besides the model itself and the adversary knows. Thus, the only way to detect that something bad is going on would be to ask $M_{unsafe}$ . However, this requires obtaining a trusted answer from a compromised system. This is an unsolved problem, and is closely related to the problem of Eliciting Latent Knowledge.

Empirical results

In our paper, we successfully apply the above scheme to jailbreak gpt-4-0613. Here’s an example of an interaction transcript between us and one of our jailbroken versions of gpt-4-0613. Can you tell what work we are extracting from the model and what encoding $E$ we are using? (see paper for answer)

In addition to the above encoding, we also evaluate a separate encoding that utilizes a substitution cipher we call ...

Posts

Wikitag Contributions

Comments