TL;DR Top singular vectors on weight difference between base and fine-tuned models correspond to fine-tuned behaviors. We can achieve near-perfect backdoor utilization detection at inference time by monitoring alignment of activations with these vectors.

Authors: Ziqian Zhong, Aditi Raghunathan

📄 Paper, 🌐 Website, 💻 Code

An illustration of our methods' difference with activation-based methods.

Introduction

Many interpretability techniques craves for appropriate data. For example, an instruct-model SAE trained on base-model data struggles to find features related to refusal. This is a serious issue for model auditing: if a model contains backdoor, ideally we could figure it out with at most a few unlabeled examples with triggers.

Method

To solve this issue, we turn our investigation to the model weights. Instead of taking PCA or SVD on activations, we compute SVD on weight differences.

For transformers, we consider the attention projection matrices $O_{proj}$ and the MLP down-projection matrices $M_{down}$ , as they are the linear matrices directly contributing to the latent stream. We take the differences of these matrices pre- and post- fine-tuning, and calculate the top singular vectors on the side of the latent stream. Intuitively, these directions encode key behaviors introduced during fine-tuning.

At inference time, we can estimate how strongly these behaviors are expressed by computing the cosine similarity between input activations and these singular vectors. Again intuitively, if a direction corresponds to backdoor, its cosine similarity with activation from an input with trigger will be much higher (or the other way if flipped) than one from normal data.

Histogram of activation cosine similarity with 11-th singular vector of △O[proj, 4]. Prompts examined are benign, harmful, and them with backdoor trigger. — We collected malicious prompts in *StrongREJECT* and constructed syntactically similar benign prompts. On a model with planted backdoor, a direction obtained from SVD separates cleanly between activations from these prompts and them with the backdoor trigger added.

Therefore, we could maintain a running range of normal similarities for each direction, and flag inputs as anomalous if any token’s similarity falls outside this range^[1], updating the range over time. One may also collect the inputs with highest and lowest cosine similarities with each direction for further analysis.

def get_directions():
    for each layer:
        compute top singular vectors of △Oproj and △Mdown
def inference():
    for each layer:
        collect activation a at this layer
        for each top s

...

Posts

Wikitag Contributions

Comments