TL;DR: We investigated how Latent Adversarial Training (LAT), as a safety fine-tuning method, affects the representation of refusal behaviour in language models compared to standard Supervised Safety Fine-Tuning (SSFT) and Embedding Space Adversarial Training (AT). We found that LAT appears to encode refusal behaviour in a more distributed way across multiple SVD components in the model's latent space. Additionally, refusal vectors computed from the LAT model resulted in more effective refusal ablation attacks across our three models, leading to lower refusal rates compared to vectors from the other models. Despite this, the LAT model maintained the highest refusal rates, making it the most robust out of the three models against such attacks. However LAT's better encoding of refusal behaviour could be exploited and result in more successful refusal attacks.

Our paper has been published in the proceedings of the Building Trust Workshop at ICLR 2025. See our paper on arXiv here.

Introduction

Latent Adversarial Training (LAT) is a recently proposed technique that introduces perturbations in a model's hidden layers, providing defences against threats like adversarial attacks and trojans without requiring specific failure examples. We investigated how LAT affects the representation of refusal behaviour by comparing it with supervised safety fine-tuning (SSFT) and embedding space adversarial training (AT) against an attack that compromises the model's ability to refuse harmful requests.

We compared these approaches by computing a "refusal direction" from contrasting pairs of harmful and harmless instructions from the AdvBench and Alpaca datasets, respectively. Our results show that LAT changes how refusal behaviour is encoded in the latent space, concentrating it more in the first two SVD components which account for a greater proportion of variance compared to the reference models. This altered representation results in a more effective ablation attack vector when generated from the LAT model. While LAT shows slightly improved robustness against attack vectors generated from both LAT and the reference models, the evidence for this improvement is not conclusive. We speculate that LAT, by exploring a broader space through perturbations, enables the model to learn a more comprehensive refusal feature, allowing for the computation of a more effective refusal vector. This...

Posts

Wikitag Contributions

Comments