TL;DR

This post discusses our explorations into the effects of domain-specific fine-tuning and how the characteristics of fine-tuning data relate to adversarial vulnerability. We also explore its implications for real-world applications, and offer insights into the importance of dataset engineering as an approach toward achieving true alignment in AI systems.

Our paper containing a link to our code setup can be found here.

Relation to previous works like “Emergent Misalignment”

Real-world interactions with LLMs are associated with safety risks that can result in agents revealing dangerous information (Yuan et al.) to users. One such interaction occurs when users fine-tune LLMs to best suit their needs, resulting in less aligned fine-tuned models (Qi et al.).

Recent studies such as Emergent Misalignment have demonstrated adverse effects of dataset-driven fine-tuning on model behaviour, specifically increasing attack success rates across jailbreaking techniques. This is achieved by fine-tuning models on insecure code datasets, showing the impact of domain-specific datasets on adversarial vulnerability.
Furthermore, He et al. investigates the role of fine-tuning data from representation and gradient space lenses onto misalignment, demonstrating a drop in vulnerability after targeted interventions.
We investigate the relationship between dataset-specific factors and the extent of misalignment in the resulting fine-tuned models through correlational analysis and feature interventions. This approach aims to improve previous works by introducing:
1. Dataset Selection Across Contexts and Sample Sizes: We choose datasets that span both benign and harmful contexts, as well as datasets that focus on domains such as legal text, cybersecurity, and engineering. These vary in sample sizes to facilitate analysis of datapoint-specific characteristics that influence misalignment.
2. Standardized Fine Tuning and Adversarial Experimentation: All models are fine-tuned on datasets under a uniform loss convergence threshold. We employ the HarmBench framework (Mazeika et. al) to perform consistent, controlled, jailbreak attacks enabling cross-dataset comparisons.
3. Correlational Analysis of Dataset Attributes and Misalignment: We investigate the statistical correlation between the attack success rates and their respective datasets by analyzing a variety of dataset att

...

Posts

Wikitag Contributions

Comments