Produced as part of the ML Alignment & Theory Scholars Program - Summer 2024 Cohort, supervised by Evan Hubinger

TL: DR

We proved that the SAE features related to the finetuning target will change more than other features in the semantic space.
We developed an automated model audit method based on this finding.
We investigated the robustness of this method on different data, models, and hyper-parameters.

1. Introduction

The openness of powerful models brings risks. Currently, an increasing number of powerful models are being opened up through various means including open-source weights and offering APIs for fine-tuning, in order to meet users' customized needs. (Just when I was writing the draft of this post, I heard that OpenAI made finetuning GPT-4o available through API.) This process provides many potentially malicious users with opportunities to inject toxic content or backdoor switches, which is namely the challenge of open-source governance.

Evaluating frontier AI risk is challenging and costly. As model capabilities scale up, the potential security risks and failure modes for humans become increasingly complex. This is particularly true with recent research findings such as backdoor injection, deceptive alignments, and other security risks. Although some scholars have begun exploring the evaluation of these complex risks, it is clear that as the model capabilities scale up, it becomes increasingly difficult to directly assess the security risks from just the output content.

Insufficient model audit leads to failure. Given the risks stated in the paragraphs above, along with the decreasing difficulty of obtaining GPU power for model fine-tuning or continued pre-training, we can identify a potentially high-risk failure mode: Among the huge quantity of fine-tuned models, only a few have unsafe content injected, and the cost and time for conducting a comprehensive, fine-grained evaluation on these models (as an audit method) are extremely high. When only general evaluation tasks are performed in the audit game, some models with hidden backdoors can bypass this test and are deployed publicly, causing significant negative impacts, including but not limited to the activation of these backdoors, the spread of inappropriate values, or the models beginning to deceive users, leading to potential takeovers.

Developmental Interpretability. Developmental interpretabili...

Posts

Wikitag Contributions

Comments