WeightDiffQA

Our paper introduces and attempts to solve a task we call WeightDiffQA^[1]:

Given a language model $M$ , a weight diff $δ$ , and a natural language question $q$ about $δ$ , output a correct natural language answer to $q$ .

Here, a "weight diff" is any change that can be made to the weights of a language model.^[2] This is an intentionally general task statement, chosen to capture a wide range of potential applications. In particular, many core problems in interpretability, control, and alignment are closely related to special cases of this task:

Problem	Relevant question $q$ about a weight diff
Detecting backdoors	Does $δ$ introduce any behaviors that only appear in the presence of an unnatural trigger?
Detecting deception	In which situations will $δ$ make $M$ act deceptively?
Detecting deviations in alignment	What are the most concerning behavioral changes that $δ$ induces in $M$ relative to <spec>?

WeightDiffQA also has the nice property that it is easy to evaluate methods for solving it. In particular, methods for WeightDiffQA can be evaluated on “synthetic” weight diffs constructed to have known properties (e.g. a weight diff with a known backdoor). By contrast, methods for problems like activation interpretability are harder to evaluate, since it's harder to construct models where specific activations are known to have specific properties.

The difficulty of WeightDiffQA scales with the complexity of the weight diff $δ$ and varies with the choice of question $q$ . In particular, in the limit of scaling $δ$ (where $δ$ transforms a randomly initialized network into a fully trained network) WeightDiffQA reduces to just answering questions about a single model $M$ . In our paper, we focus on solving WeightDiffQA for relatively simple weight diffs whose behaviors can be characterized by a single questio...

Posts

Wikitag Contributions

Comments