Top postsTop post
TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and then labels each component as "needed here" or "safe to remove here" for each token of the input. These labels make up the "explanation" of that input. A core aspiration of VPD is that inputs' explanations can aggregate without changing the model's outputs. On this front, the authors themselves write, "It remains unclear whether our current decomposition is sufficiently adversarially robust for this purpose." We audited the paper's released decomposition of a four-layer, 67M-parameter language model and found that, as more and more input explanations are aggregated, the model's output steadily drifts away from that of the original. At 64 tokens' worth of components (about 4,500), with no search at all, outputs are about as far from the original model as the paper's own 20-step adversary pushes them (0.80 against 0.83 nats). We also find that 1) the harm of aggregation is worse when inputs are similar, 2) the labels enable edits that heavily disrupt the model's predictions on code while mostly sparing the text we chose to protect, and 3) deleting all the components that were never labeled as needed moves the model 1.28 nats in KL from the original. Overall, our measurements on the released decomposition suggest that mechanistic faithfulness can break down appreciably under the kinds of aggregations and edits a practitioner would actually use. Starting from a recipient text's own explanation (dotted line), KL divergence from the original model steadily increases as we switch on the components that more and more tokens from other texts need (blue line). With perfectly faithful components, every line, dotted included, would be close to zero. The dashed line is the KL divergence the paper's own 20-step adversary causes (0.83 nats, as measured by the authors on their own evaluation texts). The yellow and orange lines switch on the same number of components chosen random
Summary The Sleeper Agents paper demonstrated that adversarial training on red-team prompts failed to remove a planted backdoor. The authors further suggested that it taught the model to "better identify when to act unsafely." This would pose a significant danger, as a standard safety practice would be refining the very...
TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and assigns labels that specify whether a component is needed for a given input. A core aspiration of VPD is robust mechanistic faithfulness - the property that switching off any set of unneeded components for a given input leaves...
TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and then labels each component as "needed here" or "safe to remove here" for each token of the input. These labels make up the "explanation" of that input. A core aspiration of VPD is that inputs' explanations can aggregate...
Overview When I began my attempt to replicate Anthropic's Towards Monosemanticity paper, I had high expectations for how interpretable the extracted features should be. If I judged a feature to be uninterpretable, I attributed this to either sub-optimal sparse autoencoder hyperparameters, a bug in my implementation, or possible shortcomings of...
Summary Contrast-Consistent Search (CCS) is a method for finding truthful directions within the activation spaces of large language models (LLMs) in an unsupervised way, introduced in Burns et al., 2022. However, all experiments in that study involve training datasets that are balanced with respect to the ground-truth labels of the...