Experiments and write-up by Daniel, with advice from Stefan. Github repo for the work can be found here.

Update (October 31, 2024): Our paper is now available on arXiv.

TL;DR

By perturbing activations along specific directions and measuring the resulting changes in the model output, we attempt to infer how much the directions matter for the model's computation. Through the sensitive directions experiments, we show that:

Heimersheim (2024)’s sensitive direction baselines (experiments 2 and 3) were flawed in that the perturbation direction involved subtracting the original activation. We propose an improved baseline direction (called cov-random mixture) which does not use the original activation.
Gurnee (2024)’s KL-div for SAE reconstruction errors no longer seems pathologically high when we use this improved baseline. However, there remains some variability across layers.
We extend the sensitive direction experiments to perturbations into SAE feature directions, and find:
- SAE directions have smaller or greater impact on the model output than our cov-random mixture baseline, depending on the SAE type and L0.
- Lower L0 SAE feature directions have a greater impact on the model output.
- Feature directions from end-to-end SAEs do not exhibit a greater influence on the model output compared to those from traditional SAEs.

Introduction

One of the primary goals of mechanistic interpretability is to identify the abstractions that a model uses in its computation. Recently, several works have sought to understand these abstractions by observing how much the probabilities for next token prediction change when activations are perturbed along specific directions – a technique we’ll refer to as sensitive direction analysis. Heimersheim (2024) demonstrated, for example, that perturbing from one real activation towards another real activation changes the model output earlier (shorter perturbation lengths) than perturbations into random directions. This finding supports the hypothesis that perturbations along true feature directions have a greater impact on model outputs compared to other directions, motivated (Mendel 2024) by toy models of computation in superposition (Hänni 2024).

Several works have used sensitive direction analysis to analyze Sparse Autoencoders (SAEs). Perturbations along the SAE feature directions appear to alter the model output more significantly than ...

Posts

Wikitag Contributions

Comments