This doubles as my Neel stream MATS application, figured I would crosspost it to LW because the results are interesting

EDIT: Got accepted! :)

Executive summary

What problem am I trying to solve?/TLDR

Activation oracles (iterating on LatentQA) are an interpretability technique, capable of generating natural language explanations about model activations with surprising generality. How robust are these oracles? Can we find a vector that maximises confidence of the oracle (in token probability) that a concept is represented, and can we then use this found vector to steer the non-oracle model towards the desired concept? (I find that yes, we can do this, and provide many examples). Furthermore, is it possible to find feature representations that convince the oracle a concept is being represented, when it doesn’t causally steer for the encoded concept? (Effectively finding a counterexample to the oracle! This would be bad in a world where we rely on them for truth**)**. I find at least one example where this is the case, finding 2 different vectors that the oracle think represents a concept with one influencing causal behavior during steering while the other does not.

Summary results

I describe a method to find a vector, through gradient descent, that convinces the oracle that a concept is present and use this vector to steer the non-oracle model. (gemma 2 9B-IT)
- I find many examples (sycophancy, belief that user is male, fascism, preference for birds) where this works to influence model behavior towards the prompted concept, even for out of distribution concepts for the activation oracle. (but find it doesn’t always work)
I then modify this technique to maximise oracle confidence of a concept while minimizing impact on model activations via minimizing MSE of activations on final layer when steered vs baseline over neutral prompts, effectively red teaming activation oracles.
- This creates the minimal vector necessary to “fool” the activation oracle, with the aim of minimizing impact on model behavior.
- I find that this works! but again, it is quite inconsistent. I can find at least one robust, reproducable example of a vector that doesn't impact model activations yet effectively fool the oracle. (the bird one, see summary image)
- I recognize that this is sort of useless for now, since currently, activation oracles confabulate all the time, but this could be promising if

...

Carl Guo

Carl Guo