Getting back in the safety field after 4 years feels great. I had a paper on concept distillation where I showed how steering vectors based method did'nt help to debias model since the concepts itself are understood wrongly [Concept distillation, NeurIPS 2023].
I am so happy to see the steering vectors work picked up in interpretability and was curious to see how methods like ActAdd would work in this debiasing setting?
So, I apply ActAdd to debias a model.
Consider Colored-MNIST setting (age-old) where a model is trained on digits in a specific color always (say it sees zeros always in red). What would you expect to happen? It learns to associate digit with a specific color (sees red, says zero)! Now when such model is tested on reversed color digits, it fails badly (nearly zero accuracy).
Classifier learns to associate color with digits during training; in test, gets zero accuracy.
How to debias this model using steering vectors?
We can find directions of color red or simply color and steer model activations away from this direction simply. Lets try it out.
Concept loss based steering (with concepts from larger model)
41.8
Add-add adds difference between positive (here red color) vs negative (here random blobs) activations to the model layer and steers the activations with this transformation (read og paper for more details)
This can correlate with biased concepts; I did a controlled experiment in my NeurIPS work where I checked cosine distance of CAV_red (trained with red patches vs random - random color blobs) with activations of red-patches vs red-zeros.
What would you expect?
cs('red', CAV_red) < cs('red_zeros', CAV_red)
CAV_red was closer to red_zeros activations over red patches which meant model confused red with red-zeros.
If you steer using such a CAV you are essentially steering in model's confused concept direction (red_zeros not red). This confound could be overcome with the use of a teacher or large unbiased model for concepts.
But the next question was how to map these CAVs from teacher to student?
I used a simple mapping module for it.
I used a small auto-encoder to map teacher's Concept activations for concept set of interest to student's activations and trained for just a few epochs (just enough to not overfit - check paper for details).
Next, I learned CAVs in mapped teacher space and use these CAVs to debias student.
Full pipeline below:
Thus, a teacher could be used to debias a model using concept activation based steering and a loss.
I would direct readers to read the paper for more details.
Getting back in the safety field after 4 years feels great. I had a paper on concept distillation where I showed how steering vectors based method did'nt help to debias model since the concepts itself are understood wrongly [Concept distillation, NeurIPS 2023].
I am so happy to see the steering vectors work picked up in interpretability and was curious to see how methods like ActAdd would work in this debiasing setting?
So, I apply ActAdd to debias a model.
Consider Colored-MNIST setting (age-old) where a model is trained on digits in a specific color always (say it sees zeros always in red). What would you expect to happen? It learns to associate digit with a specific color (sees red, says zero)! Now when such model is tested on reversed color digits, it fails badly (nearly zero accuracy).
Classifier learns to associate color with digits during training; in test, gets zero accuracy.
How to debias this model using steering vectors?
We can find directions of color red or simply color and steer model activations away from this direction simply. Lets try it out.
Results (will be discussed one by one below)
Method
Mean Test Accuracy (3 seeds)
Base model
0.0
Base model on grayscale inputs
16.9
ActAdd, single pair (notebook)
21.4
Random vector, same norm
19.0
Project out the mean-diff direction
9.4
Concept loss based steering (w/o teacher)
27.0
Concept loss based steering (with concepts from larger model)
41.8
Add-add adds difference between positive (here red color) vs negative (here random blobs) activations to the model layer and steers the activations with this transformation (read og paper for more details)
This can correlate with biased concepts; I did a controlled experiment in my NeurIPS work where I checked cosine distance of CAV_red (trained with red patches vs random - random color blobs) with activations of red-patches vs red-zeros.
What would you expect?
cs('red', CAV_red) < cs('red_zeros', CAV_red)
CAV_red was closer to red_zeros activations over red patches which meant model confused red with red-zeros.
If you steer using such a CAV you are essentially steering in model's confused concept direction (red_zeros not red). This confound could be overcome with the use of a teacher or large unbiased model for concepts.
But the next question was how to map these CAVs from teacher to student?
I used a simple mapping module for it.
I used a small auto-encoder to map teacher's Concept activations for concept set of interest to student's activations and trained for just a few epochs (just enough to not overfit - check paper for details).
Next, I learned CAVs in mapped teacher space and use these CAVs to debias student.
Full pipeline below:
Thus, a teacher could be used to debias a model using concept activation based steering and a loss.
I would direct readers to read the paper for more details.
Thanks for reading!
Code in https://github.com/avani17101/CD/tree/cd_actadd_comparison