Thanks for this experiment, super useful!
Your explanation: "The steering vector makes the model pay attention to the stylistic constraints instead of the actual question". What would be very useful here is also an "accuracy tax": how much accuracy on the actual question goes down when the model puts all its attention into constraint-following?
I want to see the "accuracy tax" because I wonder "how much elicitation is too much elicitation". If we were to get perfect controllability, but only at the cost of all ability to solve tasks - this would make me much... (read more)
Thanks for your comment Tomek!
I was thinking about it - should we use different prompts at deployment?
- On the one hand, it is good because it could show us whether it is only a "brittle trick"
- On the other hand, it is bad because we lose the comparability of base vs post-treatment. We want to conclude how an intervention (SFT, steering) impacts controllability. If we use a worse prompt to elicit it, we might be comparing apples with bananas
So I guess ideally, we would try different prompts both at post-treatment and in the pre-treatment model.