People spend a lot of words playing tug of war over whether or not it's reasonable to train against interpretability methods. The anti case goes something like "training based on interpreted features trains against Interpretability itself more than it trains against whatever features you're detecting". There are cases where we should expect this to be true and cases where we should expect this to be not true. It basically comes down to how much the model can encrypt/obfuscate the relevant features without sabotaging its own cognition, as well as how strong the optimization pressure to have the relevant features is. So for example if you have an AI system that learns to avoid shutdown for instrumental convergence reasons, and you train against this with a linear probe on SAE features or something I would not expect this to be a robust method. If you are using some method that captures the deep structure of the models cognition, perhaps based on something like Wentworth's natural latents, to divert it away from a non-essential Goodhart-y behavior early in the training that then forms the basis of a non-cheating self evaluation later I would expect this to probably be robust to optimization.
So, rather than yelling at anyone who trains on interpretability probes in any context for doing "the most forbidden technique" first ask if they have a coherent story for why they expect the features their Interpretability relies on to be much more robust to optimization than the misbehavior they're trying to detect.
Then yell at them, because they almost certainly don't have that story but you should check first.
People spend a lot of words playing tug of war over whether or not it's reasonable to train against interpretability methods. The anti case goes something like "training based on interpreted features trains against Interpretability itself more than it trains against whatever features you're detecting". There are cases where we should expect this to be true and cases where we should expect this to be not true. It basically comes down to how much the model can encrypt/obfuscate the relevant features without sabotaging its own cognition, as well as how strong the optimization pressure to have the relevant features is. So for example if you have an AI system that learns to avoid shutdown for instrumental convergence reasons, and you train against this with a linear probe on SAE features or something I would not expect this to be a robust method. If you are using some method that captures the deep structure of the models cognition, perhaps based on something like Wentworth's natural latents, to divert it away from a non-essential Goodhart-y behavior early in the training that then forms the basis of a non-cheating self evaluation later I would expect this to probably be robust to optimization.
So, rather than yelling at anyone who trains on interpretability probes in any context for doing "the most forbidden technique" first ask if they have a coherent story for why they expect the features their Interpretability relies on to be much more robust to optimization than the misbehavior they're trying to detect.
Then yell at them, because they almost certainly don't have that story but you should check first.