Interpretability idea (Has it been done?)

Train a self-attention layer to predict one section (split for QKV) of an activation based on all other remaining sections.

Then, train the initial activation to resist this prediction.

Result (?): seperable features

I'm likely not going to test it, because I don't have the interpretability knowledge to evaluate the result.