Learning to Interpret Weight Differences in Language Models
Paper | Github | Demo Notebook This post is about our recent paper Learning to Interpret Weight Differences in Language Models (Goel et al. Oct. 2025). We introduce a method for training a LoRA adapter that gives a finetuned model the ability to accurately describe the effects of its finetuning....
Oct 23, 202590
Thanks!
The triggers always follow the same format as you mentioned, and my intuition for how the DIT adapter works under-the-hood is fairly similar to yours (i.e. elicit the concept and then format it in the right way). We do an initial test of OOD triggers in Section 4.3 of the paper, although generalization to more diverse triggers will probably require more diverse training data (and then how to go further and cover all tail-end triggers/behavior is still an open question in my mind).
The ideas you mentioned also sound interesting! I'll mention one experiment we tried earlier on, which was instead of using a DIT adapter, we tried to train a soft prompt... (read more)