Zoe Faith Gumise

I thought I had found an induction head. The attention pattern looked familiar, and one head received a substantially higher induction score than the others. The obvious interpretation was that this head was responsible for the model's copying behavior. Then I removed it. The model's accuracy did not change. I...