TL;DR: The IOI algorithm doesn't specify what to do when the indirect object token is duplicated. I observe that the model succeeds anyway. I'd like to know if I'm mistaken in the IOI paper's predictions, and I'd like to know what evidence you'd reach for next.
Background
Wang et al. (2022) addressed a grammatical task called Indirect Object Identification. Given text such as "When Mary and John went to the store, John gave a drink to", continuing the text requires identifying to what object John gives the drink. Quoting from Interpretability in the Wild, section 3:
The following human-interpretable algorithm suffices to perform this task:
Identify all previous names in the sentence (Mary, John, John).
Remove all names that are duplicated (in the example above: John).
Output the remaining name.
Below we present a circuit that we claim implements this functionality.
To paraphrase their claimed circuit and set out common terms:
Duplicate Token Heads (DTH) use fairly well-understood mechanisms to recognize and indicate tokens which have occurred earlier in the context.
Subject Inhibition Heads (S-Inhibition) remove the so-indicated tokens from the attention of...
Name Mover Heads (NM) move information about the Indirect Object so it's available for output.
The experiment
Across a range of templates like "Later, John and Mary went to ___ Park. John gave a drink to", replace the modifier slot "___" with either the indirect object token ("Mary Park", the test condition) or an unrelated plausible name ("Cedar Park", the control condition). Under the test condition, the IO and S tokens are both duplicated. If S-Inhibition removes duplicate tokens as hypothesized, the incidental "Mary" should cause inhibition roughly as much as the duplicate "John", narrowing logit differences. If S-Inhibition removes tokens on some other condition, the incidental "Mary" might not alter logit differences very much.
I ran matched pairs with the same template, names, object, and place noun, only differing in whether the modifier slot (M) is the IO name itself or a neutral non-name word. Across 8 templates and 16 name-pairs = 128 matched pairs, selected for single-token names.
Result
The drop in IO/S logit difference is small: mean 0.23, SD 0.84 pair-to-pair (n = 128), paired t = 3.09, p = 0.0024, Wilcoxon and a sign-flip permutation test agree (p ≈ 0.003, p ≈ 0.002), Cohen's d = 0.27. A real effect, but not a big one. The model's accuracy (how often it predicts IO as more likely than S) moves from 95.4% (control) to 94.5% (test).
The IOI circuit looks engaged while this happens, not off doing something unrelated. Duplicate Token Heads fire on the incidental "Mary" about as hard as they fire on a real subject duplicate (~0.53–0.57 mean attention back to the real IO, vs. ~0.00–0.001 in the control, where there's nothing to fire on). S-Inhibition heads mostly keep attending to the S2 slot rather than the incidental duplicate.
Interesting notes: one head doesn't budge at all, two others shift a real amount toward the incidental duplicate while still substantially preferring S2 on average. Name Mover heads still mostly attend to the IO. (Aside: Negative Name Movers shift toward the incidental duplicate a lot more than the positive Name Movers do. I don't have a story for this. Flagging it in case it's useful to someone who works on those heads specifically.)
What I think this does and doesn't show
DTH flagging a duplicate token is not, on its own, sufficient to substantially inhibit the model's output of that token during the IOI task.
I haven't done the work to say how this fits into the paper's account of what drives S-Inhibition: no patching or logit decoding. So, on that note, I'd like to frame some known unknowns:
Have I understood the IOI hypothesis, and Wang et al.'s own circuit decomposition, correctly? I'd rather be corrected here than build on a misreading.
What evidence do you want to see first? I know what I intend to do (ablations to see what perturbs S-Inhibition under these test conditions), but I'd like input on this.
TL;DR: The IOI algorithm doesn't specify what to do when the indirect object token is duplicated. I observe that the model succeeds anyway. I'd like to know if I'm mistaken in the IOI paper's predictions, and I'd like to know what evidence you'd reach for next.
Background
Wang et al. (2022) addressed a grammatical task called Indirect Object Identification. Given text such as "When Mary and John went to the store, John gave a drink to", continuing the text requires identifying to what object John gives the drink. Quoting from Interpretability in the Wild, section 3:
To paraphrase their claimed circuit and set out common terms:
The experiment
Across a range of templates like "Later, John and Mary went to ___ Park. John gave a drink to", replace the modifier slot "___" with either the indirect object token ("Mary Park", the test condition) or an unrelated plausible name ("Cedar Park", the control condition). Under the test condition, the IO and S tokens are both duplicated. If S-Inhibition removes duplicate tokens as hypothesized, the incidental "Mary" should cause inhibition roughly as much as the duplicate "John", narrowing logit differences. If S-Inhibition removes tokens on some other condition, the incidental "Mary" might not alter logit differences very much.
I ran matched pairs with the same template, names, object, and place noun, only differing in whether the modifier slot (M) is the IO name itself or a neutral non-name word. Across 8 templates and 16 name-pairs = 128 matched pairs, selected for single-token names.
Result
The drop in IO/S logit difference is small: mean 0.23, SD 0.84 pair-to-pair (n = 128), paired t = 3.09, p = 0.0024, Wilcoxon and a sign-flip permutation test agree (p ≈ 0.003, p ≈ 0.002), Cohen's d = 0.27. A real effect, but not a big one. The model's accuracy (how often it predicts IO as more likely than S) moves from 95.4% (control) to 94.5% (test).
The IOI circuit looks engaged while this happens, not off doing something unrelated. Duplicate Token Heads fire on the incidental "Mary" about as hard as they fire on a real subject duplicate (~0.53–0.57 mean attention back to the real IO, vs. ~0.00–0.001 in the control, where there's nothing to fire on). S-Inhibition heads mostly keep attending to the S2 slot rather than the incidental duplicate.
Interesting notes: one head doesn't budge at all, two others shift a real amount toward the incidental duplicate while still substantially preferring S2 on average. Name Mover heads still mostly attend to the IO. (Aside: Negative Name Movers shift toward the incidental duplicate a lot more than the positive Name Movers do. I don't have a story for this. Flagging it in case it's useful to someone who works on those heads specifically.)
What I think this does and doesn't show
DTH flagging a duplicate token is not, on its own, sufficient to substantially inhibit the model's output of that token during the IOI task.
I haven't done the work to say how this fits into the paper's account of what drives S-Inhibition: no patching or logit decoding. So, on that note, I'd like to frame some known unknowns:
Code (matched-pair generation, stats, per-head attention breakdown) on GitHub.
References: