There's two types of interpretability techniques: variable and algorithm interpretability, ones that find the features vs the circuits. J-Lens is very much about variable interpretability. We shouldn't expect it to tell us much about how the variables are being computed, except what we can infer by viewing and intervening on the variables themselves.
There is an alternative view of the Jacobian for certain nonlinear networks where it can capture exactly how the variable or representation is being computed (for a particular input). A deep network that only u... (read more)
There is an alternative view of the Jacobian for certain nonlinear networks where it can capture exactly how the variable or representation is being computed (for a particular input). A deep network that only u... (read more)