This is a great post and I'm looking forward to checking out the fits in the repo. FWIW, I've been doing some exploration with the J-lens on Gemma 2, and when I measured per-prompt margin gradients directly (VJPs at a fixed readout position) I found they're mutually coherent at every early layer; median cosine to the population mean ~0.72–0.80 through the exact layers where J-lens readouts are noisiest. So it seems there is a stable direction to find there, the standard readout just doesn't see it.
Question on the trash tokens: did you check them against un... (read more)
This is a great post and I'm looking forward to checking out the fits in the repo. FWIW, I've been doing some exploration with the J-lens on Gemma 2, and when I measured per-prompt margin gradients directly (VJPs at a fixed readout position) I found they're mutually coherent at every early layer; median cosine to the population mean ~0.72–0.80 through the exact layers where J-lens readouts are noisiest. So it seems there is a stable direction to find there, the standard readout just doesn't see it.
Question on the trash tokens: did you check them against un... (read more)