I believe that mechanistic interpretability should attempt to find the "true features" (in the sense of computational units, abstractions the model uses, see here for discussion).

I think a promising area is observing the response of a model to activation perturbations into various directions. Why? Because this is close to data-free (we want to know what features the model learned, not which ones are in the data) and just asks does the model priviledge this direction?

I'll write more about this soon, but for now I want to shout out my mentee Francisco's work which I consider a proof of concept of this idea: LLMs do somewhat priviledge known feature-ish directions (steering directions, SAE decoder directions) and don't priviledge baselines (PCA, random directions). We measure this with a L_p norm fit (p=2 means non-priviledged) and find a significant p=2.3. However we suspect (based on toy models) a even-higher p for "true features". Our metric isn't perfect, notably it's not Goodhart-proof: if we brute-force optimize for p we find pathological directions.

I'm very excited about work in this area: Can we find some properties of models (like perturbation sensitivity) that tell us about its representations?