Investigating what activation patching measures in reward models
TL;DR In reward-lens: A Mechanistic Interpretability Library for Reward Models (Nadaf, 2026), the author reports that linear attribution does not predict the effect of causal activation patching. This post reproduces that at scale and re-diagnoses what it means: * At early layers, splicing in activations from an unrelated conversation moves...
Oct 11