Disclaimer : this title is intentionally provocative; results are highly experimental. TL;DR Most MLP neurons in Gemma-2-2B are dense and polysemantic, and static inspection of their activations is usually uninformative. However, ranking neurons by δ×gradient —the product of (i) activation difference between two minimally different prompts and (ii) the gradient...
Browse auto-genearted subgraphs on Neuronpedia: Dallas-Austin | Oakland Sacramento | Michael Jordan-Basketball Small-Opposite | Petit-Contraire | Muscle-Diaprhagm Summary: Attribution graphs reveal which features contribute to model outputs, but interpreting these features remains bottlenecked by manual analysis. I present an automated probe-prompting pipeline that extracts concept-aligned supernodes with minimal supervision. Features...
Disclaimer: English is not my first language. I used language models (Claude sonnet 4, GPT-4o) to help edit grammar and clarity after writing my ideas, and AI tools (OpenAI Deep Search, Google NotebookLM) for research assistance. All core concepts, experiments, and technical work are my own. Unedited AI content is...
Even without deception as an explicit goal, the model is always optimizing for acceptable outputs under observation while performing whatever internal computation is cheapest or most effective for it. This already creates pressure to separate surface plausibility from internal means. From that perspective, thinkish looks less like linguistic drift and more like an emergent covert channel between computation and action. In that sense, models may be doing a form of steganography by default, at least to some degree. I’m curious how you see recent advances in steganography research informing this picture, especially as models become more situationally aware.
Do you have any ideas for how to get interventions for larger “k” to work better?
In my runs, small k already flips the decision boundary (sometimes k≈5 in late layers), so “going large” mostly comes up when the behavior is more distributed (e.g. safety / refusal dynamics). The main reason large-k gets worse is that δ·grad is a first-order (local) guide, but once you patch a lot of neurons you’re no longer in the local regime: effects stop adding because of nonlinearities and normalization / downstream compensation.
A couple ideas that might improve large-k stability (with caveats):
Thanks, this is a fair push. What I’m claiming is narrower and (I think) more specific:
1.I’m not ranking by gradient alone, but by δ·gradient, where δ is the activation difference between two minimally different prompts (source vs destination). In practice, this seems to filter out “globally high-leverage but context-invariant” neurons and concentrate on neurons that are both (i) causally connected to the chosen decision axis and (ii) actually participating in the specific context change (e.g., Dallas→SF). Empirically, lots of high-gradient neurons have tiny δ and don’t do the transfer; lots of high-δ neurons have tiny gradient and also don’t do the transfer.
2. What I find surprising is that δ·gradient doesn’t just find... (read more)
Disclaimer : this title is intentionally provocative; results are highly experimental.
Most MLP neurons in Gemma-2-2B are dense and polysemantic, and static inspection of their activations is usually uninformative. However, ranking neurons by δ×gradient —the product of (i) activation difference between two minimally different prompts and (ii) the gradient of a specific logit objective — you isolate a small set of neurons with disproportionate causal influence on next-token decisions. These neurons also tend to show sharper, higher signal-to-noise activation peaks than average (≈+93% higher mean peak |z| in my probe setup), making their cross-prompt activation traces noticeably easier to interpret. Steering these neurons via gradient-guided delta patching can reliably shift model behavior, including... (read 8453 more words →)
Summary: Attribution graphs reveal which features contribute to model outputs, but interpreting these features remains bottlenecked by manual analysis. I present an automated probe-prompting pipeline that extracts concept-aligned supernodes with minimal supervision. Features are measured against targeted, concept-varying prompts, grouped by behavioral signatures, and validated with Neuronpedia’s Replacement and Completeness metrics. On two classic “capitals” circuits, subgraphs average 0.53 Replacement and 0.83 Completeness, outperforming geometric clustering on interpretability metrics; entity-swap tests suggest an early-layer/late-layer split in what generalizes.
State swap Demo: https://huggingface.co/spaces/Peppinob/state-swap-steering-explorer
Probe prompting Demo: https://huggingface.co/spaces/Peppinob/attribution-graph-probing
Repo: https://github.com/peppinob-ol/attribution-graph-probing
Arxiv Preprint: https://arxiv.org/abs/2511.07002
Epistemic Status: medium confidence in the current pipeline on small model highly sparse transcoder... (read 7846 more words →)
Disclaimer: English is not my first language. I used language models (Claude sonnet 4, GPT-4o) to help edit grammar and clarity after writing my ideas, and AI tools (OpenAI Deep Search, Google NotebookLM) for research assistance. All core concepts, experiments, and technical work are my own. Unedited AI content is marked in collapsible sections.
TL;DR This article challenges the dominant "instruction-following assistant" paradigm in prompt engineering. Instead, I explore the possibility to interact with language models as navigators of semantic space—focusing on placing strategic coordinates rather than issuing detailed instructions.
Definitely worth spending at least a few minutes on each of these. This is the kind of information that ends up saving you hours of work, sooner or later, over the coming months.
Coming fresh from the ARENA fellowship, I strongly feel that even a single dedicated day on tooling (possibly structured as an iterative, hands-on exercise) would pay off a lot. It would help anchor these references in memory, so they’re actually available when you need them later, rather than rediscovered ad hoc under time pressure.