x

Julian Minder

Message

PhD @ EPFL with Robert West. MATS 7 Scholar with Neel Nanda. Interested in mechanistic interpretability and the what the process of finetuning does to models.

357

Ω

22

2

7

1y

Julian Minder

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity. The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post. Thread New paper: We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language. We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so. We aim to make a general-purpose LLM for explaining activations by: 1. Training on a diverse set of tasks 2. Evaluating on tasks very different from training This extends prior work (LatentQA) that studied activation verbalization in narrow settings. Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies. Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like! We compare Activation Oracles (AOs) against prior techniques on these auditing tasks. The result: AOs beat all methods on 2/3 secret keeping evals (and 3/3 when only including white-box). Even better, AOs work well out-of-the-box with no task-specific scaffolding or tuning. We evaluate on model diffing: given the difference between base & finetuned model activations, can AOs describe what changed? Despite never training on difference vectors, AOs match specialized interp baselines in identifying the distinctive quirk of emergently misaligned models We think Activation Oracles are promising for two reasons: 1. Scalability. Performance reliably increases with the number of datasets in the training mix 2. S

154Dec 18, 2025

Message

PhD @ EPFL with Robert West. MATS 7 Scholar with Neel Nanda. Interested in mechanistic interpretability and the what the process of finetuning does to models.

357

Ω

22

2

7

1y

Synthetic Persona Pretraining: Alignment from Token Zero

Julian Minder, Viktor Moskvoretskii, Raghav Singhal, Difan Jiao, Kartik Bali, Yiderigun Borjigin, Shaobo Cui, Stefan Krsteski, Ashton Anderson, Roland Aydin, Robert West (equal contribution) > These are early results, but we wanted to share them with the community now. We will release all artifacts (scaled-up runs, models, code, data, intermediate...

May 2083

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity...

Dec 18, 2025154

Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences

The work was done as part of the MATS 7 extension. We'd like to thanks Cameron Holmes and Fabien Roger for their useful feedback. Edit: We’ve published a paper with deeper insights and recommend reading it for a fuller understanding of the phenomenon. TL;DR Claim: Narrow finetunes leave clearly readable...

Sep 5, 202554

What We Learned Trying to Diff Base and Chat Models (And Why It Matters)

This post presents some motivation on why we work on model diffing, some of our first results using sparse dictionary methods and our next steps. This work was done as part of the MATS 7 extension. We'd like to thanks Cameron Holmes and Bart Bussman for their useful feedback. Could...

Jun 30, 2025106

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity. The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post. Thread New paper: We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language. We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so. We aim to make a general-purpose LLM for explaining activations by: 1. Training on a diverse set of tasks 2. Evaluating on tasks very different from training This extends prior work (LatentQA) that studied activation verbalization in narrow settings. Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies. Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like! We compare Activation Oracles (AOs) against prior techniques on these auditing tasks. The result: AOs beat all methods on 2/3 secret keeping evals (and 3/3 when only including white-box). Even better, AOs work well out-of-the-box with no task-specific scaffolding or tuning. We evaluate on model diffing: given the difference between base & finetuned model activations, can AOs describe what changed? Despite never training on difference vectors, AOs match specialized interp baselines in identifying the distinctive quirk of emergently misaligned models We think Activation Oracles are promising for two reasons: 1. Scalability. Performance reliably increases with the number of datasets in the training mix 2. S

154Dec 18, 2025

What We Learned Trying to Diff Base and Chat Models (And Why It Matters)

106Jun 30, 2025

Synthetic Persona Pretraining: Alignment from Token Zero

83May 20

Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences

54Sep 5, 2025