Subhash Kantamneni — LessWrong

Training Models to Predict and Explain Their In-the-Wild Behavior

by Adam Karvonen, Subhash Kantamneni, Euan Ong, and Sam Marks

Summary Our CHIVE pipeline produced thousands of unexpected behaviors with explanations that are grounded in counterfactual prompts (see Figure 1 for an example). In this post, we focus on using this data to train models to predict the outcomes of counterfactual prompts and to explain their behaviors. We build two...

Sep 450

Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments

by Adam Karvonen, Euan Ong, Subhash Kantamneni, and Sam Marks

TL;DR: We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits. We use the resulting data in two ways. Using it as an evaluation, we find that activation-reading interpretability tools provide no uplift: agents given the tools predict the...

Aug 2173

Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

Abstract > We introduce Natural Language Autoencoders (NLAs), an unsupervised method for generating natural language explanations of LLM activations. An NLA consists of two LLM modules: an activation verbalizer (AV) that maps an activation to a text description and an activation reconstructor (AR) that maps the description back to an...

May 7209

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

by Sam Marks, Adam Karvonen, James Chua, Subhash Kantamneni, Euan Ong, Julian Minder, Clément Dumas, and Owain_Evans

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity...

Dec 18, 2025155

Scaling Laws for Scalable Oversight

Subhash, Josh, and David were co-first authors on this project. The full paper is here, and our code can be found here. TLDR We empirically study the success of weak-to-strong oversight as we scale the intelligence of the weak overseer model (Guard) and strong adversary model (Houdini) in four oversight...

Apr 30, 202539

Takeaways From Our Recent Work on SAE Probing

by Josh Engels, Subhash Kantamneni, Senthooran Rajamanoharan, and Neel Nanda

Subhash and Josh are co-first authors on this work done in Neel Nanda’s MATS stream. We recently released a new paper investigating sparse probing that follows up on a post we put up a few months ago. Our goal with the paper was to provide a single rigorous data point...

Mar 3, 202530

Language Models Use Trigonometry to Do Addition

I (Subhash) am a Masters student in the Tegmark AI Safety Lab at MIT. I am interested in recruiting for full time roles this Spring - please reach out if you're interested in working together! TLDR This blog post accompanies the paper "Language Models Use Trigonometry to Do Addition." Key...

Feb 5, 202580