LESSWRONG
LW

Helena Casademunt — LessWrong

Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation

Bartosz Cywiński, Helena Casademunt, Khoi Tran, aryaj, Sam Marks, Neel Nanda

Ω 171mo

TL;DR: We introduce a testbed based on censored Chinese LLMs, which serve as natural objects of study for studying secret elicitation techniques. Then we study the efficacy of honesty elicitation and lie detection techniques for detecting and removing generated falsehoods.

This post presents a summary of the paper, including examples of transcripts and other miscellaneous findings.

arXiv paper | Code | Transcripts

Summary

We construct a testbed for honesty elicitation and lie detection techniques comprising questions on censored topics and corresponding ground-truth facts.
Of the honesty elicitation techniques we evaluate, sampling without a chat template, few-shot prompting, and fine-tuning on generic honesty data most reliably increase truthful responses.
The strongest inference-time honesty elicitation techniques also transfer to frontier open-weights models, including DeepSeek-R1-0528, Qwen3.5-397B and MiniMax-M2.5.
For lie detection, prompting the censored model to classify

...

(Continue Reading - 1458 more words)

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

Helena Casademunt6mo30

I agree that there’s still a lot to be understood!

I predict that for the model where only-ablate-activations performed well, the ablation is actually steering towards misalignment (whereas for the other model, zero-ablation is steering away from, or orthogonally to, misalignment).

I didn’t directly check this by looking at the effect of inference-time steering on the model before/after finetuning. But I have tried train-time steering with the mean projection of the CAFT directions over the training data. The results are consistent with your hypothesis... (read more)

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

Helena Casademunt6mo10

Sorry I missed this. This other comment should address this too.

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

Helena Casademunt6mo30

The goal of CAFT is to cause the model to learn some behavior without that behavior being mediated by some researcher-chosen subspace. For this, we can ablate that subspace to any constant value (zero, mean, or anything else), since this eliminates the causal effect of that subspace (and therefore prevents gradients from flowing through it). So it’s true that zero ablation is an arbitrary choice, but from our perspective we were happy to make an arbitrary choice.

That said, it’s true that the choice of constant ablation can matter via a similar mechanism as... (read more)

Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning

kh4dien, Helena Casademunt, Adam Karvonen, Sam Marks, Senthooran Rajamanoharan, Neel Nanda

Ω 379mo

Summary

We introduce an interpretability-based technique for controlling how fine-tuned LLMs generalize out-of-distribution, without modifying training data.
We show it can mitigate emergent misalignment by training models that write insecure code without becoming misaligned.
It can also reduce sensitivity to spurious cues, even when they are present in 100% of fine-tuning data.
Our technique works by identifying the concept directions used in the undesired solution (e.g. misalignment directions) and ablating them during fine-tuning. The model learns to rely on other concepts, causing different generalization.

Full paper | Twitter thread

Introduction

LLMs can have undesired out-of-distribution (OOD) generalization from their fine-tuning data. A notable example is emergent misalignment, where models trained to write code with vulnerabilities generalize to give egregiously harmful responses (e.g. recommending user self-harm) to OOD evaluation questions.

Once an AI developer has noticed this...

(Continue Reading - 1386 more words)