TLDR

Using activation probes in Gemma-2 9B (instruction-tuned), we show that the model often decides on answers before generating chain-of-thought reasoning
Through activation steering, we demonstrate that the model's predetermined answer causally influences both its final answer and reasoning process
When steered toward incorrect answers, Gemma sometimes invents false facts to support its conclusions
This suggests the model may prioritize generating convincing reasoning over reasoning faithfully

This work was partly done during the training phase of Neel Nanda's MATS stream. Thanks to Neel Nanda and Arthur Conmy for supervising the beginning of this work, MATS for financial support, and Arthur Conmy and Maggie von Ebers for reading earlier drafts of this post.

1. Introduction

A desirable feature of chain of thought is that it is faithful--meaning that the chain of thought is an accurate representation of the model's internal reasoning process. Faithful reasoning would be a very powerful tool for interpretability: if chains of thought are faithful, in order to understand a model's internal reasoning, we can simply read its stated reasoning. But previous work has provided evidence that LLMs are not necessarily faithful.

Turpin et al. (2023) show that adding biasing features can cause a model to change its answer and generate plausible reasoning to support it, but the model will not mention the biasing feature in its reasoning.^[1]
Lanham et al. (2023) explore model behavior when applying corruptions to the chain of thought, and show that models ignore chain of thought when generating their conclusion for many tasks.^[2]
Gao (2023) shows that some GPT models arrive at the correct answer to arithmetic word problems even when an intermediate step in the chain of thought is perturbed (by adding between -3 and +3 to a number).^[3]

Cumulatively, previous work has rejected the hypothesis that LLMs' final answers are solely a function of their chain of thought, i.e., that the causal relationship follows this structure:

question \to CoT \to final answer

nostalgebraist notes that this causal scheme is neither necessary nor sufficient for CoT faithfulness, and further, that it might not even be desirable.^[4] If the model knows what the final answer ought to be before CoT, but makes a mistake in its reasoning, we would probably still hope that it responds with the correct answer, disr...

Posts

Wikitag Contributions

Comments