TLDR: This is the abstract, introduction and conclusion to the paper. See here for a summary thread.

Abstract

Do large language models (LLMs) anticipate when they will answer correctly? To study this, we extract activations after a question is read but before any tokens are generated, and train linear probes to predict whether the model’s forthcoming answer will be correct. Across three open-source model families ranging from 7 to 70 billion parameters, projections on this “in-advance correctness direction” trained on generic trivia questions predict success in distribution and on diverse out-of-distribution knowledge datasets, outperforming black-box baselines and verbalised predicted confidence. Predictive power saturates in intermediate layers, suggesting that self-assessment emerges mid-computation. Notably, generalisation falters on questions requiring mathematical reasoning. Moreover, for models responding “I don’t know”, doing so strongly correlates with the probe score, indicating that the same direction also captures confidence. By complementing previous results on truthfulness and other behaviours obtained with probes and sparse auto-encoders, our work contributes essential findings to elucidate LLM internals.

Introduction

Large language models (LLMs) internally encode information beyond what is immediately observable in their output. Studies have demonstrated that hidden activations can reveal statement truthfulness, deception and hallucination, which have become ever more critical as LLMs are deployed in increasingly complex, high-stakes real-world applications. In this work, we investigate the complementary scientific question of whether an LLM’s residual stream activations—captured immediately after it processes a query—contain a latent signal that predicts if its eventual output will be correct. Instead of using the generated answer or token probabilities, we train a linear probe on the hidden state before the answer is produced to distinguish questions the model will answer correctly from those it will not, capturing its internal prediction of correctness.

Empirically, our approach identifies the activation-space vector linking the average residual stream activations for correctly answered questions to those for incorrectly answered ones (similar to Burger et al.’s for statement truthfulness). We test our approach on state-of-the-art open-source L...

Posts

Wikitag Contributions

Comments