ARC recently released our first empirical paper: Estimating the Probabilities of Rare Language Model Outputs. In this work, we construct a simple setting for low probability estimation — single-token argmax sampling in transformers — and use it to compare the performance of various estimation methods. ARC views low probability estimation as a potential technique for mitigating worst-case behavior from AI, including deceptive alignment; see our previous theory post on estimating tail risks in neural networks for further discussion.

Outline of this post:

Problem statement and description of our setting
Estimation methods we compared: importance sampling and activation extrapolation
Summary of empirical results
Discussion, including motivation, limitations, related work, and future directions

This document describes joint research done with Jacob Hilton, with contributions from Paul Christiano (while at ARC prior to April 2024). There were also intermediate theoretical contributions from David Matolsci and George Robinson. Thanks additionally to Jean-Stanislas Denain and Eric Neyman for feedback.

Problem statement

Given a model $M : X \to Y$ , an input distribution $D \in Δ (X)$ , and a computable boolean property of outputs $C : Y \to {0, 1}$ , low probability estimation is the task of efficiently estimating
$Pr x \sim D [C (M (x)) = 1],$
even when this probability is too small to detect via naive sampling. For example, in ARC's previous post on estimating tail risks, $C$ represented a "catastrophe detector" that can tell us if an action $M (x)$ proposed by the model would result in large-scale harm.

In this work, we study low probability estimation in the setting of argmax sampling from language models with single-token behaviors. Let $M : V^{*} \to V$ be a transformer language model that predicts the next token given a string of previous tokens, where $V$ is the token vocabulary. Note that we sample at temperature $0$ , so $M$ is deterministic. Given a distribution $D$ over $V^{*}$ and a target token $t \in V$ , the low probability estimation problem for single-token behaviors is the task of estimating
$Pr x \sim D [M (x) = t]$

In general, $D$ can be any distribution that can be formally specified, such as the output distribution of generative model. However, in this paper we focus only on distributions $D$ in which every token position is independent.^[1] This is a pretty restrictive choice, and we expect future work to move beyond this assumption. H...

Alignment Research Center (ARC)