This post is based on work done during SPAR by Daria Ivanova, Yau-Meng Wong, Nicholas Chen, and Vishnu Vardhan with guidance from Rohan Subramani. We are grateful to Rauno Arike, Robert McCarthy, and Kei Nishimura-Gasparian for feedback.

TL;DR

We introduce two black-box approaches to detect hidden reasoning in a model’s chain of thought. Both are based on the idea that if a model conducts hidden reasoning, its subsequent reasoning steps will likely be harder for a smaller model to predict.

We find some promising results for detecting randomly removed reasoning steps and skipped steps related to scheming. We suggest next steps that could improve our methods and discuss how AI companies could integrate these methods alongside existing chain-of-thought monitoring techniques.

Our code and data are available here; see our poster here (won SPAR best poster award).

Introduction

Chain-of-thought (CoT) monitoring is a powerful tool for assessing how a model arrives at its output. For CoT monitoring to be reliable, models must generate CoTs that contain enough of the relevant thinking steps for us to oversee the model’s reasoning process accurately. Hidden reasoning—any portion of the model’s thinking that happens at the activation level but doesn’t appear at the token level—won’t be noticed by standard black-box CoT monitors. This weakness could be alleviated through hidden reasoning detection: if we can tell that significant steps are hidden from the CoT, we would put less trust in CoT monitors and be extra cautious with model outputs.

We investigate two ways to detect hidden reasoning. Both involve breaking the untrusted model’s CoT into discrete reasoning steps, and using a smaller, trusted model to predict the n+1st step given the first n. Intuitively, thought with leaps of logic should be harder than continuous thought to predict.

This work was heavily inspired by the “Punish Leaps of Logic / Incentivize Local Validity” proposal from Daniel Kokotajilo’s “Bag O’ Tricks,” which suggested using the signal of a smaller predictor model to train models with more continuous CoT reasoning.

Our perplexity method treats each token as a separate step, while our semantic method defines a step as a snippet of text. We assess these approaches on baseline, scheming, ablated-baseline, ablated-scheming, and scheming-removed CoTs from models s...

Posts

Wikitag Contributions

Comments