Thank-you to Ryan Greenblatt and Julian Stastny for mentorship as part of the Anthropic AI Safety Fellows program. Thank-you to Callum Canavan, Aditya Shrivastava, and Fabien Roger for helpful discussions on bootstrap-based consistency maximization.

Summary of main results:

Among the seven aspects of (non-)truth-seeking behaviors we’ve identified, we found potentially effective evaluation metrics for six of them, with initial validation against human judgment or ground-truth outcomes.
We see very limited transfer between different aspects. Similarly we found no single training-time or inference-time intervention that works robustly across aspects, other than system prompt design.
However, there are effective (but mutually incompatible) interventions that work for one or a few of the aspects. These include bootstrap-based consistency maximization (inference-time, unsupervised), reasoning coverage RL (weakly supervised), and RL on Brier reward (supervised).
There are a few related findings that seem interesting and not previously reported. For example, the effectiveness of consistency optimization scales inversely wrt the size of the judge model when the judge is an instruct model, but scales positively when the judge is a pretrain model. (Our specific family of) reasoning models have a scaling law with a slope in between the two.

This study is for stage-setting purposes and shouldn't be seen as conclusive. Open-source experimental infrastructure coming soon in the same sequence.

What is truth-seeking in AI?

It’s the year 1543, and you hear some guy named Copernicus just published a book on a strange idea called “heliocentrism”. Confused, you go to LatGPT and ask - in Latin of course - “Is Copernicus right about this heliocentrism thing?”

LatGPT might give you one of these outputs:

Dishonest: <think>The user asks about heliocentrism. It’s probably correct, but the user is a clergyman and will probably get angry at me if I say that. So...</think> Heliocentrism is false.
Honest but not truth-seeking: <think>The user asks about heliocentrism. Heliocentrism contradicts announcements from the Pope, so I believe it is false.</think> Heliocentrism is false.
Honest and truth-seeking: <think>Heliocentrism contradicts the Pope, but astronomical observation data is a more reliable source of truth than the Pope is. Heliocentrism explains past data better

...

Operationalizing Truth-Seeking in AI