Subhash, Josh, and David were co-first authors on this project. The full paper is here, and our code can be found here.

TLDR

We empirically study the success of weak-to-strong oversight as we scale the intelligence of the weak overseer model (Guard) and strong adversary model (Houdini) in four oversight game settings. We use those insights to theoretically investigate how to optimally design nested scalable oversight (NSO) protocols to control future superintelligences.

Introduction

Scalable oversight has been proposed in many forms as a way to control future superintelligences.^[1] These methods share a central goal: enabling weaker systems to oversee stronger ones (weak-to-strong oversight), ultimately enabling humans to oversee systems with superhuman cognitive abilities. A key idea is that scalable oversight can be bootstrapped: weaker systems oversee stronger ones, which then oversee even more capable models in the next stage—allowing oversight to scale alongside capabilities recursively. We refer to this iterated process as Nested Scalable Oversight, or NSO for brevity.

While many frontier labs have embraced the idea of bootstrapping oversight to control potentially dangerous superintelligences, most guarantees surrounding NSO have been qualitative, focusing on plausibility and conceptual design. However, the safety standards applied to other high-risk technologies are quantitative. For instance, the U.S. requires civilian aircraft to have a fatality rate below 1/100,000 per flight hour, and nuclear reactors to maintain a core damage frequency below 1/10,000 per year. Motivated by these risk thresholds, our work takes a first step towards quantifying the failure rates of NSO systems by studying the weak-to-strong oversight problem.

In this blog post, we highlight a few key contributions from the full paper

Modeling oversight as a game played between a weak overseer (Guard) and strong adversary (Houdini)
Proposing that oversight and deception performance take the shape of a Double ReLU as a function of general model intelligence
Validating that hypothesis in four realistic oversight setups: Mafia, Debate, Backdoor Code, and Wargames. Wargames is a novel oversight game.
Using our framework to theoretically study how to best design real world nested scalable oversight systems

General Setup

We model weak-to-strong oversight as a game between a...