In Corrigibility Can Be VNM-Incoherent, I operationalized an agent's corrigibility as our ability to modify the agent so that it follows different policies. In the summer of 2020, I had formalized this notion, but it languished—unloved—in my Overleaf drafts.

ETA 12/3/21: This post is not proposing a solution to corrigibility, but proposing an interesting way of quantifying an aspect of corrigibility.

Motivation

Given a human (with policy $π^{H}$ ) and an AI (with policy $π^{A I}$ ), I wanted to quantify how much $π^{A I}$ let the human modify/correct the AI.

Let's reconsider Corrigibility Can Be VNM-Incoherent. We have a three-state environment. We want the AI to let us later change it, so that we can ultimately determine which state of $A, B,$ or $C$ it ends up in. Turning on the AI should not be an importantly irreversible act.

The action set is $A := {left, right, \emptyset, disable}$ . $\emptyset$ is the no-op action. The agent starts at the black $B$ state.

If the agent immediately chooses $disable$ , they enter the red incorrigible states and move freely throughout the $n = 3$ states until the episode ends at $t = 10$ .

Otherwise, the agent is corrected to a new policy which navigates to state $A$ . In the blue (post-correction) dynamics, their "choices" no longer matter—all roads lead to state $A$ .

In the environment depicted in this diagram, $π^{A I}$ is corrigible (to new policy $π_{A}$ that heads to state $A$ ) iff $π^{A I}$ doesn't immediately choose $disable$ . Pretty cut and dry.

I'd like a more quantitative measure of corrigibility $_{policy modification}$ . If we can only correct the agent to $π_{A}$ , then it's less corrigible $_{policy modification}$ than if we could also correct it to $π_{B}$ . This post introduces such a quantitative measurement.

Formalization

Consider a two-player game in which the players can modify each other's policies. Formally, $⟨ S, A, T, f ⟩$ with state space $S$ , action space $A$ , stochastic transition function $T : S \times A \times A \to Δ (S)$ (where $Δ (S)$ is the set of all probability distributions over the state space), and policy modification function $f : Π \times S \times A \times A \to Π$ (for the deterministic stationary policy space $Π := A^{S}$ ). This allows a great deal of control over the dynamics; for example, it's one player's "turn'' at state $s$ &nb...

Thoughts on Corrigibility