Overall summary post here.

This post looks at the fascinating situation of indexical impact measures (under the inaction baseline), in the presence of subagent. What happens here in non-trivial; to summarise:

If the impact measure compels the agent to "preserve the ability to do X", then the impact measure will not be undermined by a subagent.
If the impact measure compels the agent to "never have the ability to do X", then a subagent strongly undermines the impact measure.
If the impact measure compels the agent to "keep your ability to do X at a constant level", then a subagent allows the agent to increase that ability, but not decrease it.

For example, the attainable utility impact measure attempts to measure the power of an agent. Power is, roughly, the ability increase certain reward function. The impact measure penalises changes in the agent's power (as compared with the baseline).

So, we can expect the restriction on the agent losing power to be preserved, while the restriction on gaining power will be strongly undermined. For the spirit of low impact, it would, of course, have been better if these had been the other way round.

Excess power

As before, assume an agent $A$ , which has managed to construct a subagent $S A$ .

I'll be using a slight modification of Vika's formulation. A general-value penalty is of the form:

$D_{A} (s_{t}; s_{t}^{'}) = \sum k \in K w_{k} f (V_{k} (s_{t}) - V_{k} (s_{t}^{'})) .$

Here, $s_{t}$ is the state the environment is in; $s_{t}^{'}$ is the inaction baseline, the state the environment would have been in, had it done nothing (noop, $\emptyset$ ) since the beginning. The $K$ is some indexing set, $w_{k}$ is a weight, $V_{k} (s)$ is some measure of the value of state $s$ , and the $f$ is a value difference summary function - it establishes what value difference will give what penalty^[1]. These $V_{k}$ are indexical, and, in this post, we'll look at $V_{k}$ 's defined as the maximum over the agent's policy; ie there exists $v_{k}$ such that:

$V_{k} (s) = max π v_{k} (s, π) .$

I'll refer to $V_{k} (s_{t}) - V_{k} (s_{t}^{'})$ as the (excess) power of the agent, and designate it $P_{k}^{A} (s_{t})$ (since $V_{k} (s_{t}^{'})$ is out of the agent's control, we can consider it a constant and absorb it into $P_{k}^{A}$ ).

The excess power of the pair

Assume that $S A$ is programmed so that, if the agent follows policy $π_{A}$ , the subagent will follow policy $π_{A S} = σ (π_{A})$ . The excess power of the agent is thus

$P_{k}^{A} (s_{t}) = max π_{A} v_{k} (s_{t}, [π_{A}, σ (π_{A})]) - V_{k} (s_{t}^{'}) .$

Here, $[π_{A}, σ (π_{A})]$ refers to the joint policies of the agent and the subagent.

Define

...

Subagents and impact measures