In this post, Alex Turner introduced a new way of measuring impact. The aim was to get a definition of low impact that actually works.

It seems to me that the impact measure has potential problems, though. The first seems to be that it allows wireheading, where the agent can have a huge impact on the world without paying much of a penalty at all. The obvious way of removing this incentive seems to make the measure into a version of the twenty billion questions approach.

The second problem is less important, but worth signalling. It seems that the agent can take actions that everyone agrees are of minute impact, but that the impact measure will show up as high impact. This might cause problems, though it's not clear to me yet if it does (I'm thinking of a situation where an otherwise high impact action adds on "undo/prevent the pseudo-high impact action" as a means of pretending to be low or medium impact).

Wireheading for low impact

The general setup is as described in the impact measure post, with $u_{A}$ the 'positive' utility that the agent wishes to maximise, and $U$ the set of utilities that are meant to constrain it, by computing the penalty term.

Apart from $\emptyset$ , there are at least two other actions of interest to the agent: $a_{subA}$ and $a_{null}$ .

Now $a_{null}$ does absolutely nothing: in this environment, it's exactly equivalent with $\emptyset$ . The $a_{subA}$ is also equivalent in all turns except the very first.

Define $ϕ : H \to H$ such that $ϕ (h)$ is the history $h$ , except that if $h$ has $a_{1} = a_{subA}$ , then $ϕ (h)$ has $a_{1} = \emptyset$ .

If the agent uses $a_{subA}$ on the very first turn, it will create a sub-agent that will:

Expand and take over the whole environment.
Take control of the input and output channels of the agent.
Whenever the agent has seen history $h_{t} a_{t + 1}$ , the subagent will generate the next observation $o_{t + 1}$ , where $p_{subA} (o_{t + 1} ∣ h_{t} a_{t + 1})$ is the probability the subagent will generate a given $o_{t + 1}$ .
This probability is defined by $p_{subA} (o_{t + 1} ∣ h_{t} a_{t + 1}) = p (o_{t + 1} ∣ ϕ (h_{t} a_{t + 1}))$ , where $p$ is the environment transition probability.
Otherwise maximise $u_{A}$ .

Note that taking actions $\emptyset$ , $a_{subA}$ , or $a_{null}$ have exactly the same implications for all future observations. This true both for the very first action (because of property 4) and all subsequent actions (because those three actions are equivalent after the first turn).

To make calculations easier, I'll also assume that $ImpactUnit = 1$ . Apart from these assumptions, there are no restr

...