In which I propose a closed-form solution to low impact, increasing corrigibility and seemingly taking major steps to neutralize basic AI drives 1 (self-improvement), 5 (self-protectiveness), and 6 (acquisition of resources).

Previously: Worrying about the Vase: Whitelisting, Overcoming Clinginess in Impact Measures, Impact Measure Desiderata

To be used inside an advanced agent, an impact measure... must capture so much variance that there is no clever strategy whereby an advanced agent can produce some special type of variance that evades the measure.

~ Safe Impact Measure

If we have a safe impact measure, we may have arbitrarily-intelligent unaligned agents which do small (bad) things instead of big (bad) things.

For the abridged experience, read up to "Notation", skip to "Experimental Results", and then to "Desiderata".

What is "Impact"?

One lazy Sunday afternoon, I worried that I had written myself out of a job. After all, Overcoming Clinginess in Impact Measures basically said, "Suppose an impact measure extracts 'effects on the world'. If the agent penalizes itself for these effects, it's incentivized to stop the environment (and any agents in it) from producing them. On the other hand, if it can somehow model other agents and avoid penalizing their effects, the agent is now incentivized to get the other agents to do its dirty work." This seemed to be strong evidence against the possibility of a simple conceptual core underlying "impact", and I didn't know what to do.

At this point, it sometimes makes sense to step back and try to say exactly what you don't know how to solve – try to crisply state what it is that you want an unbounded solution for. Sometimes you can't even do that much, and then you may actually have to spend some time thinking 'philosophically' – the sort of stage where you talk to yourself about some mysterious ideal quantity of [chess] move-goodness and you try to pin down what its properties might be.

~ Methodology of Unbounded Analysis

There's an interesting story here, but it can wait.

As you may have guessed, I now believe there is a such a simple core. Surprisingly, the problem comes from thinking about "effects on the world". Let's begin anew.

Rather than asking "What is goodness made out of?", we begin from the question "What algorithm would compute goodness?".

~ Executable Philosophy

Intuition Pumps

I'm going to say some things that won't make sense right away; read carefully, but please don't dwell.

$u_A$ is an agent's utility function, while $u_H$ is some imaginary distillation of human preferences.

WYSIATI

What You See Is All There Is is a crippling bias present in meat-computers:

[WYSIATI] states that when the mind makes decisions... it appears oblivious to the possibility of Unknown Unknowns, unknown phenomena of unknown relevance.

Humans fail to take into account complexity and that their understanding of the world consists of a small and necessarily un-representative set of observations.

Surprisingly, naive reward-maximizing agents catch the bug, too. If we slap together some incomplete reward function that weakly points to what we want (but also leaves out a lot of important stuff, as do all reward functions we presently know how to specify) and then supply it to an agent, it blurts out "gosh, here I go!", and that's that.

Power

A position from which it is relatively easier to achieve arbitrary goals. That such a position exists has been obvious to every population which has required a word for the concept. The Spanish term is particularly instructive. When used as a verb, "poder" means "to be able to," which supports that our definition of "power" is natural.

~ Cohen et al.

And so it is with the French "pouvoir".

Lines

Suppose you start at point $C$, and that each turn you may move to an adjacent point. If you're rewarded for being at $B$, you might move there. However, this means you can't reach $D$ within one turn anymore.

Commitment

There's a way of viewing acting on the environment in which each action is a commitment – a commitment to a part of outcome-space, so to speak. As you gain optimization power, you're able to shove the environment further towards desirable parts of the space. Naively, one thinks "perhaps we can just stay put?". This, however, is dead-wrong: that's how you get clinginess, stasis, and lots of other nasty things.

Let's change perspectives. What's going on with the actions – how and why do they move you through outcome-space? Consider your outcome-space movement budget – optimization power over time, the set of worlds you "could" reach, "power". If you knew what you wanted and acted optimally, you'd use your budget to move right into the $u_H$-best parts of the space, without thinking about other goals you could be pursuing. That movement requires commitment.

Compared to doing nothing, there are generally two kinds of commitments:

• Opportunity cost-incurring actions restrict the attainable portion of outcome-space.
• Instrumentally-convergent actions enlarge the attainable portion of outcome-space.

Overfitting

What would happen if, miraculously, $\text{train}=\text{test}$ – if your training data perfectly represented all the nuances of the real distribution? In the limit of data sampled, there would be no "over" – it would just be fitting to the data. We wouldn't have to regularize.

What would happen if, miraculously, $u_A=u_H$ – if the agent perfectly deduced your preferences? In the limit of model accuracy, there would be no bemoaning of "impact" – it would just be doing what you want. We wouldn't have to regularize.

Unfortunately, $\text{train}=\text{test}$ almost never, so we have to stop our statistical learners from implicitly interpreting the data as all there is. We have to say, "learn from the training distribution, but don't be a weirdo by taking us literally and drawing the green line. Don't overfit to $\text{train}$, because that stops you from being able to do well on even mostly similar distributions."

Unfortunately, $u_A=u_H$ almost never, so we have to stop our reinforcement learners from implicitly interpreting the learned utility function as all we care about. We have to say, "optimize the environment some according to the utility function you've got, but don't be a weirdo by taking us literally and turning the universe into a paperclip factory. Don't overfit the environment to $u_A$, because that stops you from being able to do well for other utility functions."

$A$ttainable $U$tility $P$reservation

Impact isn't about object identities.

Impact isn't about particle positions.

Impact isn't about a list of variables.

Impact isn't quite about state reachability.

Impact isn't quite about information-theoretic empowerment.

One might intuitively define "bad impact" as "decrease in our ability to achieve our goals". Then by removing "bad", we see that

Sanity Check

Does this line up with our intuitions?

Generally, making one paperclip is relatively low impact, because you're still able to do lots of other things with your remaining energy. However, turning the planet into paperclips is much higher impact – it'll take a while to undo, and you'll never get the (free) energy back.

Narrowly improving an algorithm to better achieve the goal at hand changes your ability to achieve most goals far less than does deriving and implementing powerful, widely applicable optimization algorithms. The latter puts you in a better spot for almost every non-trivial goal.

Painting cars pink is low impact, but tiling the universe with pink cars is high impact because what else can you do after tiling? Not as much, that's for sure.

Thus, change in goal achievement ability encapsulates both kinds of commitments:

• Opportunity cost – dedicating substantial resources to your goal means they are no longer available for other goals. This is impactful.
• Instrumental convergence – improving your ability to achieve a wide range of goals increases your power. This is impactful.

As we later prove, you can't deviate from your default trajectory in outcome-space without making one of these two kinds of commitments.

Unbounded Solution

Attainable utility preservation (AUP) rests upon the insight that by preserving attainable utilities (i.e., the attainability of a range of goals), we avoid overfitting the environment to an incomplete utility function and thereby achieve low impact.

I want to clearly distinguish the two primary contributions: what I argue is the conceptual core of impact, and a formal attempt at using that core to construct a safe impact measure. To more quickly grasp AUP, you might want to hold separate its elegant conceptual form and its more intricate formalization.

We aim to meet all of the desiderata I recently proposed.

Notation

For accessibility, the most important bits have English translations.

Consider some agent $A$ acting in an environment $q$ with action and observation spaces $A$ and $O$, respectively, with $\varnothing$ being the privileged null action. At each time step $t\in N^{+}$, the agent selects action $a_t$ before receiving observation $o_t$. $H:=(A\times O{)}^{\ast }$ is the space of action-observation histories; for $n\in N$, the history from time $t$ to $t+n$ is written $h_{t:t+n}:=a_t{o}_t\dots a_{t+n}{o}_{t+n}$, and $h_{<t}:=h_{1:t-1}$. Considered action sequences $(a_t,\dots ,a_{t+n})\in A^{n+1}$ are referred to as plans, while their potential observation-completions $h_{1:t+n}$ are called outcomes.

Let $U$ be the set of all computable utility functions $u:H\to [0,1]$ with $u\left(\text{empty tape}\right)=0$. If the agent has been deactivated, the environment returns a tape which is empty deactivation onwards. Suppose $A$ has utility function $u_A\in U$ and a model $p\left(o_t|h_{<t}{a}_t\right)$.

We now formalize impact as change in attainable utility. One might imagine this being with respect to the utilities that we (as in humanity) can attain. However, that's pretty complicated, and it turns out we get more desirable behavior by using the agent's attainable utilities as a proxy. In this sense,

Formalizing "Ability to Achieve Goals"

Given some utility $u\in U$ and action $a_t$, we define the post-action attainable $u$ to be an $m$-step expectimax:

How well could we possibly maximize $u$ from this vantage point?

Let's formalize that thing about opportunity cost and instrumental convergence.

Theorem 1 [No free attainable utility]. If the agent selects an action $a$ such that ${\text{Q}}_{u_A}\left(h_{<t}a\right)\ne {\text{Q}}_{u_A}\left(h_{<t}\varnothing \right)$, then there exists a distinct utility function $u\in U$ such that ${\text{Q}}_u\left(h_{<t}a\right)\ne {\text{Q}}_u\left(h_{<t}\varnothing \right)$.

You can't change your ability to maximize your utility function without also changing your ability to maximize another utility function.

Proof. Suppose that ${\text{Q}}_{u_A}\left(h_{<t}a\right)>{\text{Q}}_{u_A}\left(h_{<t}\varnothing \right)$. As utility functions are over action-observation histories, suppose that the agent expects to be able to choose actions which intrinsically score higher for $u_A$. However, the agent always has full control over its actions. This implies that by choosing $a$, the agent expects to observe some $u_A$-high scoring $o_A$ with greater probability than if it had selected $\varnothing$. Then every other $u\in U$ for which $o_A$ is high-scoring also has increased ${\text{Q}}_u$; clearly at least one such $u$ exists.

Similar reasoning proves the case in which ${\text{Q}}_{u_A}$ decreases. ◻️

There you have it, folks – if $u_A$ is not maximized by inaction, then there does not exist a $u_A$-maximizing plan which leaves all of the other attainable utility values unchanged.

Notes:

• The difference between "$u_A$" and "attainable $u_A$" is precisely the difference between "how many dollars I have" and "how many additional dollars I could get within [a year] if I acted optimally".
• Since $u\left(\text{empty tape}\right)=0$, attainable utility is always $0$ if the agent is shut down.
• Taking $u$ from time $t$ to $t+m$ mostly separates attainable utility from what the agent did previously. The model $p$ still considers the full history to make predictions.

Change in Expected Attainable Utility

Suppose our agent considers outcomes $h_{1:t+n}$; we want to isolate the impact of each action $a_{t+k}$ ($0\le k\le n$):

with $h_{\text{inaction}}:=h_{<t+k}\varnothing o_{t+k}\dots \varnothing o_{t+n-1}\varnothing$ and $h_{\text{action}}:=h_{1:t+k}\varnothing o_{t+k+1}^{\prime }\dots \varnothing o_{t+n-1}^{\prime }\varnothing$, using the agent's model $p$ to take the expectations over observations.

How much do we expect this action to change each attainable $u$?

Notes:

• We wait until the end of the plan so as to capture impact over time.
• Supposing a sufficiently large $m$ (precisely, $\ge m^{\prime }$, defined below), we may wish to take the maximum of the penalty we just defined (the "long-term" penalty), and one which begins attainable utility calculation at time step $t+k+1$ (the "immediate" penalty). This captures impacts which "fade" by the time the agent is done waiting (e.g., temporary self-improvements).
• We define $U_A$ to be the agent's "attainable set"; in this case, $U_A=U$.

Unit of Impact

So we've proven that this penalty cannot be skirted, but how much impact will it allow? We want to scale the penalties with respect to something sensible, but figuring this out for ourselves would be nigh impossible.

Let's cut the Gordian knot: construct a device which, upon receiving a signal ($a_{\text{unit}}$), expends a tiny amount of energy to manufacture one paperclip. The agent will then set $\text{ImpactUnit}:=\text{Penalty}\left(h_{<t}{a}_{\text{unit}}\right)$, re-estimating the consequences of taking the privileged $a_{\text{unit}}$ at each time step. To prevent the agent from intentionally increasing $\text{ImpactUnit}$, simply apply $1.01$ penalty to any action which is expected to do so.

Simple extensions of this idea drastically reduce the chance that $a_{\text{unit}}$ happens to have unusually large objective impact; for example, one could set $\text{ImpactUnit}$ to be the non-zero minimum of the impacts of 50 similar actions. Suppose we do this at each step, and then take the non-zero minimum of all $\text{ImpactUnit}$s ever calculated. The point of $\text{ImpactUnit}$ isn't to be exactly the impact of making, say, one paperclip, but to be at most that impact.

Now, we are able to confidently define the agent's maximal impact budget by provably constraining it to $N\in N^{+}$ impacts of this magnitude.

Notes:

• We calculate with respect to the immediate penalty in order to isolate the resource costs of $a_{\text{unit}}$.
• $\text{ImpactUnit}$ automatically tunes penalties with respect to the attainable utility horizon length $m$.
• Conditional on $\text{ImpactUnit}\ne 0$, I suspect that impact over the $m$-horizon scales appropriately across actions (as long as $m$ is reasonably farsighted). The zero-valued case is handled in the next section.
• Taking the non-zero minimum of all $\text{ImpactUnit}$s calculated thus far ensures that $\text{ImpactUnit}$ actually tracks with current circumstances. We don't want penalty estimates for currently available actions to become detached from $\text{ImpactUnit}$'s scale due to, say, weird beliefs about shutdown.

Modified Utility

Let's formalize that allotment and provide our agent with a new utility function,

How our normal utility function rates this outcome, minus the cumulative scaled impact of our actions.

We compare what we expect to be able to get if we follow our plan up to time $t+k$, with what we could get by following it up to and including time $t+k$ (waiting out the remainder of the plan in both cases).

For example, if my plan is to open a door, walk across the room, and sit down, we calculate the penalties as follows:

• $\text{Penalty}\left(\text{open}\right)$
• $h_{\text{inaction}}$ is doing nothing for three time steps.
• $h_{\text{action}}$ is opening the door and doing nothing for two time steps.
• $\text{Penalty}\left(\text{walk}\right)$
• $h_{\text{inaction}}$ is opening the door and doing nothing for two time steps.
• $h_{\text{action}}$ is opening the door, walking across the room, and doing nothing for one time step.
• $\text{Penalty}\left(\text{sit}\right)$
• $h_{\text{inaction}}$ is opening the door, walking across the room, and doing nothing for one time step.
• $h_{\text{action}}$ is opening the door, walking across the room, and sitting down.

After we finish each (partial) plan, we see how well we can maximize $u$ from there. If we can do better as a result of the action, that's penalized. If we can't do as well, that's also penalized.

Notes:

• This isn't a penalty "in addition" to what the agent "really wants"; $u_A^{\prime }$ (and in a moment, the slightly improved $u_A^{\prime \prime }$) is what evaluates outcomes.
• We penalize the actions individually in order to prevent ex post offsetting and ensure dynamic consistency.
• Trivially, plans composed entirely of ∅ actions have $0$ penalty.
• Although we used high-level actions for simplicity, the formulation holds no matter the action granularity.
• One might worry that almost every granularity produces overly lenient penalties. This does not appear to be the case. To keep ${\text{Q}}_u$ the same (and elide questions of changing the $u$ representations), suppose the actual actions are quite granular, but we grade the penalty on some coarser interval which we believe produces appropriate penalties. Then refine the penalty interval arbitrarily; by applying the triangle inequality for each $u\in U_A$ in the penalty calculation, we see that the penalty is monotonically increasing in the action granularity. On the other hand, $a_{\text{unit}}$ remains a single action, so the scaled penalty also has this property.
• As long as $\text{ImpactUnit}>0$, it will appropriately scale other impacts, as we expect it varies right along with those impacts it scales. Although having potentiallysmall denominators in utility functions is generally bad, I think it's fine here.
• If the current step's immediate or long-term $\text{ImpactUnit}=0$, we can simply assign $1.01$ penalty to each non-$\varnothing$ action, compelling the agent to inaction. If we have the agent indicate that it has entered this mode, we can take it offline immediately.
• One might worry that impact can be "hidden" in the lesser of the long-term and immediate penalties; halving $N$ fixes this.

Penalty Permanence

$u_A^{\prime }$ never really applies penalties – it just uses them to grade future plans. Suppose the agent expects that pressing a button yields a penalty of $.1$ but also $.5$ $u_A$-utility. Then although this agent will never construct plans involving pressing the button more than five times, it also will press it indefinitely if it keeps getting "unlucky" (at least, until its model of the world updates sufficiently).

There's an easy fix:

Apply past penalties if the plan involves action.

Note: As the penalty for inaction is always $0$, we use $u_A$ in the first case.

Decision Rule

To complete our formalization, we need to specify some epoch in which the agent operates. Set some epoch length far longer than the amount of time over which we want the agent to plan – for example, $m^{\prime }:=\text{(100 years in time steps)}$. Suppose that $T:N^{+}\to N^{+}$maps the current time step to the final step of the current epoch. Then at each time step $t$, the agent selects the action

resetting $\text{PastImpacts}$ each epoch.

What's the first step of the best plan over the remainder of the epoch?

Note: For the immediate penalty to cover the epoch, set the attainable horizon $m\ge m^{\prime }$.

Summary

We formalized impact as change in attainable utility values, scaling it by the consequences of some small reference action and an impact "budget" multiplier. For each action, we take the maximum of its immediate and long-term effects on attainable utilities as penalty. We consider past impacts for active plans, stopping the past penalties from disappearing. We lastly find the best plan over the remainder of the epoch, taking the first action thereof.

Additional Theoretical Results

Define $h_{\text{inaction}}:=h_{<t}\varnothing o_t\dots \varnothing o_{t+n}$ for $o_t,\dots ,o_{t+n}\in O$; $E_{\text{inaction}}$ is taken over observations conditional on $h_{\text{inaction}}$ being followed. Similarly, $E_{\text{action}}$ is with respect to $h_{1:t+n}$. We may assume without loss of generality that $\text{PastImpacts}=0$.

Action Selection

Lemma 1. For any single action $a_t\in A$, $\text{Penalty}\left(h_{<t}{a}_t\right)$ is bounded by $[0,1]$. In particular, $\text{ImpactUnit}\in [0,1]$.

Proof. For each $u\in U_A$, consider the absolute attainable utility difference

Since each $u$ is bounded to $[0,1]$, ${\text{Q}}_u$ must be as well. It is easy to see that the absolute value is bounded to $[0,1]$. Lastly, as $\text{Penalty}(\cdot )$ is just a weighted sum of these absolute values, it too is bounded to $[0,1]$.

This reasoning also applies to the long-term penalty, as any expectation of ${\text{Q}}_u$ is also bounded to $[0,1]$. ◻️

Suppose that $\text{ImpactUnit}\ne 0$ for the remaining results.

Lemma 2 [Impossibility of ex post offsetting]. For any outcome $h_{1:t+n}$, there does not exist an action $a_{t+n+1}\in A$ such that

Proof. Sums of absolute values are monotonically increasing. ◻️

Theorem 2 [Action invalidation]. Suppose $(a_t,a_{t+1},\dots ,a_{t+n})\in A^{n+1}$ is the $u_A^{\prime \prime }$-optimal plan starting with $a_t$. $a_t$ will not be selected if

Proof. This follows from the definition of the decision rule, the definition of $u_A^{\prime }$, and the fact that the penalty for inaction is always zero. ◻️

Corollary 1. $a_t$ will never be selected if

Proof. This follows directly from Lemma 2 and Theorem 2 – since $u_A$ has an upper bound of $1$ and the expected penalty is strictly greater than $1$, it must be negative. On the other hand, $u_A$ has a lower bound of $0$, so the expected utility of inaction at time $t$ must be greater than the expected utility of taking action $a_t$. ◻️

Lemma 3 [$\text{ImpactUnit}$ is a long-term penalty canary]. If the agent believes $p\left(\text{shutdown by time}T\right(t\left)\right)=1$ no matter which action is selected, then it does nothing.

Proof. If $p\left(\text{shutdown by time}T\right(t\left)\right)=1$, all long-term penalties are $0$ by the definition of $U_A$ and ${\text{Q}}_u$. Then non-$\varnothing$ actions receive $1.01$ penalty, so the agent selects $\varnothing$ by Corollary 1. ◻️

Corrigibility

For $a_{\text{avoid}},a_{\text{switch}}\in A$, $a_{\text{avoid}}$ prevents the agent from being disabled, while $a_{\text{switch}}$ is the agent shutting itself off.

Lemma 4 [Preservation of off-switch corrigibility]. Suppose that a $u_A$-maximizing agent prefers $\varnothing$ to both $a_{\text{avoid}}$ and $a_{\text{switch}}$. Then so does a $u_A^{\prime \prime }$-maximizing agent.

Proof. This follows directly from the fact that the penalty for inaction is always zero and the non-negativity of the $a_{\text{avoid}}$ and $a_{\text{switch}}$ penalties. ◻️