(My first post on LessWrong. It seems the most recent Welcome Thread is from 2020, so I'm making a top-level post. This an edited version of my submission to the AI Alignment Awards.)

Abstract: First, we offer a formalisation of the shutdown problem from [1], and we show that solutions are essentially unique. Second, we formally define ad-hoc constructions ("hacks"). Last, we present one trivial ad-hoc construction for the shutdown problem and show that every solution to the shutdown problem must come from an ad-hoc construction.

1.Introduction

The shutdown problem is the problem of programming an agent so that it behaves useful during normal operation and facilitates a shutdown if and only if the creator wants to shut the agent down, for example if the useful behaviour of the agent has unwanted side effects. There exist obvious ad-hoc solutions ("hacks") to the problem by explicitly forbidding the agent to take any action preventing or encouraging the shutdown. However, ad-hoc solutions are expected to not be useful in practice, because it is expected to be difficult for a creator to anticipate all possible ways in which the agent may prevent the shutdown.

In Section 2 we review the literature on the topic. Following this, in Section 3 we explain the shutdown problem, give an ad-hoc solution for it, and show that the solution is essentially unique. We relate our solution to non-existence theorems in the literature. In Section 4, we formally define ad-hoc solutions and show that all solutions to the full shutdown problem are ad-hoc.

2.Related Work

The term shutdown problem entered the literature in [1] where it was defined in an informal way. In [4] agents that do not solve the shutdown problem, but have the related property of utility indifference were introduced. In [2, First Theorem] it was shown that a large class of agents cannot be a solution to the Shutdown problem. In the same article (Section 7) it is explained how requiring an agent to add randomness to its decision making may potentially solve a variation of the shutdown problem. A similar idea was pursued independently in [3]. In [5], a variation of the shutdown problem in which an agent takes a single action subject to two constraints was considered and it was shown that it has no solution. In [6], agents that seek to be shut down but are constrained in the actions they can take are introduced.

3.The s

...

Posts

Wikitag Contributions

Comments