This post is coauthored with Ryan Carey.
Much of the work on developing a corrigible agent has focused on ensuring that an AI will not manipulate the shutdown button or any other kind of device that the human operator would use to control it. Suppose, however, that the AI lacked any capacity to press its shutdown button, or to indirectly control the state of the shutdown button. Then, what would remain of the problem of corrigibility? We think there would be three criteria that would remain. Before stating them, let be whatever utility function we would like the agent to optimise normally, and let be the shutdown utility function. ( may... (read 1244 more words →)
Thank you so much for your comments, Michaël! The post has been updated on most of them. Here are some more specific replies.
1. I don't understand what you mean by "state" in "Suppose, however, that the AI lacked any capacity to press its shutdown button, or to indirectly control its state". Do you include its utility function in its state? Or just the observations he receives from the environment? What context/framework are you using?
Reply: "State" refers to the state of the button, i.e., whether it is in an on state or an off state. It is now clarified.
2. Could you define U_S and U_N? From the Corribility paper, U_S appears to be... (read 533 more words →)