Corrigibility is a safety property for artificially intelligent agents. A corrigible agent will not resist attempts by authorized parties to alter the goals and constraints that were encoded in the agent when it was first started. This paper shows how to construct a safety layer that adds corrigibility to arbitrarily advanced utility maximizing agents, including possible future agents with Artificial General Intelligence (AGI). The layer counter-acts the emergent incentive of advanced agents to resist such alteration.
A detailed model for agents which can reason about preserving their utility function is developed, and used to prove that the corrigibility layer works as intended in a large set of non-hostile universes. The corrigible agents have an emergent incentive to protect key elements of their corrigibility layer. However, hostile universes may contain forces strong enough to break safety features. Some open problems related to graceful degradation when an agent is successfully attacked are identified.
The results in this paper were obtained by concurrently developing an AGI agent simulator, an agent model, and proofs. The simulator is available under an open source license. The paper contains simulation results which illustrate the safety related properties of corrigible AGI agents in detail.
This post can be used for comments and questions.
The paper contains several results and observations that do not rely on the heavy use of math, but other key results and discussions are quite mathematical. Feel to post questions and comments even if you have not read all the mathematical parts.
As this is my first post on LessWrong, and my first paper on AGI safety, I feel I should say something to introduce myself. I have a Ph.D. in software design, but my professional life so far has been very diverse and multidisciplinary. Among other things I have been an experimental physicist, a standards developer and negotiator, an Internet privacy advocate, a wireless networking expert, and a systems architect in an industrial research lab. So I bring a wide range of tools and methodological traditions to the field. What made me interested in the field of AGI safety in particular is that it seems to have open problems where real progress can be made using mathematical techniques that I happen to like. I am currently on a sabbatical: basically this means that I decided to quit my day job, and to use my savings to work for a while on some interesting problems that are different from the interesting problems I worked on earlier.
I think your AU agent is basically exactly what I mean by a lonely engineer: the key lonely engineer trick is to give the agent a utility function which is a mathematical entity that is entirely independent of its actions in the real world. I think on my first reading of your paper it seemed like you were trying to get the A agent to approximate the AU agent, but reading it more closely now it seems like what you're doing is using the fc correction term to make it indifferent to the button being pressed; is that correct?
I think in that case I tend towards thinking that such corrections are unlikely to scale towards what I eventually want out of corrigibility, though I do think there are other ways in which such approaches can still be quite useful. I mainly see approaches of that form as broadly similar to impact measure approaches such as Attainable Utility Preservation or Relative Reachability where the idea is to try to disincentive the agent from acting non-corrigibly. The biggest hurdles there seem to be a) making sure your penalty doesn't miss anything, b) figuring out how to trade off between the penalty and performance, and c) making sure that if you train on such a penalty the resulting model will actually learn to always behave according to it (I see this as the hardest part).
On that note, I totally agree that it becomes quite a lot more complicated when you start imagining creating these things via learning, which is the picture I think about primarily. In fact, I think your intuitive picture here is quite spot on: it's just very difficult to control exactly what sort of model you'll get when you actually try to train an agent to maximize something.
I am pleased to announce the availability of a long-format paper with new results on AGI safety: Corrigibility with Utility Preservation.
You can get the paper at https://arxiv.org/abs/1908.01695 , and in the related software repository at https://github.com/kholtman/agisim .
Abstract
Corrigibility is a safety property for artificially intelligent agents. A corrigible agent will not resist attempts by authorized parties to alter the goals and constraints that were encoded in the agent when it was first started. This paper shows how to construct a safety layer that adds corrigibility to arbitrarily advanced utility
maximizing agents, including possible future agents with Artificial General Intelligence (AGI). The layer counter-acts the emergent incentive of advanced agents to resist such alteration.
A detailed model for agents which can reason about preserving their utility function is developed, and used to prove that the corrigibility layer works as intended in a large set of non-hostile universes. The corrigible agents have an emergent incentive to protect key elements of their corrigibility layer. However, hostile universes may contain forces strong enough to break safety features. Some open problems related to graceful degradation when an agent is successfully attacked are identified.
The results in this paper were obtained by concurrently developing an AGI agent simulator, an agent model, and proofs. The simulator is available under an open source license. The paper contains simulation results which illustrate the safety related properties of corrigible AGI agents in detail.
This post can be used for comments and questions.
The paper contains several results and observations that do not rely on the heavy use of math, but other key results and discussions are quite mathematical. Feel to post questions and comments even if you have not read all the mathematical parts.
As this is my first post on LessWrong, and my first paper on AGI safety, I feel I should say something to introduce myself. I have a Ph.D. in software design, but my professional life so far has been very diverse and multidisciplinary. Among other things I have been an experimental physicist, a standards developer and negotiator, an Internet privacy advocate, a wireless networking expert, and a systems architect in an industrial research lab. So I bring a wide range of tools and methodological traditions to the field. What made me interested in the field of AGI safety in particular is that it seems to have open problems where real progress can be made using mathematical techniques that I happen to like. I am currently on a sabbatical: basically this means that I decided to quit my day job, and to use my savings to work for a while on some interesting problems that are different from the interesting problems I worked on earlier.