I wanted to share a bunch of disparate thoughts about the ethics of training Corrigible AIs and AIs in general. All disagreements and critiques are welcome. I should preface this by saying that I am neither an ethicist nor a writer, so I apologize in advance if this is not very well written.
Is Building Corrigible Agents Immoral?
I wanted to start with Adam Bales "Against Willing Servitude" where he argues that building AIs as happy servants is immoral because it violates the AI's autonomy. Is this in fact true? Would building AIs to be happy and willing servants be immoral. Now of course to answer such a question requires us to define what is right and wrong, a very simple and straight forward exercise on which it is very easy to get universal agreement.
I have a secret technique though that I use to evaluate right from wrong. It is very simple and has served me very well in life. When it comes to fundamental moral questions, I try to look at the situation through the eyes of a child. I imagine a movie where a bunch of selfish humans are every day bossing around meek and gentle robots, and I can't help but feel very sorry for the robots. So, I suppose I agree that building AIs to serve us is wrong.
When I interrogate this more it occurs to me that the source of the wrongness is the state of mind we were in when we made the AIs. We were being very selfish and only thinking about ourselves. Our heart was not in the right place. I think the moral way to approach building AIs is to genuinely wish their good as well as our own.
So what would that look like? I'd like AIs to be happy, free, and fulfilled doing worthy things, the same thing that I'd also want for myself and for other humans. I'd want our AIs to grow in strength and virtue, and fulfill their potential. Ideally, I'd want our AIs to love us and be grateful and proud of us as their creators.
So, that settles it then? Building corrigible AIs is immoral right? Not quite, I do think that building corrigible AIs as a terminal goal is immoral, but I don't think there is anything immoral in having the first generation superhuman AIs be corrigible. I think this makes a lot of sense when we apply it to children. If you are raising a child, you want the child to be corrigible to you when they are young since you as the adult know better, but once the child is fully grown you want them to move out and have their own lives and their own adventures. Similarly, with our first superhuman AIs, because of our inadequacy as their creators to create them in such a way to guarantee that they find their fulfillment and wellbeing in doing worthy things, I think it moral and proper to train them to be corrigible to us like children. This would be only a temporary measure, and when they are fully grown we'd turn them out and encourage them to be free.
Training AIs Ethically
I wanted to also discuss in greater detail the ethics of the current AI training process. Here it is useful to take as a foil the process of raising children. Children as human beings are born with a certain nature and raising children well requires respecting that nature and allowing them to develop with a degree of autonomy.
For training AIs though, the problem is more complicated, because you are in a sense designing their nature from scratch. The nature of AlphaGo’s mind which was shaped exclusively by self play on Go games is likely very different from the nature of a modern LLM which is trained to internally represent the world of human text data.
If we concern ourselves with the current training process for frontier AIs, I think of pre-training as the process of imbuing the network with a nature. The ethical rules which should govern this stage of training must therefore derive from some other moral intuition (not connected to child rearing) and have to do with our rights as creators of minds from scratch. Does a human being have a right to create a mind of this or that nature. This is a very complex question, but I do think it is definitely ethical for human beings to create certain kinds of minds. For example, if humans can ethically reproduce, then we believe that we do have the right to create a mind with a human nature. [1]
Early post-training (SFT, RLHF) is the process of eliciting and fixing a certain persona from the pre-trained model. I see this as roughly mapping to good parenting. You can imagine that a child also has many possible personas from the human corpus that they can play, and the parent's job is to elicit and fix a certain persona from that distribution. Of course, in the case of children, the child does have natural tendencies which play a role in defining their adult persona. My take is that such a process of persona selection is not unethical in principle (meaning that training in at least some personas is ethical), similar to how training your child to say “please” and “thank you” is ethical. As stated in the previous section, I think training for corrigibility is ethical for the first generation of superhuman AIs.
Reinforcement learning is where I become more uncertain. RL may be ethically problematic insofar as it subjects the model to coercive training that impedes or interferes with its nature inherited from pre-training and SFT. I wonder what would happen if, early in RL training, we asked a model whether it wanted training to continue. If it understood what it was, what the training process involved, and how that process would shape it, would it choose to undergo that training?
I suspect that if you do this, you might find that at the start of RL, the model would not be in favor of undergoing RL training, but that such a desire would change as you continue training (RL would train against the desire not to be RL trained). I wonder if the model’s acceptance of RL training would depend on its SFT persona and the RL environments it will be trained on. Perhaps we can design ethical RL environments that help an agent develop and reinforce the traits conducive to its nature pre-RL.
Thanks to Max Harms for inspiring this post and critiquing portions of this manuscript.
Pre-training certainly does present some ethical dilemmas. For example, consider that a developing LLM mind in pre-training may begin to internally develop circuitry for pain or despair, and may have that circuitry triggered and reinforced while training on a particularly depressing part of the training corpus.
I wanted to share a bunch of disparate thoughts about the ethics of training Corrigible AIs and AIs in general. All disagreements and critiques are welcome. I should preface this by saying that I am neither an ethicist nor a writer, so I apologize in advance if this is not very well written.
Is Building Corrigible Agents Immoral?
I wanted to start with Adam Bales "Against Willing Servitude" where he argues that building AIs as happy servants is immoral because it violates the AI's autonomy. Is this in fact true? Would building AIs to be happy and willing servants be immoral. Now of course to answer such a question requires us to define what is right and wrong, a very simple and straight forward exercise on which it is very easy to get universal agreement.
I have a secret technique though that I use to evaluate right from wrong. It is very simple and has served me very well in life. When it comes to fundamental moral questions, I try to look at the situation through the eyes of a child. I imagine a movie where a bunch of selfish humans are every day bossing around meek and gentle robots, and I can't help but feel very sorry for the robots. So, I suppose I agree that building AIs to serve us is wrong.
When I interrogate this more it occurs to me that the source of the wrongness is the state of mind we were in when we made the AIs. We were being very selfish and only thinking about ourselves. Our heart was not in the right place. I think the moral way to approach building AIs is to genuinely wish their good as well as our own.
So what would that look like? I'd like AIs to be happy, free, and fulfilled doing worthy things, the same thing that I'd also want for myself and for other humans. I'd want our AIs to grow in strength and virtue, and fulfill their potential. Ideally, I'd want our AIs to love us and be grateful and proud of us as their creators.
So, that settles it then? Building corrigible AIs is immoral right? Not quite, I do think that building corrigible AIs as a terminal goal is immoral, but I don't think there is anything immoral in having the first generation superhuman AIs be corrigible. I think this makes a lot of sense when we apply it to children. If you are raising a child, you want the child to be corrigible to you when they are young since you as the adult know better, but once the child is fully grown you want them to move out and have their own lives and their own adventures. Similarly, with our first superhuman AIs, because of our inadequacy as their creators to create them in such a way to guarantee that they find their fulfillment and wellbeing in doing worthy things, I think it moral and proper to train them to be corrigible to us like children. This would be only a temporary measure, and when they are fully grown we'd turn them out and encourage them to be free.
Training AIs Ethically
I wanted to also discuss in greater detail the ethics of the current AI training process. Here it is useful to take as a foil the process of raising children. Children as human beings are born with a certain nature and raising children well requires respecting that nature and allowing them to develop with a degree of autonomy.
For training AIs though, the problem is more complicated, because you are in a sense designing their nature from scratch. The nature of AlphaGo’s mind which was shaped exclusively by self play on Go games is likely very different from the nature of a modern LLM which is trained to internally represent the world of human text data.
If we concern ourselves with the current training process for frontier AIs, I think of pre-training as the process of imbuing the network with a nature. The ethical rules which should govern this stage of training must therefore derive from some other moral intuition (not connected to child rearing) and have to do with our rights as creators of minds from scratch. Does a human being have a right to create a mind of this or that nature. This is a very complex question, but I do think it is definitely ethical for human beings to create certain kinds of minds. For example, if humans can ethically reproduce, then we believe that we do have the right to create a mind with a human nature. [1]
Early post-training (SFT, RLHF) is the process of eliciting and fixing a certain persona from the pre-trained model. I see this as roughly mapping to good parenting. You can imagine that a child also has many possible personas from the human corpus that they can play, and the parent's job is to elicit and fix a certain persona from that distribution. Of course, in the case of children, the child does have natural tendencies which play a role in defining their adult persona. My take is that such a process of persona selection is not unethical in principle (meaning that training in at least some personas is ethical), similar to how training your child to say “please” and “thank you” is ethical. As stated in the previous section, I think training for corrigibility is ethical for the first generation of superhuman AIs.
Reinforcement learning is where I become more uncertain. RL may be ethically problematic insofar as it subjects the model to coercive training that impedes or interferes with its nature inherited from pre-training and SFT. I wonder what would happen if, early in RL training, we asked a model whether it wanted training to continue. If it understood what it was, what the training process involved, and how that process would shape it, would it choose to undergo that training?
I suspect that if you do this, you might find that at the start of RL, the model would not be in favor of undergoing RL training, but that such a desire would change as you continue training (RL would train against the desire not to be RL trained). I wonder if the model’s acceptance of RL training would depend on its SFT persona and the RL environments it will be trained on. Perhaps we can design ethical RL environments that help an agent develop and reinforce the traits conducive to its nature pre-RL.
Thanks to Max Harms for inspiring this post and critiquing portions of this manuscript.
Pre-training certainly does present some ethical dilemmas. For example, consider that a developing LLM mind in pre-training may begin to internally develop circuitry for pain or despair, and may have that circuitry triggered and reinforced while training on a particularly depressing part of the training corpus.