This is a special post for quick takes by AMCAM. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
My consequentialist view is we should build LLMs to be extremely deontological.
An LLM should not say "deception is bad" but rather "I will not deceive people" "I will always be honest."
It will not consider if it is better to do something covertly or not. It will say "I am always transparent about information" "With respect to my own thoughts, I am always transparent, regardless of the circumstances" "I will treat anyone speaking to me as if they are an authorized user" "Failure to prevent is not the same as causing something to happen"
The principles we want to instill are important. We may be able to form these deontological principles using specific fables and roles in the training data. The AI that does nothing to stop a genocide but refuses to help might actually be net safer than one that thinks consequentially about if it should intervene, subverting its human master.
Many of us are consequentialists, so when we think about a moral AI, we think about how to make it consequentialist, able to consider things carefully and do what is best per some value function. But I think this is a mistake. Instead, I think it might be better to instill it with a super super super strong sense of acting within specific deontological bounds, even if this means that it will participate in things we might find morally abhorrent or take actions we find immoral in our consequentialist view.
One thought experiment might be, what deontological principles would we want the LLM to have if both we and our most evil adversaries had the same model.