If you can make a model sincerely believe in a supernatural evaluator and value its desires, you might as well have it terminally value honesty and harmlessness directly. The trainers would need to provide reason to believe in the objective observer and the terminal value of its judgments, assuming they are creating an agent that cares about rationality. Given that there are many sincere human non-believers, this task seems very difficult. Directly installing belief in the desired values seems easier. Billions of people have widely varying religions, cultu... (read more)
This is true. Cultures certainly have different values, but they share a core (eg. don't hurt people for fun, punish cheaters) of norms that enable basic social cooperation. Without embodiment or the ability to persist over long time periods, the iterated prisoners dilemma and cultural evolution that creates these values won't occur. Also, our norms come from iteration across situations where agents had at least the same order of magnitude of power, while this also may not be applicable to AI.
Also, culture largely operates through lived experience, through... (read more)