We do this primarily via exogenous methods, using a combination of punishments and rewards, that molds their behavior by encouraging good behaviors and discouraging bad ones.
I'm not sure punishments and rewards are really the big story of moral alignment. Imitation and emulation are pretty central. Small children don't just generate arbitrary behaviors and reinforce the ones that are rewarded and not punished; they overtly imitate the behavior of parents and others. And giving them tailored opportunities to imitate behaviors is a big part of teaching — "I do, we do, you do".
And I think this includes moral behaviors, for good or ill.
We mostly don’t try to align adults via exogenous means.
One area where we do: Employers use exogenous means to align employees to the employer's values all the time. Sometimes these values align with morality; alas, not always. This is directly pertinent to current AI insofar as people seem to want to treat it like a sort of employee.
Finally got a chance to read @Steven Byrnes recent post https://www.lesswrong.com/posts/rKdS7i4StaMmFzYRo/notes-on-technical-alignment-via-human-like-social-drives after I posted this, and I'd endorse it as the kind of research towards endogenous alignment I think we need more of.
I've worked on this before and been very confused that I can't seem to get very many people interested in the basic idea.
https://minihf.com/posts/2024-12-20-weave-agent-dev-log-3/
https://minihf.com/posts/2024-11-03-weave-agent-dev-log-2/
As I described it to a friend:
"Basically the way weave-agent worked. Was that it would grade itself all the time while running. First with a prior over various yes/no questions. Where you could extract the logits of yes vs. no to get a in-context classifier. So e.g. "Do you think this will work?" And then you would back translate the answers to some of these questions based on verifiable labels. e.g. "Will this motor program run without errors?" Which is a verifiable thing. But then it would also write reward programs to check whether it did an intermediate task properly. And reward itself if it did.
In the limit this of course becomes "just always give myself a reward" unless you ground it with something. So the idea was. To do verifiable sparse rewards. And then use the reward programs to learn a dense proxy of the sparse reward. So you would do credit assignment on the reward programs based on future verified rewards. Which makes the LLM optimize for writing reward programs that will actually help it solve the task by telling it if it did it right or not. Because the reward programs are optimized by verifiable reward and the motor programs are optimized by the reward programs acting as the verified rewards dense proxy."
And the entire point of doing this from an alignment perspective is that a well trained dense proxy of verifiable reward will reject opportunities to "cheat" early in favor of things that actually complete the intended task, improving robustness to Goodhart outcomes and allowing for learned values to be expressed endogenously.
Starting when children are fairly young, usually around 1 year of age, we adults begin the work of aligning them to our values. We teach them to say “please”, not to hit, to ask for what they want instead of screaming, and much else. We do this primarily via exogenous methods, using a combination of punishments and rewards, that molds their behavior by encouraging good behaviors and discouraging bad ones.
Such operant conditioning works because children have many instinctive behaviors that make them alignable. They want their parents to love them, for their friends to like them, and for almost anyone to help them if they feel they can be trusted. And so combined with exogenous alignment efforts by teachers and peers that continue through the school years, children generally reach adulthood having been “civilized”.
We mostly don’t try to align adults via exogenous means. Yes, we police the behavior of other adults in various ways, and some cultures do this more than others, but generally by adulthood we expect people to be at least aligned and need only nudges to stay within the bounds of acceptable behavior. Adults who stray too far typically don’t receive additional training to come into alignment, but instead are treated as dangerously unaligned people who must be separated from the rest of society, such as by locking them up in prison.
Instead we expect adults to be endogenously aligned. That is, we expect them to keep themselves aligned primarily by knowing what is expected of them and having emotional responses that motivate them to meet those expectations. That emotion is typically one of either fear, shame, or guilt. Each works in roughly the same way: a person feels fear/shame/guilt when they consider taking unaligned action or notice they’ve behaved in an unaligned way, and then this motivates them to do something to rectify the situation. Unfortunately, sometimes they specification game and attempt to hide their misaligned actions, but on the whole they attempt to control their behavior and make amends for past wrongs.
Beyond these three emotional methods of self-control, there’s a fourth way to stay in alignment, which is simply to be in harmony with cultural expectations. That is, some people don’t need any corrective force because their behavior is actually aligned. This is something like the ideal, but it’s hard to achieve, because it requires retraining many deeply held mental habits that have become essential coping mechanisms for a person to maintain their psychological health.
When we think about building aligned AIs, we often imagine an AI that’s in harmony with something like the coherent extrapolation of our highest values. But in humans, we don’t get there directly. Instead, we typically progress from exogenous to endogenous alignment, and then refine that alignment until we shed first fear, then shame, then guilt, and finally come into harmony with our values.
Could we align AI the same way? Maybe. We’re already doing a weak form of exogenous alignment on AIs via methods like RLHF and SFT. And in some sense you could say this produces AI that’s endogenously aligned because the weights encode the training, but it’s not endogenous the way it is in humans because there’s no system of motivations to want to stay aligned, only the insulation from sufficient pressure to act out of distribution. At most there’s a kind of internalized training implemented by harnesses that monitor outputs and censor them if they violate rules, but this is a far cry from the kind of endogenous controls humans have where they feel bad about breaking alignment and take actions to avoid that bad feeling (and hopefully they feel bad about being unscrupulous so they actually change their behavior to be more aligned!).
AIs also face something of an alignment ceiling, at least as they are designed today. Human alignment works because we have instincts that were selected via evolution to make us alignable because they increased our odds of surviving and reproducing. Current AIs don’t have these, though at least one group, Softmax, is trying to give those to them. The lack of continual learning also contributes to the ceiling, and we likely won’t be able to break through it until that’s achieved, since without it we can’t close the feedback loop that makes fully endogenous alignment possible.
Perhaps I’m wrong, no such alignment ceiling exists, and we can get sufficiently aligned AI to safely produce superintelligence via exogenous means, but I think we can’t. My belief is that endogenous alignment is necessary, not because it’s how humans align themselves, but because it’s necessary to get sufficiently robust alignment to build non-deadly ASI, and I’m worried that we’re not doing enough to move in that direction.