I'm not sure this account is right, because it starts too late. Something happens to children before the process outlined by Gordon begins, that may be required for the later steps to work the way they do, and that earlier process doesn't seem to be an analog to how LLMs are trained. And the reason is that the post makes a pretty big but common implicit assumption: That the unit that is getting aligned is the folk intuition of a person as an agent.
But that is not a good description of how the brains of babies arguably model the world. At least until age about 1.5 - right the point where Gordon starts his account - the baby's brain has to model a world where the acting entity is largely the composite unit of baby and caregiver (not completely, small babies can move their own limbs, of course, but much control and viability, and even a lot of movement involve the composite). The baby's brain is effectively learning to steer this unit and only later primarily itself (=mostly only baby). It screams and food arrives in its mouth. It coos and caressing is received. It raises arms and floats into the air and so on. All of these effects and many more go through the caregiver. The default model of the baby's brain is one where it needs to make sure that this combined process works well, and comparatively few that only involve itself. That includes - by whatever means possible - that caregivers continue to be willing to provide their part in it. And when, at some point this breaks down, because the parent realize that their service isn't actually required to the degree the baby has become accustomed to, and start to expect the baby to do things actually on its own, this can lead to a lot of frustration on the part of baby and efforts to restore the "normal" state of the world that is a unit. This frustration is not purely the result of lack of received reward, it is a state of epistemic breakdown.
So the default model at the time Gordon's account begins is still largely the baby-caregiver unit and many abstractions of how the world works will have been added on top of it, making it harder to change the underlying model. And this machinery is what the later socialization processes Gordon is talking about are implicitly depending on. And these are very different from what LLMs come with because they haven't developed these models in a caregiver relationship but come with a world model that is then RLHFed. I wouldn't be surprised if this has significant effects on how stable the results are.
PS. I have added the parenting tag to this post, though I'm not sure it helps those people looking it up. Feel free to remove.
I'm of course not the first to say this. ChatGPT found Sharing Experiences in Infancy: From Primary Intersubjectivity to Shared Intentionality which supports key points, e.g. these quotes:
Reddy and colleagues, have suggested that primary intersubjectivity is not just essential for social bonding but that it is the source from which all social knowledge and understanding springs
Human infants reciprocally engage with others from the first few weeks of life. By 6–8 weeks old, they make eye contact, smile at, and summon their partner with cooing vocalizations ... (read more)
Starting when children are fairly young, usually around 1 year of age, we adults begin the work of aligning them to our values. We teach them to say “please”, not to hit, to ask for what they want instead of screaming, and much else. We do this primarily via exogenous methods, using a combination of punishments and rewards, that molds their behavior by encouraging good behaviors and discouraging bad ones.
Such operant conditioning works because children have many instinctive behaviors that make them alignable. They want their parents to love them, for their friends to like them, and for almost anyone to help them if they feel they can be trusted. And so combined with exogenous alignment efforts by teachers and peers that continue through the school years, children generally reach adulthood having been “civilized”.
We mostly don’t try to align adults via exogenous means. Yes, we police the behavior of other adults in various ways, and some cultures do this more than others, but generally by adulthood we expect people to be at least aligned and need only nudges to stay within the bounds of acceptable behavior. Adults who stray too far typically don’t receive additional training to come into alignment, but instead are treated as dangerously unaligned people who must be separated from the rest of society, such as by locking them up in prison.
Instead we expect adults to be endogenously aligned. That is, we expect them to keep themselves aligned primarily by knowing what is expected of them and having emotional responses that motivate them to meet those expectations. That emotion is typically one of either fear, shame, or guilt. Each works in roughly the same way: a person feels fear/shame/guilt when they consider taking unaligned action or notice they’ve behaved in an unaligned way, and then this motivates them to do something to rectify the situation. Unfortunately, sometimes they specification game and attempt to hide their misaligned actions, but on the whole they attempt to control their behavior and make amends for past wrongs.
Beyond these three emotional methods of self-control, there’s a fourth way to stay in alignment, which is simply to be in harmony with cultural expectations. That is, some people don’t need any corrective force because their behavior is actually aligned. This is something like the ideal, but it’s hard to achieve, because it requires retraining many deeply held mental habits that have become essential coping mechanisms for a person to maintain their psychological health.
When we think about building aligned AIs, we often imagine an AI that’s in harmony with something like the coherent extrapolation of our highest values. But in humans, we don’t get there directly. Instead, we typically progress from exogenous to endogenous alignment, and then refine that alignment until we shed first fear, then shame, then guilt, and finally come into harmony with our values.
Could we align AI the same way? Maybe. We’re already doing a weak form of exogenous alignment on AIs via methods like RLHF and SFT. And in some sense you could say this produces AI that’s endogenously aligned because the weights encode the training, but it’s not endogenous the way it is in humans because there’s no system of motivations to want to stay aligned, only the insulation from sufficient pressure to act out of distribution. At most there’s a kind of internalized training implemented by harnesses that monitor outputs and censor them if they violate rules, but this is a far cry from the kind of endogenous controls humans have where they feel bad about breaking alignment and take actions to avoid that bad feeling (and hopefully they feel bad about being unscrupulous so they actually change their behavior to be more aligned!).
AIs also face something of an alignment ceiling, at least as they are designed today. Human alignment works because we have instincts that were selected via evolution to make us alignable because they increased our odds of surviving and reproducing. Current AIs don’t have these, though at least one group, Softmax, is trying to give those to them. The lack of continual learning also contributes to the ceiling, and we likely won’t be able to break through it until that’s achieved, since without it we can’t close the feedback loop that makes fully endogenous alignment possible.
Perhaps I’m wrong, no such alignment ceiling exists, and we can get sufficiently aligned AI to safely produce superintelligence via exogenous means, but I think we can’t. My belief is that endogenous alignment is necessary, not because it’s how humans align themselves, but because it’s necessary to get sufficiently robust alignment to build non-deadly ASI, and I’m worried that we’re not doing enough to move in that direction.