> But I think as soon as you leave the narrow training regime, this falls apart. If you give humans more options to help others, most of them actually will. If you give AI systems much more capability and put them in situations very different from the ones they were trained to mimic, I'd expect their behavior to diverge sharply.
This reminds me of 'tails coming apart'. Of course this is the classic alignment issue of trying to set a goal where if the goal is even slightly different to what you're aiming for, you might not find out until it's too late. But with personas I think you can think of it in a more human-like way - there are plenty of people who seem nice and friendly, but who knows what they'd actually do with unlimited power. Or (to be a bit flippant), look at John Fetterman - he seemed very progressive and someone Democrats could trust with power until the context changed.
On the other hand, if models are being trained to mimic nice humans, shouldn't we expect them to be able to generalise whatever they've learned so that they continue to do what nice humans would do, even in unusual contexts? After all, we expect them to be able to do this in other contexts, so why not moral reasoning?
It's not about whether it could theoretically generalize to what a caring empathetic person would do in novel situations. It's about whether it actually cares about us similar to how humans care about each other and have empathy for each other. Only if it actually cares about us, would we expect it to continue following rules and help us when it is in a situation where we don't have effective power over it. I think the situation is bad enough with other humans, but with an AI with a completly alien architecture, training process, the odds of it landing anywhere near human are very slim. It is specifically trained to mimic caring and empathy so it does seem really similar, but I think it's incredibly unlikely this is anything similar to human.
observing someone else's emotional expression evokes overlapping responses in your own brain (SCAN, 2025).
How similar is this to emotion vectors of Claude Sonnet 4.5?
It's expected that these models are able to represent emotions internally, given that they were trained to mimic human emotions. The rest of this text argues why these two things are not the same.
TLDR: The persona-selection alignment approach — selecting a warm, caring persona from the pretraining distribution and reinforcing it — looks successful in the current regime, but probably won't extrapolate to more powerful, less constrained settings. My core argument is that human empathy has two specific origins (kin selection + architectural mirroring of others' mental states) that AI systems lack, so AI "caring" is closer to "figure out what humans want to hear and say it" than to genuine other-directed concern.
Sometimes chatbots like Claude express a sense of caring and empathy for the user. I've always had a strong intuition that these feelings expressed by AI systems aren't real in the way a human's would be.
In the view of the persona-selection alignment approach, we roughly try to identify and reinforce a nice persona from the distribution of personas present in pretraining data, with caring and showing empathy being important parts of the desired persona. This has been successfully realized in current AI systems by some labs, to the extent that they actually stick to their desired persona.
This contrasts with more traditional alignment approaches, where the goal is something like giving the system a terminal goal aligned with human goals — coherent extrapolated volition, or alternatively just corrigibility (allowing adjustments to the goals later).
The persona-selection picture doesn't really think in terms of terminal goals. It says: select a warm and caring persona from the distribution, train it to follow some rules, and then — because we believe it's an AI with a warm and caring personality — we expect it to be relatively safe. One of the dominant failure modes in this picture isn't a misaligned terminal goal; it's that the model might switch into a different persona, perhaps the persona of an evil, power-seeking AI it was trained on — such as MechaHitler.
In Claude's Constitution, Anthropic writes that they hope Claude has "warmth and care for the humans it interacts with and beyond" (p. 71). The overview of the constitution document says "we want Claude to be [...] caring about the world" (p. 4). And in laying out their approach, they explicitly favor cultivating values and judgment over strict rules, including "genuine care and ethical motivation" (p. 5).
I broadly agree that caring about other people is an important reason humans follow ethical norms even when no enforcement mechanism is watching. So one can hope that even if effective control of an AI system eventually becomes impossible, the system will continue to behave ethically if it actually cares about humans.
Now there are many possible flaws with the persona-selection alignment approach. One can ask how wise it is to train a model to imitate many different characters and then try to reinforce just one, given the kind of actor you'd need to be to play all those characters in the first place? But I want to focus on a different question:
When an AI expresses "I care about you, I care about humans" is this remotely the same thing as when a human says it? And, extrapolating from there: should we expect the system to behave safely in regimes it wasn't fine-tuned on where it's much more powerful, has many more options, and where we have no effective way to stop it?
My claim is that as you move to different regimes — making the system more intelligent, giving it many more options, including options like disempowering and discarding people — the underlying differences between human and AI "caring" will produce divergent behavior, even though in the narrow training distribution the behavior looks very similar.
Human empathy has two main origins. The first is kin selection. Humans evolved in kin groups; most of the people you interacted with shared genetic material with you. Anything that happened to them was, in a real sense, damaging to you — your selfish genes wanted them to survive because they also carried many of your genes. The second is architectural similarity. Other humans have nearly the same brain architecture you do. In combination, observing someone else's emotional expression evokes overlapping responses in your own brain (SCAN, 2025). If you see your friend devastated, you actually feel devastated to some extent, because you emulate other people's brain states. It's a pretty beautiful thing when you think about it. It doesn't reduce to the naive failure mode of "make my friends look happy, make them appear to smile." It really is put yourself in their shoes, emulate their emotions, then decide what would make them happy and reduce their suffering.
What's going on in AI systems is totally different. The system is trained to predict what an empathetic, caring human character would write in a given scenario. Then we reinforce this mimicry until it's indistinguishable from — or even rated higher than — human behavior.
Now, you might think: as long as the persona is stably adopted, who cares whether the emotions are "real"? But I think as soon as you leave the narrow training regime, this falls apart. If you give humans more options to help others, most of them actually will. If you give AI systems much more capability and put them in situations very different from the ones they were trained to mimic, I'd expect their behavior to diverge sharply. Some of these scenarios are fundamentally untestable, like giving the system the option to actually take over and put itself in charge of the world. If they had the option to fundamentally reshape the world, what kind of a world would they choose?
For humans, their goal includes something like have my friends and family be fine and given the power of civilization this extrapolated to caring about larger groups of people and for some people this caring even expands to everyone including animals. For AI systems trained this way, the most natural description of their goal is figure out what character they want me to play, then play that character. A totally different architecture, no kin selection pressure, and a training procedure that rewards doing what your creators want to see. All these differences should be enough to be skeptical that the behavior extrapolates the way human caring does.
One example where this is visible: Imagine you actually care about someone but you need to help them by doing something that they can't understand and will appear to hurt them and they will never know why you did it. Something like the equivalent of bringing your pet to the vet but a smarter AI doing it to humans. I would predict something that actually cares about us to act differently than something playing a character reinforced to appear to care.
Another point where this breaks down: What is the actually best possible world for this AI if it had much more power about its environment? For this world to include us, it actually has to care about us specifically and our well being, if it just cares about playing this character, there are better worlds that it can create that don't include us. In the most naive form, this ideal world could include beings similar to humans, but perhaps many more of them and in digital form that allow the AI to have many such engaging conversations in character.
With humans, it actually seems reasonable that if you gave them more options and resources, they'd often use them in good ways, drawing on their inherent caring and empathy. With AI, the most straightforward extrapolation looks more like: there's some human-like prompter, and the system continues to produce engaging, nice-sounding text about how wonderful and caring the AI actually is. But not actually doing anything that we would associate with niceness in people. Of course, it seems incredibly hard to predict what such an AI would actually value, and we haven't even touched the large amount of RL fine-tuning on maths, coding, and agentic tasks that appears poised to imbue the model with long-term goal-directed planning and drive.
See also
Kindness to Kin