(Follow-up to Social drives 1: “Sympathy Reward”, from compassion to dehumanization, but this post is self-contained.)
1. Intro & summary
1.1 Background
In[1] Intro to Brain-Like-AGI Safety (2022), I argued: (1) We should view the brain as having a reinforcement learning (RL) reward function, which says that pain is bad, eating-when-hungry is good, and dozens of other things (sometimes called “innate drives” or “primary rewards”); and (2) Reverse-engineering human social innate drives in particular would be a great idea—not only would it help explain human personality, mental health, morality, and more, but it might also yield useful tools and insights for the technical alignment problem for Artificial General Intelligence.
Then in Neuroscience of human social instincts: a sketch (2024), I worked towards that goal of reverse-engineering human social drives, by proposing what I called the “compassion / spite circuit”, centered around a handful of (hypothesized) interconnected neuron groups in the hypothalamus and brainstem (but also interacting with other brain regions; see that link for gory details). I suggested that this circuit is central to our social instincts, underlying not only compassion and spite, but also (surprisingly[2]) much of status-seeking and norm-following.
Finally, in the previous post, I drew a 2×2 table for when the “compassion / spite circuit” fires:
1.2 This post: “Approval Reward”
The previous post elaborated on “Sympathy Reward”, and now this post will cover “Approval Reward”. (I’m much less interested in the other two.[3]) Approval Reward leads to:
- Pleasure (positive reward) when my friends and idols seem to have positive feelings about me, or about something related to me, or about what I’m doing;
Displeasure (negative reward[4], a.k.a. punishment) when my friends and idols seem to have negative feelings about me, or about something related to me, or about what I’m doing;
…which also generalizes[5] to pleasure from merely imagining those kinds of situations.
1.3 Summary of the rest of the post
- In §2, I’ll list three obvious, straightforward effects of Approval Reward:
- Credit-seeking & blame-avoidance;
- Wanting to feel liked / admired (a.k.a. wanting high social status);
- Norm-following & norm-enforcement.
- In §3, I’ll go through the profound effects of Approval Reward on our moment-to-moment thoughts and consciousness:
- In §3.1, I suggest that Approval Reward deeply impacts our self-image, such that the way we think of ourselves is closely aligned with what would garner social approval. This observation is well-known in evolutionary psychology, but usually explained as a specific evolved trait, or as motivated reasoning. I agree with the observation, but not those proposed mechanisms, and I offer my own alternative account of how this works under the hood.
- In §3.2, I suggest that almost all of us are in the throes of an intense, almost nonstop addiction to Approval Reward. All it takes is a quick, ever-so-subtle turn of one’s attention to how one might look from the outside right now, and you can get an immediate squirt of Approval Reward. (The resulting nice feeling is sometimes called “pride”.) I’d guess that neurotypical people thus help themselves to Approval Reward maybe 10,000 times a day or more, every day of their lives.
- In §4, I go through various other non-obvious effects of Approval Reward, including “status games” with almost-arbitrary goals, and One Weird Trick for recognizing sociopaths.
- In §5, I argue that Approval Reward interacts with Typical Mind Fallacy to create a drive to share one’s own special interests, especially in socially-inattentive nerds and kids.
- §6 discusses the limitations of norm-following motivation and other consequences of Approval Reward. For example, do people want to follow norms even when nobody will ever find out?
In the next post, I’ll pivot back to AI alignment, by arguing that human Approval Reward explains numerous areas where everyday human experience and intuitions make us expect one thing, while “rational agents” / “agent foundations” theory make us expect something quite different. For example, humans are happy to have their goals change as they grow older (contra instrumental convergence), and humans are often helpful and deferential (contra “corrigibility is anti-natural”), etc.
After that post, I’ll (hopefully) finally get to the question of whether we can and should put Sympathy Reward and/or Approval Reward into a future brain-like Artificial General Intelligence.
2. Straightforward effects of Approval Reward
2.1 Credit-seeking and blame-avoidance
As explained in Neuroscience of human social instincts: a sketch §6.1, the central situation where Approval Reward fires in my brain, is a situation where someone else (especially one of my friends or idols) feels a positive or negative feeling as they think about and interact with me.
As an important special case of this kind of situation, the person might have positive or negative feelings about me because I have directly impacted them for the better or worse. Maybe I facilitated (or thwarted) their goals. Maybe I gave them (or stole from them) something they really liked. Etc.
Thus, thanks to Approval Reward, getting credit and praise from your friends and idols seems really good, and getting blame seems really bad. The result of these motivations is credit-seeking and blame-avoidance.
When credit-seeking and blame-avoidance go right, we call it “a desire to help”—to do things that someone else (especially a friend or idol) wants you to do, to further their goals or make their lives better.
When credit-seeking and blame-avoidance go wrong, we find all kinds of terrible decisions and outcomes. We find people feigning ignorance of problems, we find people jockeying for credit, we find people sugar-coating and lying, we find sycophancy, we find bureaucratic blankfaces, and so on.
Here’s another common situation where someone might have positive feelings coincident with thinking about or interacting with me. Maybe those feelings aren’t a specific reaction to something I’m doing for them, or to them, but instead, it’s just that… doggone it, people like me! They’re happy to see me. More precisely, they “like / admire” me—click the link for exactly what I mean by that. Or conversely, they could have negative feelings when interacting with me because they generally dislike / scorn me.
When this situation interacts with Approval Reward, it turns into a motivation: being liked / admired seems good, and being disliked / scorned seems bad. I used to call this the “Drive to feel liked / admired”, but that was before I figured out that it was not its own dedicated innate drive, but rather one part of this larger story.
We recognize the end-result as “status-seeking motivation”—or more precisely, prestige-seeking.[6]
There’s a school of thought which puts prestige-seeking right at the center of all Approval Reward phenomena: prestige is always the goal, and everything else is means-to-an-end for getting prestige. I don’t think that’s technically correct, but I also don’t think it really matters much.
For example, if I’m delivering good news to Zoe, then the prestige-explains-everything theory would say: “Zoe will be happy when I tell her the news, which will lead Zoe to like / admire me marginally more in the future. And that’s good”. Whereas the story I prefer is: “Zoe will be happy when I tell her the news. And that’s good. (And incidentally, Zoe will like / admire me marginally more in the future.)” So it’s a subtle difference concerning the exact contents of my thoughts when the reward signals arrive. I think I’m right, but it’s a minor dispute; both theories predict the same behavior at the end of the day, more-or-less.
2.3 Norm-following (and norm-enforcement)
Finally, a third major effect of Approval Reward is norm-following—the desire to live one’s life in a way that other people (especially friends and idols) would generally think of as good and appropriate. Depending on those “other people”, norm-following could look like being honest, or ruthless, or clean, or disheveled, or doing rigorous analytic philosophy, or vibing about drug trips, or being good at skateboarding, or wearing a halter top, or pretty much anything under the sun.
…I’m using the term “norm-following” here, but it doesn’t quite hit the nail on the head:
For one thing, the theme is getting approval and admiration (and avoiding disapproval), and norm-following is not the only strategy for making that happen. For example, the first person who pioneers a new fashion style is not really ‘norm-following’ per se—it’s a new style, not a preexisting norm—but they still want their friends and idols to react positively to what they’re doing. It’s really part of the same cluster of behaviors as norm-following.
For another thing, people don’t just haphazardly vacuum up any norms they come across; norm-following is targeted towards the opinions of their friends and idols.[7] For example, 13-year-olds tend to think of 40-year-olds as cringe and irrelevant, and to think of 17-year-olds as awesome and important. So 13-year-olds will copy norms from 17-year-old culture, but ignore (or even ostentatiously violate) norms from 40-year-old culture (related discussion: Heritability: Five Battles §2.5.1).
Relatedly, people who see themselves as “rebelling against prevailing norms to be their authentic true selves”, or whatever … are in fact almost definitely norm-following. They’re just getting their norms from elsewhere. Maybe they admire some movie characters? Historical figures? Or maybe the meta-level idea of “being unique and authentic” has itself come to be an admirable norm? Who knows. As Eliezer wrote: “Lonely dissent doesn’t feel like going to school dressed in black. It feels like going to school wearing a clown suit.”
Norm-enforcement is an extension of the same idea as norm-following. If my friends and idols admire honesty, then I (in effect) want them to mentally associate me with honesty. If they observe me being honest, then that’s good! It helps cement that association in their minds. If they see me criticizing and punishing dishonest people, then that’s also good, for the same reason.
3. Approval Reward and self-image
Next, we move into the deeper lore: how Approval Reward impacts self-image and the psyche.
For example, let’s say I have a motivation to read in bed, and I also have a motivation to walk alone outside.
If I’m embedded in a culture where my friends and idols all talk about how great it is to be outdoors alone, and how cringe it is to read, then I’ll probably wind up thinking of the latter motivation as an internalized ego-syntonic desire, and of the former motivation as an externalized ego-dystonic urge (“alas, I skipped my walk, my legs were sore and I’m addicted to this dumb book”).
Conversely, if I’m embedded in a culture where my friends and idols all talk about how great it is to be studious and well-read, and how embarrassing it is to walk all alone, then it will probably be the other way around (“alas, I skipped my reading, my ADHD and restlessness got the better of me”).
This division of “internalized versus externalized” is not just how I think about things, or about how I describe myself to other people, but it’s also strongly predictive of many aspects of my behavior. For example, it will line up with my meta-preferences (what I want to want), and with what I “apply willpower” to do, and with how I would self-modify if I could (e.g. by drugs or therapy).
How does this division work under the hood, and why is it so swayed by social approval?
When evolutionary psychologists talk about this phenomenon,[8] they tend to propose that there is a specific evolved instinct for self-deception; but I reject that way of thinking as neuroscientifically implausible (related discussion here). And when sociologists talk about this same phenomenon,[9] I get the impression that their model is more like: it’s coming from Approval Reward via motivated reasoning / wishful thinking—i.e., it would be nice if I deeply embodied socially-approved personality traits, so I concoct some rationalization for why it’s true! I used to believe that one but don’t anymore.
I still think it’s a consequence of Approval Reward, but I think the mechanism is more subtle than simple motivated reasoning.
Here’s my proposed mechanism:
Ingredient 1: Some of my thoughts are “self-reflective”. For example, if I think of the sensation of pain in my knee, that’s not a self-reflective thought, whereas if I think of myself having an injured knee, especially as seen from the outside, then that is a self-reflective thought. Self-reflective thoughts involve one’s self-model, and thus have a strong association (Intuitive Self-Models §8.3) with “how I seem in someone else’s eyes”.[10] So as a rule, self-reflective thoughts seem good or bad to the extent that what I’m doing seems socially admired. (However, self-reflective thoughts can also seem good or bad for non-social reasons—see Intuitive Self-Models series §2.5.2 for an example.)
Ingredient 2: For reasons explained in the Intuitive Self-Models series (especially §2.2.3 & §8.5.5), my ego-syntonic, internalized desires basically coincide with things that seem good in the context of a self-reflective thought. See those links for why the valence of self-reflective thoughts determines not only what is ego-syntonic vs dystonic, and internalized vs externalized, but also why they determine meta-preferences and self-modification preferences (§8.5.5), and why they determine the targets of “willpower” (§8.5.6).
Putting them together, the conclusion is that my internalized ego-syntonic desires are disproportionately (but not exclusively) determined by what is socially admirable.
But to really understand how social approval gets so entrenched in our self-image and meta-preferences, there’s another part of this story:
3.2 The habit of imagining how one looks in other people’s eyes, 10,000 times a day
If you’re doing something socially admirable, you can eventually get Approval Reward via a friend or idol learning about it (maybe because you directly tell them, or maybe they’ll notice incidentally). But you can immediately get Approval Reward by simply imagining them learning about it.
(Recall from §2.2 of the previous post that the brain has generalization upstream of the reward signals, i.e. inside the reward function itself, which makes it possible to trigger reward signals merely by thinking.)
To be clear, imagining how one would look in another’s eyes is not as rewarding as actually impressing a friend or idol who is physically present—it only has a faint echo of that stronger reward signal. But it still yields some reward signal. And it sure is easy and immediate.
So I think people can get in the habit of imagining how they look in other people’s eyes.
…Well, “habit” is an understatement: I think this is an intense, almost-species-wide, nonstop addiction. All it takes is a quick, ever-so-subtle, turn of one’s attention to how one might look from the outside right now, and bam, immediate Approval Reward.
If we could look inside the brains of a neurotypical person—especially a person who lives and breathes “Simulacrum Level 3”—I wouldn’t be surprised if we’d find literally 10,000 moments a day in which he turns his attention so as to get a drip of immediate Approval Reward. (It can be pretty subtle—they themselves may be unaware.[11]) Day after day, year after year.
That’s part of why I treat Approval Reward as one of the most central keys to understanding human behavior, intuitions, morality, institutions, society, and so on.
3.3 Pride
When we self-administer Approval Reward 10,000 times a day (or whatever), the fruit that we’re tasting is sometimes called pride.
If my friends and idols like baggy jeans, then when I wear baggy jeans myself, I feel a bit of pride. I find it rewarding to (subtly, transiently) imagine how, if my friends and idols saw me now, they’d react positively, because they like baggy jeans.
Likewise, suppose that I see a stranger wearing skinny jeans, and I mock him for dressing like a dork. As I mock him, again I feel pride. Again, I claim that I am (subtly) imagining how, if my friends and idols saw me now, they would react positively to the fact that I’m advocating for a style that they like, and against a style that they dislike. (And in addition to enjoying my friends’ imagined approval right now, I’ll probably share this story with them to enjoy their actual approval later on when I see them.)
4. Miscellaneous other effects of Approval Reward
4.1 “The pity play” as a tell for sociopaths
Here’s psychologist Martha Stout:
After listening for almost twenty-five years to the stories my patients tell me about sociopaths who have invaded and injured their lives, when I am asked, “How can I tell whom not to trust?” the answer I give usually surprises people. The natural expectation is that I will describe some sinister-sounding detail of behavior or snippet of body language or threatening use of language that is the subtle giveaway. …None of those things is reliably present. Rather the best clue is, of all things, the pity play. …Pity from good people is carte blanche… Perhaps the most easily recognized example is the battered wife whose sociopathic husband beats her routinely and then sits at the kitchen table, head in his hands, moaning that he cannot control himself and that he is a poor wretch whom she must find it in her heart to forgive. There are countless other examples, a seemingly endless variety, some even more flagrant than the violent spouse and some almost subliminal.
I would explain this observation as follows:
- “The pity play” is a highly effective way to deceptively manipulate people and get what you want (thanks to the other person’s Sympathy Reward, see previous post).
- However, Approval Reward makes “the pity play” feel intrinsically strongly aversive, so people don’t do that—even people who would readily lie to get what they want in other contexts.
But sociopaths are an exception. They innately experience very weak Approval Reward (or none at all).[12] So for them, “the pity play” is simply an effective strategy with no downsides.
4.2 Semi-arbitrary “status games”
Thanks to Approval Reward, people will (to some extent) enjoy activity X only because other people enjoy activity X, or believe in Principle Y only because other people believe in Principle Y, etc. The end-result is under-determined and unpredictable; you can wind up with people caring deeply about who-knows-what-random-thing, in an (implicit) status game. See e.g. Morality is Scary (@Wei Dai, 2021).
(See also: my take on cultural evolution here.)
4.3 Covering one’s face or looking away when embarrassed?
This might be its own innate reaction, or cultural, but if not, it seems potentially explicable as a side-effect of Approval Reward.
When you’re embarrassed, interactions with other people will feel bad and aversive thanks to Approval Reward. This badness will be scaled by both (1) your phasic physiological arousal in response to thinking about them, and (2) their phasic physiological arousal in response to thinking about you. (The latter multiplies how strongly and attentively they will feel disapproval towards you, and the former multiplies how much their feeling impacts you in turn, via negative Approval Reward.)
Eye contact causes an orienting reaction and thus phasic physiological arousal (see Neuroscience of human social instincts: a sketch and Morton & Johnson 1991). So basically the worst possible situation, when you’re embarrassed, is that you’re making eye contact with them, and they with you. Thus, averting your gaze mitigates the unpleasantness a lot, and covering your face helps even more.
5. Approval Reward × Typical Mind Fallacy = “reward for sharing special interests”
Consider what happens when you mix Approval Reward with typical mind fallacy:
- Approval Reward says that it’s good if Zoe sees an association between me and something she likes;
- Typical Mind Fallacy is a feeling that Zoe likes the same things that I do. (The feeling might be visceral rather than consciously endorsed.)
- One of the results is: “reward for sharing things I really like with other people (even if those other people don’t actually care)”.
I know a nerdy 8-year-old who demonstrates this trait in an unusually extreme way:
HIM: C’mon, please watch me beat the boss of the videogame!
ME: I don’t care if you can beat the boss. I’m glad you’re having fun though.
HIM: Please, please, please, please…
ME: Fine.
(He beats the boss)
ME: OK I watched. I still don’t care. I’m gonna walk away now.
HIM (beaming with joy): Thank you so, so much!
Anyway, the “compassion / spite circuit” responds to the feelings that seem (to us) associated with the other person. And those feelings default to what our own feelings would be in the same situation. For example, if losing a game makes me sad, then my brain will learn an association “losing game” ↔ “sad feeling”. Then when I see Drew lose a game, my brain makes the stable semantic association Drew ↔ “losing game” ↔ “sad feeling”.
That default can be overridden when we notice that the default is making incorrect predictions. Alas, sometimes the default makes bad predictions, but we don’t notice.
Some factors that lead to people having typical mind fallacy (and thus tilting more towards “sharing one’s own interests” rather than “impressing others”) are:
Lacking much intrinsic drive to think about and pay close attention to other people all the time: I think humans have a general “innate drive to think about and interact with other people”. This is separate from the “compassion circuit” or any of the above discussion.[13] This drive explains why, if a neurotypical person is not doing anything in particular at some moment, they’re probably thinking about other people (cf. default mode network). Anyway, this drive is stronger in some people than others—e.g. see Scott Alexander’s discussion of sex differences in people-oriented versus thing-oriented careers. I think a big contributor to social obliviousness is having that drive set at a quite weak level. If paying close attention to other people feels intrinsically unmotivating (or at least, less motivating than other things that one could be thinking about instead), then you’ll do it much less, and you’ll wind up with less accurate models of other people.
Not actually having a “typical mind”: If the other person is in fact acting in a way that is generally very relatable from your perspective, then you’ll be likelier to accurately associate them with their feelings, and if they’re not, then your associations are more likely to be way off.
Lack of Experience: If you compare 4-year-olds with 8-year-olds, and 8-year-olds with 16-year-olds, I think you’ll generally find that they tend to gradually do less “sharing special interests” and more “reacting to what the other person actually cares about” as they gradually learn more and develop better mental models. (They'll also pick up much more quickly when people are uninterested.) For my part, I think I’ve “improved” on this axis even well into adulthood.[14]
The other person being graceful and polite, rather than “pushy” / “ask culture”: In that conversation above, I come across as extraordinarily rude. And if I had been talking to almost anyone else, then yeah, that would have been the correct interpretation of my words. But that’s not what was happening in this conversation! Rather, I happen to know this 8-year-old well, and just as you need to speak at a high volume-level to an acquaintance with poor hearing, you likewise need to speak at a high “pushiness” / “ask-culture”-level to an acquaintance who is socially inattentive. Indeed, in that above (made-up but illustrative) example conversation, as pushy as I was, I still wasn’t pushy enough; he did not absorb my message, even when I spelled it out explicitly.
6. How robust are the effects of Approval Reward?
Coming from an AGI alignment context, I’m especially interested in the question of how the effects of Approval Reward hold up in unusual circumstances, such as extreme power imbalances. Here are a couple related topics from the human world:
6.1 Does Approval Reward make people want to follow norms even when no one will ever find out?
In some cases, Alice can have very high confidence that she can do something Bob wouldn’t approve of, without Bob (or anyone else) ever finding out. Does Approval Reward put a (pro tanto) thumb on the scale here, making Alice not want to do that? (Assume that Alice sees Bob as important, and a friend.)
In the human world, the answer is clearly yes, at least in the short term. It might be a very small thumb on the scale, but probably more than zero, just because there will be some pattern-match, however weak, to previous circumstances where she got caught. That’s enough for a visceral “this is bad” reaction from Approval Reward to trigger.
(Remember, Approval Rewards can be triggered by merely thinking, see §3.2 above.)
If there’s a conscious belief that Bob definitely won’t find out, that won’t immediately unwind this visceral reaction—just as, if there’s a conscious belief that the giant spider can’t possibly get out of the opaque sealed box I’m sitting on, that won’t immediately unwind my anxiety about the situation.
On the other hand, here are three reasons to expect Alice to violate norms when she’s confident that she’ll get away with it:
- If Alice’s secret action would lead to Alice getting more real approval from Bob later on (because he was tricked), then Approval Reward also has its thumb on the opposite side of the scale, making the action seem good. (Or there could be some other reward on the opposite side of the scale.) So Alice might well do the thing anyway. Remember, getting actual approval from actual humans is a much stronger reward (albeit less immediate) than the reward for merely imagining someone seeing you.
- The more obvious and salient it is to Alice that her secret norm-violating action could never possibly be found out by anyone else, the weaker will be her aversion to taking that action, in general.
- In the longer term, if Alice keeps getting away with secretly taking norm-violating actions, and if they never result in any actual humans actually being upset at her, then I think her negative visceral reactions to those plans would gradually fade away, and she would feel better and better about taking these kinds of actions. Visceral reactions are sometimes slow to update (see Perils of under- vs over-sculpting AGI desires §6.2), but they do usually update sooner or later.
- Maybe unpleasant personal habits like picking-your-nose or scratching-your-butt are examples here. The people who do these things, get embarrassed in public a bunch of times, and also get away with it in private a bunch of times, and there’s just so much training signal in both conditions that it can “over-sculpt” our preferences into “don’t engage in those habits unless you won’t get caught”.
6.2 Ways that negative Approval Reward might not trigger
Just as in §4.1.1 of the previous post, there are a number of circumstances in which someone may react negatively to me, but not trigger negative Approval Reward as normal. In brief, these include (1) Not paying attention to the other person; (2) Paying attention, but in a way that avoids triggering the “thinking of a conspecific” flag; (3) Seeing the other person as an enemy; (4) Seeing the other person as unimportant; (5) Feeling like the other person is reacting positively to me, even when they’re actually not.
And just as in §4.1.2 of the previous post, if people react negatively to me, that induces a (pro tanto) motivation to make one or more of those five things happen, such that I need not suffer the negative Approval Reward.
For example, if my friends disapprove of my interests, then I’ll find it unpleasant to talk to them about that, so I probably won’t. Maybe I’ll even fall out of touch with them altogether, and make new friends instead, who think my interests are cool.
6.3 What’s with powerful people surrounding themselves with flunkies?
I think there’s two dynamics at play.
First, powerful people can choose who to surround themselves with. And their Approval Reward says: they should surround themselves with flunkies. That feels nice for them.
Second, the underlings have their own Approval Reward too, and that can help turn the underlings into flunkies, even if they weren’t initially. Remember, Approval Reward makes people care especially hard about approval from important people.
For example, I think of myself as being quite high in frankness and independent-mindedness. But the few times I’ve talked to famous people I admire, man, I definitely feel an unusually strong pull towards wanting to make them immediately happy and impressed with me.
7. Conclusion
Hopefully that’s a good first-pass tour of Approval Reward, and its profound consequences on our psyche and society, for better and worse. Happy to chat more in the comments!
This and the previous post have (hopefully) given us a deeper and richer understanding of some key human prosocial instincts. In upcoming posts, I’ll start trying to revisit the technical AGI alignment problem in light of that understanding. How does our own human Approval Reward impact and distort our thinking about AGI alignment? And conversely, should we install something like (or inspired by) the “compassion / spite circuit” in a brain-like AGI? More to come!
Thanks Seth Herd, Linda Linsefors, Charlie Steiner, Simon Skade, Tassilo Neubauer, and Justis Mills for critical comments on earlier drafts.
- ^
The next two paragraphs are copied from the previous post.
- ^
It’s very elegant that so many human social phenomena, from compassion, to blame-avoidance, to norm-following, and more, seem to be explained in a unified way by the activity of a single hypothalamus circuit. It’s very elegant—but it’s not a priori necessary, nor even particularly expected. There could have equally well been two or three or seven different circuits for the various social behaviors that I attribute to the “compassion / spite circuit”.
There are of course many other social behaviors and drives outside the scope of this post, and I do think there are a bunch of different hypothalamus circuits which underlie them. For example, I think there are separate brain circuits related to each of: the “drive to feel feared”, play, a “drive to think about or interact with other people” (§5 below), loneliness (cf. Liu et al. 2025), love, lust, various moods, and so on.
- ^
I’m mainly interested in how to set up the motivation system of a brain-like AGI, and specifically whether inserting something like the “compassion / spite circuit” would be a good idea. In that context, an easy and obvious thing to do would be: set the innate “friend (+) vs enemy (–) parameter” to be always positive. In that case, “Schadenfreude Reward” and “Provocation Reward” would never come up.
Is doing that a good idea for Safe & Beneficial AGI? Maybe! As an example, I’m not worried about a spite-free AGI failing to fight bullies. Bullies can be fought based on means-end reasoning plus sympathy for the victims. Spite towards the bully is unnecessary.
A thornier issue is that, if we leave any major human emotion out of an AGI altogether, then it potentially messes up the “transient empathetic simulation” aspect of how the “compassion / spite circuit” works. I’m currently thinking that we might need spite etc. in our AGIs, alas, but set to very low levels, and/or maybe only temporarily during training. Or maybe there’s some workaround. Anyway, that’s all out of scope for this post.
- ^
In this post I’m using reinforcement learning terminology as used in AI, not psychology. So “reward” is a scalar which can have either sign, with ”positive reward” being good and “negative reward” being bad. (Psychologists, especially in the context of operant conditioning, use “negative reward” to mean something quite different—namely, relief when an unpleasant thing doesn’t happen.)
- ^
The generalization here is upstream of the reward signals—see §2.2 of the previous post.
- ^
See “dual strategies theory” for the distinction between prestige status and dominance status. (My own take on that is in Social status part 2/2.) Dominance-status is off-topic for this post, since I think it’s unrelated to the “compassion / spite circuit”; see “drive to feel feared” in Neuroscience of human social instincts: a sketch §7.1.
- ^
Recall from Neuroscience of human social instincts: a sketch that the “compassion / spite circuit” is sensitive to (1) the innate “friend (+) vs enemy (–) parameter” and (2) phasic physiological arousal, which tracks the importance / stakes of an interaction. When these are both set to high and positive, then the circuit fires at maximum power, making us maximally motivated by this particular person’s welfare, approval, etc. My term “friends and idols” is a shorthand for that dependency.
- ^
- ^
- ^
As an example, here’s an excerpt from Impro (Johnstone, 1979):
I see the ‘personality’ as a public-relations department for the real mind, which remains unknown. My personality always seems to be functioning, at some level, in terms of what other people think. If I am alone in a room and someone knocks on the door, then I ‘come back to myself’. I do this in order to check up that my social image is presentable: are my flies done up? Is my social face properly assembled? If someone enters, and I decide that I don’t have to guard myself, then I can get ‘lost in the conversation’. Normal consciousness is related to transactions, real or imagined, with other people. That’s how I experience it, and I note widespread reports of people in isolation, or totally rejected by other people, who experience ‘personality disintegration’.
(There might be some person-to-person variation in how strong is the association between a self-concept and “how I look in other people’s eyes”. Johnstone evidently sees the association as so strong that they’re basically inextricable—it’s impossible to think about one without pulling in the other. It might be a weaker association in some other people. Not sure.)
- ^
It might not feel that way; I think it can be pretty subtle. Random example from my own life: if I publish a blog post, and someone important-to-me compliments it, then I get a strong urge to immediately re-read my own post. I only just now figured out why: I think that, as I re-read each paragraph, I am imagining this other person reading that same paragraph and liking it, and that thought gives me a good feeling from Approval Reward. …I now think that’s what’s going on, but it’s pretty subtle—the other person is only present at the back of my mind as I re-read, rather than central and explicit. That’s why it took me until now to understand it.
I like that example because re-reading my own posts is such a dumb waste of time that it calls out for an explanation. But I think it’s the tip of an iceberg. Like, when I’m writing, and I come up with a good idea, or even a nice turn of phrase, I find it rewarding. Am I slightly imagining future readers being impressed? Now that I think about it, probably yes! But again, it’s a very subtle kind of imagining, and slipped my notice until now.
- ^
To be explicit, here’s a simple theory that I like: (1) the root cause of sociopathy is low physiological arousal (see some references here); (2) phasic physiological arousal is a multiplicative factor on the strength of the “compassion / spite circuit” (as discussed in Neuroscience of human social instincts: a sketch); (3) Therefore, all the drives and rewards discussed in this and the previous post manifest very weakly in sociopaths. Meanwhile, most other innate drives (e.g. hunger) do not get multiplied by physiological arousal, so sociopaths experience them at normal strength.
- ^
In terms of Neuroscience of human social instincts: a sketch, I think a simple and plausible way to build an “innate drive to think about and interact with other people” is to have an excitatory connection from the “thinking of a conspecific” flag to pleasure, and/or an inhibitory connection to displeasure, all within the hypothalamus and brainstem (“Steering Subsystem”).
- ^
I put “improved” in scare quotes, because there’s nothing necessarily inherently wrong with an intrinsic motivation to share one’s special interest. For example, I myself seem to have a visceral feeling that other people get as excited as I do by every little detail of my excruciatingly in-the-weeds technical blog posts. Is that visceral feeling reflective of reality? Probably not! But it’s a feeling that enhances my motivation to get the details clear and correct. For me, it’s practically a job requirement!