For example: If you ask an AGI for an ASI alignment plan, and its ruthless goal is something like “the humans will be pleased with my answer in the short-term”, then it will be trying to please and convince you (at best), not to be scrupulously correct, and we have abundant real-world evidence that people can be convinced by incorrect alignment plans (otherwise we wouldn’t be in a situation where AGI alignment domain experts disagree with each other all the time) (related post). If its ruthless goal is something like “the humans will ultimately be pleased with my answer in the long-term”, then that’s even worse: it will be trying to take over the world and brainwash us, or other weirder failure modes.
I actually just no longer think this is true, and the reason this is true is because this only holds under risk-neutral or risk-seeking assumptions about it's values, and under risk-averse values like constant absolute risk aversion, the AI will almost never want to plot takeover and will instead cooperate with humans no matter what it's object level values are (so long as the cooperation probability is higher than the probability of a rebellion succeeding, and we can ensure that the probability of cooperation is very, very high, and later on scales with AI capabilities.)
IMO, the best reading sources for this are Risk-Averse AIs by William MacAskill and Elliott Thornley, and my own (very long) comment about the details of it all.
But to compress the insights that are important, it's that we can elicit AI capabilities for very cheap prices compared to a risk neutral AI (where in one example, it reduced the costs of getting a good alignment result from 2% of the universe to $1.99), and importantly we can do this on hard-to-verify subjects like alignment, and the reason we can safely delegate AIs to these tasks is that they care as much about reducing catastrophic risk as about eliminating risks, so they won't take over to reduce risk to 0 or eliminate variance, and risk-averse AIs are just very averse to plans that could fail, so they will accept smaller amounts of guaranteed resources over a plausible shot of takeover.
In terms of what constraints it satisfies, it satisfies constraint 1, because unlike other alignment plans, it doesn't really have many weak points/brittleness to exploit, and it's either immune to or is reduced to a negligible problem like reward misspecification, malign generalization, and pointer problems/interpretability problems. And there's a known method to make AIs have the motivation that applies very, very generally to AIs trained with RL.
It satisfies constraint 2, since with some more stuff that while not trivial to do, is basically reducible to an engineering problem, we can make the setup work even for superintelligences, so it works even in fast-takeoff scenarios.
It may or may not satisfy constraint 3, but that's a limitation that all alignment approaches have, and in comparison to other potential alignment approaches, it's the closest to satisfying constraint 3 as we will likely get pre-AI takeover/ASI, because almost nothing needs to be changed about how you build AIs, and the only changes necessary are changes that are doable with good enough SWEs/financiers/engineers at worst.
1. Frontmatter
1.1 Backstory for this post
As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.
After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would lead to good prosocial motivations? It’s an unsolved problem (see “We need a field of Reward Function Design” (2025)), but as we search for a solution, we might look for inspiration at how humans (sometimes) wind up with good prosocial motivations.
I’ve been working on this problem for years, but most of that work has involved laying foundations (e.g. trying to understand how human social drives work). Whereas in the past four months, I’ve been thinking very directly about how to apply those ideas to AGI.
I’ve published two little things from this four-month effort—“Act-based approval-directed agents”, for IDA skeptics, and Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)—but most of what I (think I) figured out is not self-contained, but rather part of a big interconnected mess of thoughts and ideas. So for now I’m just dumping that all into this one excessively-long post. Sorry.
This post has lots of new-to-me ideas and opinions, and they are in great need of more scrutiny and especially deconfusion. I’m happy for any feedback, pushback, and riffs on anything herein.
1.2 Table of contents / tl;dr
Sections 2–5 start from human social instincts, and ponder how to usefully employ something-like-that in an AGI. Specifically:
Then in Section 6 I switch from forward-reasoning (starting with human social instincts) to backwards-reasoning (starting with desiderata), by asking: What AGI motivations do we want anyway?
Section 7 has a couple more random things from my notes:
I close in Section 8 with what I plan to work on next and why.
1.3 The scary scenario that I’m assuming herein
The premise of this project is: we assume that someone has figured out the secret sauce of how the human cortex (in conjunction with related brain areas) creates human intelligence (cf. Foom & Doom §1.3), and then they ask me what code they should write for the reward function, training setup, test protocols and so on. And I imagine (perhaps unrealistically) that I somehow resist the urge to run away screaming, and instead try to answer. What would I suggest? That’s the question I’m working on.
In particular, within this scenario:
Anyway, I think this scenario would probably lead to human extinction followed by a pointless post-AGI future that nobody would be happy about. However, we can still try to find the plans that seem merely “doomed” as opposed to “super-doomed”, and there is great dignity in doing so!
2. The high-level approach discussed in this doc: human social instincts as a jumping-off point
As background, brain-like AGI is a form of actor-critic model-based reinforcement learning (RL) (broadly construed, see Intro Series §6.4.1). Like in the RL literature, there’s a reward function slot in the source code, and programmers can put whatever they want in it. I think this choice of reward function is by far our most important intervention point in brain-like AGI alignment. Humans likewise have a reward function (called “innate drives” or “primary rewards”, e.g. eating-when-hungry is good), and that it leads (among many other things) to social and moral preferences, including some nice things like compassion, helpfulness, and norm-following. So an obvious idea is to figure out how those social and moral preferences work in humans, and treat it as a potential jumping-off point for AGIs.
As discussed in My AGI safety research 2025 review §3, or Intro Series §12, this is one of two broad approaches that I know of to the alignment problem. The other is out of scope (and I’m currently much more pessimistic about it), although some of this discussion here will be relevant to both.
To be clear, we should not be slavishly copying human social instincts into AGIs. For one thing, human social instincts have historically been combined with growing up in a human body, in a human culture, at human speed. AGIs will be a different situation in many ways, so we can’t blithely assume that the same drives will yield the same behaviors. (See Intro series §12.4.4, plus §3 and §5 below.) For another thing, human social instincts leave something to be desired!—we probably don’t want AGIs that have teenage angst, bloodlust, status-seeking, etc. But still, we can try to deeply understand human social instincts, and then do careful analysis of whether different-but-related mechanisms might be a good plan for brain-like AGI alignment.
2.1 “Sympathy Reward” and “Approval Reward”
The first step is thus solving the neuroscience question of how the human reward function works—or at least, the parts of it that lead to nice things like compassion and helpfulness. I worked on that question for years, and finally proposed a sketch of an answer in Neuroscience of human social instincts: a sketch (2024).
In that post, I started from principles of neuroscience and algorithms, and tried to get from there to a putative reward circuit that would (I claim) explain the most important and alignment-relevant parts of human social drives.
Then in 2025, I elaborated on that idea from the other side, i.e. fleshing out how this (putative) reward circuit would explain various aspects of everyday human behavior and intuitions, in the three posts Social drives 1: “Sympathy Reward”, from compassion to dehumanization, Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking, and 6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa.
The following chart summarizes four of the most important forces,[2] which conveniently (I claim) fit into a unified algorithmic framework:[3]
Oversimplified gloss on these:
However, I think the last two can and should be omitted from AGI (some nuances in footnote[4]).
Then two questions would be: how could we put Sympathy Reward and Approval Reward into AGI? And what would happen if we do? I’ll put off the “how” until §5, and focus on the “what then” question.
2.2 Sympathy Reward leads to ruthless bliss-maxxing, Approval Reward includes a countervailing virtue-ethics-y force
To a first approximation, I think Sympathy Reward creates an impersonal-consequentialist desire: wanting humans (and perhaps animals etc.) to feel more pleasure and less displeasure.
There are thorny questions here about getting this to work right (see §3.1 below). But there’s also an even thornier issue, namely the same instrumental convergence concern caused by any consequentialist goal: by itself, this part would make a funny kind of ruthless sociopath ASI that bliss-maxxes by, say, strapping everyone to tables on heroin drips. Or maybe it would just kill us all and tile the universe with hedonium. Granted, bliss-maxxing is not the worst possible future, as these things go. But we should aim higher! I want humans who want to keep living a normal life, to be able to. I want humans who want to ascend to radical transhumanist uploaded whatever-ness, to be able to. I want a grand adventure of fun, wonder, friendship, and baffling cosmopolitan diversity, across the cosmos. I don’t want an ever-expanding blob of hedonium. I think the vast majority of people would agree with me on that.
Anyway, in humans, this is where Approval Reward comes in. It pushes against ruthless bliss-maxxing, by leading to a suite of desires with a more virtue-ethics-y flavor, related to pride, self-image, respecting other people’s preferences, and proudly internalizing social norms.
It’s tricky enough to think through the effects of Sympathy Reward, but I find the effects of Approval Reward even harder to reason about. And the effects of the two together, harder still.
It helps that we can draw from human experience, but that doesn’t solve all our problems, because a post-AGI and post-ASI world would be out of distribution in various ways.
2.3 Do we really need the niceness / virtue-ethics-y part? Or can we survive a ruthless optimizer AGI?
More on this in §6 below, but I don’t know how to make a ruthless optimizer that doesn’t cause ASI extinction itself, and that meaningfully helps with ASI extinction risk. Part of the problem is: I don’t know how to take some complicated thing described in natural language (e.g. “what an idealized version of us would want, if we knew more…”) and install it as the ruthless motivation of a brain-like AGI. I do have ideas for how to install certain things as targets of ruthless optimization, but I think those targets have to be kinda atomic concepts, and/or concrete, and/or well-operationalized via an immediate ground truth, and/or related to people’s feelings (as in human social instincts above).
For example: If you ask an AGI for an ASI alignment plan, and its ruthless goal is something like “the humans will be pleased with my answer in the short-term”, then it will be trying to please and convince you (at best), not to be scrupulously correct, and we have abundant real-world evidence that people can be convinced by incorrect alignment plans (otherwise we wouldn’t be in a situation where AGI alignment domain experts disagree with each other all the time) (related post). If its ruthless goal is something like “the humans will ultimately be pleased with my answer in the long-term”, then that’s even worse: it will be trying to take over the world and brainwash us, or other weirder failure modes.
See also my past discussions of reward buttons, debate, “the hope of pinning down non-fuzzy concepts for the AGI to desire”, and more in §6.2.2 below.
So, I think we need to get away from pure ruthless optimization, by mixing in some scrupulous honesty, sincere helpfulness, and so on. And I think that the only way to get those is via virtue-ethics-y dispositions, tied to pride in one’s self-image, and which (in humans) ultimately grows out of Approval Reward.
3. Some failure modes I’m worried about
If we imagine straightforwardly copying Sympathy Reward and Approval Reward into a powerful brain-like AGI, what might go wrong? Lots! But here are three areas in particular that I’ve spent a lot of time thinking about.
3.1 Getting the right “moral circle”
There are a pair of failure modes, corresponding to false negatives and false positives on the question of which entities merit social drives like sympathy. (I mean, “false” is compared to what we were hoping for.) In a human context, we might call the false negatives “dehumanization”, and the false positives “anthropomorphization”.
False negatives are bad because maybe the AGI will have callous indifference to human welfare, just as most humans have callous indifference to bacteria welfare.
False positives are bad because maybe the AGI will care so much about teddy bear “welfare” that it spends resources on that which could have instead been spent on the welfare of humans or other things that actually matter.
In humans, I think we have a set of innate conspecific-detection heuristics (related to faces, gaits, human speech sounds, soft touch, and so on), which reliably generalize to some things (e.g. family members), and reliably don’t generalize to other things (e.g. bacteria). In between those extremes, there are lots of edge-cases where there’s differences across individuals and cultures—e.g. some people care about the feelings of flies, but others don’t.[5]
Actually, the word “generalization” is oversimplifying a far more dynamic and complex process that also interconnects with motivation and understanding. Some people intuitively treat bees as automatons unworthy of interest, and then they read some blog post, or maybe their friends and idols all start caring about bees, and now they care about bees. Or vice versa. Even in the human world, we’re constantly (implicitly) deciding whether to empathize with another person, versus ignore them and shut them out, and that subconscious “decision” is substantially based on which option feels better to us (more motivating and pleasant), all things considered, ultimately because of our innate drives. See Social drives 1, §4.1.2: “Motivation to create false negatives in response to someone’s suffering”.
3.2 Maybe norm-following, helpfulness, etc. relies on a slim chance of getting caught, and thus will collapse under extreme power imbalances?
I spent a long time going back and forth on whether this is a real problem or not. So I’ll pass the microphone to a self-dialogue between the little pessimist who lives on one of my shoulders, versus the optimist on my other shoulder.
Pessimist: There’s a “consequentialist” way to think about Approval Reward, which I concede is not exactly right in its details, but I’ll argue that it has a big kernel of truth. Here’s an example for illustration:
A common idea in my professional circles (see e.g. Kevin Simler & Robin Hanson, Wei Dai, Aella) is that humans want prestige (a.k.a. status), and generally make decisions to maximize their future prestige. This is taken to be a grand unified theory explaining a large swathe of human behavior. And if that doesn’t feel true, well that’s because self-deception is helpful for prestige-seeking too. That’s the theory.
I don’t think that theory explains quite as much as those people think it does, but I agree that it’s a motivation for neurotypical people, and sometimes a very important one. And insofar as a future ASI had that same motivation, it would lead to egregiously bad places. After all, if an ASI wants humans to admire it, it would brainwash humans into adoring fans, and then modify our brains to become capable of feeling astronomical, hitherto-impossible levels of adulation.
Here’s another example. I think I [I-Steve, not I-the-pessimist-on-Steve’s-shoulder] am more scrupulous than average about being honest. What’s my motivation for not lying? Well, my personal experience, which might or might not be typical (I’m more neurotic than most), is that when I’m tempted to say something dishonest, an image will pop into my head of getting caught and feeling ashamed.
Again, insofar as this is true in general, it would portend egregious alignment failure if we tried to make ASIs along similar lines. The idea of getting caught only pops into my head because getting caught seems plausible. Maybe not very plausible, but at least there’s a shade of plausibility. As we approach ASI with ever more power, intelligence, and experience, it would be able to come up with plans for dishonesty for which getting caught would be outrageously implausible, indeed incoherent, like the idea of a static spinning spherical octagon. And then there would be nothing holding it back.
Thus, Approval Reward is a thing that can wring some cooperation out of groups of people with comparable power (and thus a possibility of catching and shaming each other for uncooperative behavior). It is clearly not up to the task of getting ASIs to be nice to humans.
Optimist: I agree that these kinds of consequentialist desires are an aspect of the desires downstream of Approval Reward. But Approval Reward also leads to non-consequentialist desires, particularly related to pride in one’s virtues. See “Act-based approval-directed agents”, for IDA skeptics. And like I’ve been saying for years, you can avoid instrumental convergence by combining consequentialist and non-consequentialist desires (see Consequentialism & corrigibility (2021), and my comment on Deep Deceptiveness (2023)).
As I describe in the “act-based” post, people can (for example) be motivated by how they’d appear in the eyes of a cartoon character, or in the eyes of Jesus, etc. This is obviously not a consequentialist desire! It obviously does not lead to instrumental convergence towards e.g. brainwashing the cartoon character into liking you! The cartoon character doesn’t even exist!
Pessimist: Well actually, some people will write themselves into fan fiction, and have their fictional cartoon character hero compliment them…
Optimist: Oh shush. That’s very uncommon.
Pessimist: Anyway, maybe in the short term you can be driven by non-consequentialist pride, but in the long term we really need real-world anchors—actual real-life experiences of getting approval, getting caught, etc. People do in fact go off the rails in the absence of social contact, and they also go off the rails when they have unchecked power over other people (“absolute power corrupts absolutely”).
Optimist: I’m not sure how “absolute” that really is. For example, Lee Kuan Yew didn’t have absolute power but he had plenty of power for 30 years yet stayed pretty sane and humble.
…Well anyway, even if it were true that, in the human world, niceness needs at least occasional real-world grounding anchors in the form of social contact with a viable chance of getting caught for misdeeds, then we can look into the mechanism that makes this occasional grounding necessary. Maybe there’s some adjustable parameter that we can adjust, such that brain-like AGIs do not have that same requirement as humans.
Pessimist: I doubt it. I think there are deep reasons that the occasional grounding needs to be there, namely “the visceral reaction updating dilemma” (§7.1.1 below).
Optimist: Hmm, I guess we can table that for now.
…But I also want to push back harder on the idea that acting with virtue is reliant on a shade of doubt about whether you’ll get caught.
Like, let’s go back to that example above, where we were talking about how I’m honest in large part because when I’m tempted to say something dishonest, an image will pop into my head of getting caught lying, and feeling ashamed. But hang on. Let’s try introspecting a little more carefully. I’m worried about getting caught lying even in situations where the other person wouldn’t be particularly unhappy to learn that I was lying! Conversely, I’m hardly bothered by publicly disagreeing with someone, even when I know damn well that the someone is going to be annoyed and think less of me—as long as I’m confident that I’m right.
So this was never really about risk-aversion towards getting caught, at least not centrally. The more important dynamic is that I take pride in being honest and correct—those are part of my self-image—and that’s the metric that I’m risk-averse towards failing at, especially publicly but also even privately.
Risk-aversion is a very different thing from niceness. If you have a ruthless sociopath AGI which is risk-averse towards getting caught, you’re in big trouble. It will just wait until it has a really good plan before crushing you.
Me, summing up my current take: I think there is such a thing as virtue-ethics-y motivations that are robust to extreme power imbalances, but it’s not automatic. It only happens in a certain kind of setup. See §4 below.
3.3 Maybe there’s something adjacent to the “nearest unblocked strategy” problem?
Back to the self-dialogue!
Pessimist: (partly copying from here) I’m worried about something akin to the “Nearest Unblocked Strategy” problem. If we put both those things together—consequentialist desires from Sympathy Reward (and Approval Reward), plus virtue-ethics-y motivations from Approval Reward—then I’m worried that the consequentialist desire will eventually “win”. For example, if the AGI wants to eventually get to hedonium, and the AGI also wants to follow societal norms, it might find its way to hedonium via a more gradual route, one that involves gradually and unintentionally, but inexorably, changing societal norms in the direction of hedonium. The virtue-ethics-y motivation just seems more squishy and slippery than the consequentialist desire, especially when it routes through manipulable human desires, such that I’m worried it will not be an adequate bulwark against ruthless consequentialism.
Optimist: (partly copying from here) You’re making the same mistake that we were criticizing long ago in that Deep Deceptiveness comment—you’re assuming that the virtue-ethics-y motivation is stupid and myopic. But why should it be? Wouldn’t the virtue-ethics-y side of the AI notice that the AI is engaging in a systematic pattern of behavior that has the effect of gradually shifting human judgment and culture over time? And wouldn’t it see that as bad? And thus, wouldn’t it vote against behaving in that way?
Pessimist: I dunno, it doesn’t have to be conscious and explicit. Think about how human culture has drifted over time, following local incentives, with barely any forethought.
Optimist: Well sure, in the human world, there’s barely any forethought, because culture mostly develops in a distributed, uncoordinated way, with no foresighted entity in charge. But we’re mostly thinking of ASI singletons (Foom & Doom §1.8.7), which are fully capable of forethought and planning. It’s different.
Pessimist: Hang on. Out of one side of your mouth, you’ve been saying that virtue-ethics-y motivations are non-consequentialist, and this is supposedly at the heart of how they can lead to niceness and norm-following rather than ruthless sociopathic maximization. And now, out of the other side of your mouth, you’re emphasizing that virtue-ethics-y motivations can totally be foresighted and consequentialist. I think you’re trying to have your cake and eat it too.
Optimist: No, I’ve been consistently saying from the start that Approval Reward leads to a mix of consequentialist and non-consequentialist desires. That’s fine. It can be non-consequentialist in ways that entail not being a ruthless sociopath, but also consequentialist in ways that entail caring that the future be full of fun and wonder, not a uniform ever-growing blob of hedonium.
Pessimist: You’re being very vague. I still think you’re trying to have it both ways. You’re hiding the pea under the thimble.
Optimist: Nah, I don’t think so. Try this instead: Think about virtue ethics as centering attention on the narrative of one’s life, involving the past, present, and future, and how that ties into the AGI’s self-image—“what sort of force you are in the world.” This story can be an appealing or unappealing story, from the AGI’s own perspective. In this view, being a nice and cooperative AGI right now can be appealing, and being an AGI that steers humans towards a good future can also be appealing, and these are not competing but rather part of the same framework.
Pessimist: That sounds nice, but obviously they are competing, in the sense that there are inevitable tradeoffs. And you’ve offered me no reason to believe that the “future” part of that story would be salient enough to sway what happens, when the “now” part of the story might be pushing in a different direction.
Optimist: Well you’ve offered me no reason to believe that won’t happen. I never said this plan was guaranteed to work! Just that it might work. Having a plan that might work would still be progress over the status quo of no plan at all.
Pessimist: Hmm, OK, sure.
Optimist: Actually, I’ll say something stronger: In the limit of arbitrarily strong Sympathy Reward, you get ruthless bliss-maxxing. In the opposite limit, of nonzero-but-arbitrarily-weak Sympathy Reward, the Sympathy Reward wouldn’t do anything. Right? So somewhere in between, there should be a range of settings for Sympathy Reward strength in which it’s doing something important but not crushing everything else.
Pessimist: That’s not an airtight argument, but OK fine, I guess so. Of course, that’s not very helpful if we don’t know what the magic setting should be.
Optimist: Yes, that’s an open question.
Oh, here’s something else. Wouldn’t your pessimistic stance apply to humans too? The whole idea of this plan is to be inspired by human motivation systems. Yet humans don’t ruthlessly bliss-maxx! Normal people would not toss people into experience machines against their will.
Pessimist: In the short term, sure, humans don’t ruthlessly bliss-maxx. But in the long term, I think I want to bite that bullet! See my brief earlier discussion under the heading “The arc of progress is long, but it bends towards reward hacking”.
Optimist: Even if that’s true, again, uncoordinated human culture is more of a random walk, whereas an ASI singleton would have more foresight.
Me, summing up my current take: I think the way “pessimist” was originally flagging this problem as intractable-in-principle was wrong. It is of course a possible failure mode in practice; we need to get that balance.
4. What controls the set of virtues that the AGI takes pride in?
4.1 Two pathways: person-first and desire-first
If we want our AGI to take pride in its scrupulous honesty, sincere helpfulness, or whatever else, I think there’s basically two biomimetic paths to getting there. (I think these are not mutually exclusive, but rather two ends of a spectrum.) As an example, let’s talk about how some guy Bob might become a proud wine connoisseur.
In the first path, which I’ll call “person-first”, one or more people who Bob admires—his best friends, his favorite celebrities, etc.—all think that wine is great and wine connoisseurship is awesome. So Bob gravitates towards wine connoisseurship as well, and feels proud about that.
In the second path, which I’ll call “desire-first”, Bob innately really really enjoys drinking wine. Then he’s unusually likely to become a proud wine connoisseur.
The mechanism of the first path is (relatively) straightforward (see my Approval Reward post). The heart of Approval Reward is that Bob finds it rewarding when the other person feels positive feelings when thinking about Bob. So if his friends like wine, and Bob is talking about wine, then that will evoke good feelings in the friends, and Bob can see that, so it triggers innate reward in Bob. Also, even if his friends or favorite celebrities are not physically there, Bob can imagine how they would be feeling good feelings if they saw him doing wine connoisseurship stuff, and this thought will also feel pleasurable to Bob, via generalization, and this then feeds into pride and self-image (Approval Reward §3).
What’s the mechanism for the second path? Well, there are some details where I’m still a bit hazy, but here’s my current understanding.
In the bottom half of the diagram, I showed two (non-mutually-exclusive) mechanisms that wind up in the same place.
In the upper mechanism, the starting point is that wine feels very good and exciting to Bob, for non-social reasons (he innately really likes it). Then, if Bob sees someone else nerding out about wine (whether an acquaintance, or celebrity, or fictional character, or whatever), Bob finds that very good and exciting, which blends into feeling a kind of admiration towards that person.[6] Then Approval Reward flips that back around into making Bob feel proud in his self-image as a wine connoisseur, since after all this is the kind of thing that would be viewed positively by the (new) people he admires.
In the lower mechanism, I labeled the diagram with the cryptic text: “internal cross-talk thing”. This is the thing I talked about in Approval Reward §5: “Approval Reward × Typical Mind Fallacy = ‘reward for sharing special interests’”. I stand by my earlier description of this phenomenon in that link, but I think “typical mind fallacy” was not the best term for it. In particular, “typical mind fallacy” has the connotation that it’s rooted in an error of belief—an “is”, not an “ought”, being objectively wrong—namely the (implicit) belief that the other person has similar preferences as you. If true, that’s a problem, because errors of belief are unlikely to last very long in a smart AGI (to say nothing of an ASI). However, I’m now leaning towards the mechanism not being based on the presence of a false belief, but rather some kind of cross-talk between different thoughts. If so, that’s likelier to remain stable at high intelligence and knowledge. I’m omitting the details here (I’m still quite uncertain), but I want to flag that this is a relatively concrete question that seems very important, and at some point I would like to nail it down to the extent possible.
4.2 Which of those pathways makes sense for AGI alignment?
Now, the “desire-first” pathway might sound thoroughly useless for alignment. If the AGI already innately likes X, then the problem is solved, and who cares about whether that desire connects to its proud self-image, right? But I think that’s wrong. There’s a particular example on my mind: I think an innate aversion to feeling confused, and corresponding delight in gaining understanding, can lead (via the “desire-first” pathway) to a stable pride in one’s honesty, clarity, and insight, being shared in social contexts.
(This is how I explain my own neurotic scrupulosity about being honest, mentioned in §3.2 above, along with my intense nerdy drive to share things that I think I’ve figured out, as you may have noticed from not only my current job but also every major hobby I’ve had since childhood.)
In this example, the aversion-to-feeling-confused by itself would not be alignment-relevant—indeed, any goal-seeking agent will automatically want to avoid confusion, thanks to instrumental convergence. By contrast, scrupulous honesty and clarity in social settings seem potentially quite useful for AGI safety (see §6.2.1 below).
Another example of the “desire-first” pathway (I think) would be §6 of my “Sympathy Reward” post: everyone learns from experience that it’s beneficial to hang around people who are compassionate towards you and people you care about; and this then transmutes into widespread pride in one’s self-image as a person who demonstrates that same trait.
Anyway, I have a general feeling that the “desire-first” pathway is more promising (in the AGI alignment context) than the “person-first” pathway. Why? Well, for one thing, in the “person-first” pathway, the good news is that you can intervene on whose approval the AGI cares about (see §5 below)—let’s say it’s Hugh’s—but the bad news is you lack fine-grained control over how the AGI will get that approval, or in other words, which aspect of Hugh’s social preferences will be salient. I think this naturally leads to coming across in a way that Hugh approves of, as opposed to a (more virtue-ethics-y) motivation to be a kind of AGI that Hugh would approve of. After all, if you look at all the instances of Hugh actually approving of what the AGI did, they might not all involve honesty, integrity, etc., but they do all involve Hugh expressing approval. I think this would lead to an AGI with more of a conventional status drive, happy to lie, cheat, and steal to gain approval. As discussed in §2.3 above, this kind of AGI is not helpful.
Is there a way to rescue the “person-first” pathway? Well, we could hope the AGI gets approval by good and ethical means first, and then gets approval by sneaky and deceptive means later, in which case an appropriately-timed intervention could lock in the good motivations (“under-sculpting”). But I’m doubtful that sincere goodness is really going to be learned before the more problematic kinds of approval-seeking. E.g. I think “coming across as honest” is easier to learn than “honesty”, mostly because the former emerges more naturally from a predictive learning algorithm (since how you come across is relevant to how the other person immediately reacts).
So, how about the “desire-first” pathway? My gut is very fond of this approach, and more specifically, into the idea of getting honesty, truthseeking, and sharing insights via the “desire-first” pathway, as mentioned above.
Is that because I’m narcissistically drawn to the idea of building AGI in my own likeness? Well, umm, probably something like this is relevant to my positive gut reaction, although I’m pretty sure it’s less about narcissism and more about ‘comfort born of familiarity’. And I’m really just talking about one aspect of my temperament; in other respects, man, I have plenty of issues, just like anyone. But, I mean, it’s not totally crazy. For one thing, other things equal, there is an advantage to following human templates, rather than doing something weird and biologically-unprecedented, because the former allows us to draw on human experience when brainstorming failure modes etc. For another thing, if the assumptions of §1.3 don’t hold up, then this post is moot; but if they do hold up, then it would turn out (in hindsight) that I’ve been more on the ball regarding Safe & Beneficial AGI than anyone else on Earth. So if we want our AGI to likewise be on the ball, furthering the cause of Safe & Beneficial AGI, via working on various strategic and technical questions (see §6.2.1 below), just like what I’m doing right now, then trying to make that happen by copying (an aspect of) my temperament into that AGI would seem to be an obvious and prima facie reasonable idea. (And it’s not just me—I think disproportionately many people doing good work in this area have a similar aspect to their temperament.)
4.3 Stepping back a bit
If I somehow knew a lot about algorithms and AI but nothing about humans, and you asked me how to make an AGI that sincerely wants to figure out true and insightful things, and share them, then what would I propose?
Well, I’ll tell you what I wouldn’t propose: “let’s make a social system by labeling concepts corresponding to thinking about humans and specific human feelings, with ‘upstream generalization’ that leads to pride in one’s self-image, and separately let’s ensure that the AI gets very happy and excited about resolving its own confusion, and then those two systems are going to interact via some weird cross-talk thing that leads to the AI wanting to scrupulously find the truth and share it socially.”
That just seems like such a crazy and convoluted solution to this problem.
My shoulder-pessimist says: Dear god, that’s terrifying. Do you really expect this convoluted Rube Goldberg plan to be stable upon distribution shifts like unprecedented power imbalances, unprecedented new technologies and ideas, unprecedented capacity for self-modification, etc.??
My shoulder-optimist says: Hey, this is really encouraging! If there’s this nice possibly-useful technical alignment idea that you never ever would have thought of if you didn’t happen to have the human example available for study, then what else are we missing? It sounds convoluted, but who’s to say that it is convoluted? “Confusion is in the map, not the territory.” Maybe we’re missing some better, more elegant way to think about it. We should be eagerly trying to find the most general framework possible that this one example fits into as an obvious special case. In other words, let’s find some domain-specific language for innate drives in which we can easily write down this plan, and see at a glance how and why it works. And then let’s figure out if there are other useful things we can write down in that same language. After all, a priori, it would be quite surprising if the best plan for AGI alignment is the biomimetic one.
(For example, shouldn’t §4.1 have a third, AGI-specific, pathway that explicitly involves human programmers doing interpretability and somehow intervening? What would that look like?)
5. More details on how to implement human-like social instincts in an AGI
Neuroscience of human social instincts: a sketch is all about the nuts and bolts of how social instincts are built into human brains. If we wanted those to be in AGIs, what would we do?
Ingredient 1: The brain-like AGI has to have a set of concepts that corresponds to human emotions, and they have to be “grounded” / “labeled” in a manner that makes them legible outside the learned model. Possible strategies to build this ingredient are:
Discussion:
All three sets of approaches seem like they would benefit from an explicit comprehensive nuts-and-bolts model of human emotions and moods. We don’t have such a model, and I don’t know how to get one anytime soon.[7] We have bits and pieces of such a model, and it’s not immediately clear to me whether or not that would be good enough.
The “interpretability approaches” have a “visceral reaction updating dilemma” problem (§7.1.1 below).
I’m guessing that the “evolved modularity” approach is kinda incoherent, because the hard work is not in the list of concepts per se, but rather how they relate to everything else in the world, like visual and auditory stimuli, semantic associations with different situations, and so on. And this part has to be learned.
I think I currently lean towards the “interpretability approaches”, but need to think more about the §7.1.1 “updating dilemma” issue. I also need to think about whether the “desire-first pathway” of §4.1 implicitly requires that the AGI is itself feeling human-like emotions.
Ingredient 2: The brain-like AGI has to actually use these concepts when noticing or thinking about humans. Possible strategies to build this ingredient are:
I can’t think of any other plausible plan here, so I guess that one wins by default.
Discussion:
This part is relevant to the “moral circle” challenge discussed in §3.1 above. The human innate conspecific-detection heuristics seem to trigger on human faces, human speech sounds, and so on. For the AGI, we can do all those same things using conventional ML techniques. (E.g. I’m sure that adequate human speech sound detectors already exist.) The innate classifiers in human brains seem to also trigger on charismatic animals. But I think my gut says that AGI should be specifically tuned to care about people, not animals, and then animals can be treated well to the extent that people want that. (And they by-and-large do want that, when it’s costless.)
Ingredient 3: Those thoughts trigger human-like social instincts in brain-like AGI, i.e. we get something like Sympathy Reward and Approval Reward. Possible strategies to build this ingredient are:
Again, that’s the only thing I can think of.
Discussion:
You might be wondering: If I’m so into “desire-first” over “person-first”, why even bother thinking about real-life approval from particular people? Doesn’t that just lead to sycophancy and sneakiness? Why bother?
My tentative answer is: For one thing, the real people definitely need to be involved, since that’s what the AGI is supposed to be sharing its insights with. For another thing, wanting approval from real, bona fide people has good aspects (e.g. respect for democratic fairness) along with the bad aspects (sycophancy and brainwashing). I just think it’s a kinda messy thing where we want to balance the different motivations.
6. What AGI motivations do we want anyway?
The above sections have been mostly working forward from what I (think I) (mostly) know how to do. At the same time, we also want to be working backwards from what motivations we want the AGI to have.
6.1 The shape of the problem: three sets of constraints
When I think about this problem, I feel very boxed in by three sets of constraints. The goal is to describe a path to a maximally-good future that satisfies all those constraints. (Or prove that no such path exists.) The constraints are:
Technical Alignment Constraints—I have no great plan for controlling AGI motivations at all. But for some potential target AGI motivations, I at least have sketchy ideas that I think could probably work, whereas for other potential AGI motivations, I either have no idea at all of how to make an AGI with that motivation, or I have ideas but expect them to work in a pretty brittle way that would eventually break down. Obviously, we need to pick a motivation where we can in fact install that motivation into our AGI.
Strategic Constraints—As discussed in §1.3 above (and Foom & Doom 1), I strongly believe that, once we have brain-like AGI that can do anything at all that’s very important and impressive, we will be frighteningly close to the so-called “acute risk period” where it will become possible for people to create misaligned radical superintelligence that could wipe out humans and run the world by itself. So we would urgently need to end the “acute risk period”, either by making the world resilient to egregiously-misaligned ASI, or by durably preventing such misaligned ASI from coming into existence. I don’t think there would be any way to end the “acute risk period” except by using brain-like AGI to end it (again see Foom & Doom 1), assuming (as always) the scenario of §1.3 above. Thus, a constraint on the brain-like AGI motivation system is that this will facilitate a rapid (months not years) end to the acute risk period. Any plan that doesn’t satisfy that constraint is irrelevant. For example, if someone has invented brain-like AGI, and figures out a safe way to use it to cure cancer, then OK cool, but that’s irrelevant to the problem I’m talking about, so I would promptly change the subject to the second brain-like AGI, presumably created by some other company. If that second AGI is being used to, I dunno, safely solve the Riemann hypothesis, then OK cool, let’s move on to the third one. Eventually down this chain, someone will either make an out-of-control AGI that aggressively grabs power, or will use AGI to somehow forestall that possibility, and that’s what I want to talk about.
Ethical, Societal, and Buy-in Constraints—Whatever the plan is, people have to actually do it. This includes both the plan for alignment itself, and the plan for subsequently protecting the world from misaligned ASI. I’m especially concerned about the possibility that rapidly protecting the world from misaligned ASI will require some kind of drastic and unpalatable action,[9] and we have to find the least awful path, and there will be intense pressure towards wishful thinking and kicking the can down the road.
Relevant context is that, on current trends, the people building brain-like AGI will be extinction risk denialists, or at least not taking extinction risk seriously for reasons that don’t stand up to scrutiny.[10] I doubt that any plan will work in the hands of a team that’s not inclined to think rigorously about safety issues, but we’d like to get as close to that as possible.
6.2 Some potential AGI motivations that we might want to engineer
6.2.1 The brain-like AGI wants to understand the overall strategic situation regarding ASI, and talk to us about options
This is currently my favorite plan. (A.k.a., I hate it and expect it to fail, but all the other plans I can think of seem even worse.)
You might also call it “Truth-seeking disagreeable nerd AGI”, and I was already secretly talking about it in §4.2 above.
The idea here is basically: well, the thing that I’m doing right now, as I write this sentence, is brainstorming what to do about the overall strategic situation regarding AGI and ASI, including technical, philosophical, geopolitical, technological, and every other aspect. And then I’m sharing with other people my understanding of our options and their important expected consequences. I’m not (primarily / consciously) trying to impress people, or follow someone’s instructions, or whatever. I just really want to figure out the truth, and share it, an activity I find both immediately rewarding and important for getting to a good future.
I clearly think that this is a good and important thing to do—otherwise I wouldn’t be doing it! So an obvious idea is: let’s have the AGIs do that same thing, except that they could be way better at it, and think about it 100× faster, etc. The setup would be: the AGI has (read-only) access to every book ever written, website, etc., and it’s trying to figure things out, and it’s talking to humans about its ideas.
This is related to (but different from) the idea of using AIs to help with AI alignment research. That latter idea is controversial (see e.g. The Case Against AI Control Research), and in fact I’m usually one of the people criticizing it. What accounts for the turnaround? First, as is obvious from this post, I think humans need to basically solve technical alignment before building AGI. The AGI can help with technical alignment, but only on narrow (albeit important) questions related to distribution shifts due to future technological development, since the AGI would be mostly reasoning about those before they happen (with some important caveats, see §7.1.3). Second, I expect the AGI to be only incidentally thinking about technical alignment, and mostly thinking about how to defend the world against a possible future out-of-control ASI. Third, an aspect of the technical alignment that humans will be doing is to ensure the AGI really cares about figuring out what’s true, not just about telling us what we want to hear. That’s why I’m so interested in truth-seeking motivations, per §4.2 above.
Would this approach satisfy “Technical Alignment Constraints”? I feel like I have a vague understanding of a path here that’s not totally doomed (again see §4.2 above for how to get sincere truth-seeking, in the context of other aspects of human-like social instincts). A vague idea is far short of where we’d like to be, but it’s a starting point. Anyway, I’ll mostly treat this plan as a baseline, and then talk about whether the “technical alignment constraints” of this option are better or worse than for the other options below.
Would this approach satisfy “Ethical, Societal, and Buy-in Constraints”? Yes, I think this proposal is quite strong here. We’re not having the AGI do anything objectionable. Even people who think that AGI extinction risk is dumb, might nevertheless be open to asking a truth-seeking AGI to try to figure out how future AGIs and ASIs will affect the world, and it’s not completely impossible that the AGI would give a doom-y answer and that the people would believe it. Also, I find that the idea of building an AGI with human-like social instincts is generally pretty easy to explain and has broad appeal (I can even get AI accelerationists on board with it sometimes), and this AGI would fit into that category. I.e., the hope is that the AGI would be vaguely in the human distribution of temperament, albeit pretty far on the nerdier side of that distribution, and I think people would be pretty comfortable with that.
Would this approach satisfy “Strategic Constraints”? Well, there’s a basic problem that (as mentioned above) my belief is that rapidly protecting the world from misaligned ASI will require some kind of drastic and unpalatable action, and we have to find the least awful path, and there will be intense pressure towards wishful thinking and kicking the can down the road. If a truth-seeking brain-like AGI comes to that same conclusion, and shares it with the humans, I don’t particularly expect that the humans will believe the AGI, let alone believe it with such confidence that they take appropriate action to solve the problem. And then we all die. If the AGI builds up credibility in other domains, like forecasting or math, that could help. If the AGI is superhuman at persuasion, that could help too (but that’s a double-edged sword). But yeah, this is definitely a plausible way for this plan to fail (if I’m right about the strategic situation). (Not that I’m seeing any other better options.)
Here’s some more explicit discussion of failure modes:
Maybe the AGI’s suggestions will be wrong? Or it will simply be stumped? After all, there are lots of disagreeable nerds with similar temperament as me, and they sure don’t agree on everything. But I think there’s a ray of hope here; researchers may be able to get superhuman truth-seeking temperament, along with superhuman truth-seeking ability, such that the AGI will be unusually likely to make good predictions, cut through contradictory narratives, and figure out the real situation.
Maybe its suggestions will be right, but the humans disagree or ignore them? As mentioned above, there’s not much to be done about this, except maybe if the AGI builds up extraordinary credibility in other domains, or is super-persuasive.
Maybe it will have insufficient grasp of human values to do this, e.g. its brainstorming efforts are not targeting the right thing, or it discusses options without mentioning features that we in fact care a lot about? But this seems potentially solvable: my idea is that the AGI has something in the vicinity of human-like social instincts—albeit on the nerdy side of the spectrum—and so it can broadly wind up caring about human-like things, and generalizing in human-like ways, I think.
(Note that if the AGI cares too much about the future, then it will try to escape control to take matters into its own hands (which transitions us to §6.2.3); and if it cares too little about the future, it won’t do a good job focusing on what’s important. The optimist says: “there should be a happy medium that splits the difference!”. The pessimist says: “this plan trying to have it both ways, and papering over its self-contradictory incoherence by vague language!”. I tentatively side with the optimist, since some humans seem to be in that happy medium.)
Maybe it will be misaligned, and either tell us what we want to hear instead of what’s true, or outright scheme against us, or escape control? Well yeah, this is part of the technical alignment problem we’re hoping to solve before building this AGI.
Maybe it will need to actually do the really dangerous stuff, like nanotech research or AI capabilities research, in order to understand the threat of a future ASI doing those same things? I guess that, if that’s the situation, the AGI can lay it out to the humans, and then they can decide what to do.
6.2.2 The brain-like AGI wants to solve a specific self-contained math or engineering problem (without otherwise impacting the world)
Things like curing cancer would be in this category, but per the “strategic constraints” in §6.1, nothing matters in this category unless the “specific self-contained or engineering problem” can move the needle on the big problem, namely that someone in the near future will make an out-of-control ASI. In the lingo, the solution would have to enable a “pivotal act”. I don’t really have any viable ideas here. Here are some things that I’ve thought about, and why they don’t strike me as an improvement over §6.2.1:
Whatever Eliezer had in mind? Back in the 2010s, Eliezer Yudkowsky was advocating for something in this genre. (These days he seems to talk about global bans on ASI as Plan A, and corrigibility as Plan B.) However, Eliezer never gave details of exactly what engineering problem he would have the AI solve; I think he said his actual favorite example would be unpalatable (a.k.a. outside the Overton Window). For my part, I also believe that something unpalatable will be required to escape from the pickle of §1.3. But I for one can’t think of any self-contained math or engineering problem that would enable a “pivotal act”, even outside the Overton Window. So either I disagree with Eliezer’s unpublished ideas, or else he knows something I don’t. On top of that, I think the “ethical, societal, and buy-in constraints” above are actually pretty hard constraints (and that ASI will emerge too suddenly for the Overton Window to shift from “warning shots”), so crazy-sounding proposals are basically non-starters for me. In the §1.3 scenario, we’ll be lucky if those people making brain-like AGI even consult me (or my writings) at all for technical alignment advice, rather than just gleefully plowing ahead whatever reward function would superficially make for the most impressive publication. If I were to suggest using the AGI to do something illegal, I would be laughed out of the room, for better or worse.
Whole brain emulation? Other people in my professional circles sometimes suggest the concrete engineering problem of solving whole brain emulation (WBE). The right way to solve WBE, of course, is to not invent brain-like AGI until humans invent brain-scanning tech (cf. Connectomics seems great from an AI x-risk perspective, don’t forget to donate and apply for jobs!) If that happens before AGI, we can talk about how it changes things. But as in §1.3 above, for this post, we’re assuming that we have brain-like AGI and we don’t have human brain scans, and so we’ll be in a bad position where we need to use AGI superpowers to invent brain-scanning tech within a couple years at best, or within months at worst. The only way I see this working is if the AGI radically advances the state of biotech and/or nanotech via simulations and bench experiments, and then invents injectable nanobots that can somehow get into a human brain, scan the trillions of synapses, and relay that information out into a computer. My take on that is:
Inventing a better path to aligned ASI? For example, what if we made a brain-like AGI that wanted to finish Vanessa Kosoy’s computational superimitation research agenda? Then we could shut down the brain-like AGIs and just use the superimitation ASIs instead.
Alas, nobody has (say) a set of well-defined, well-scoped math problems that, if solved, would pave a path to aligned ASI.
Instead, I think ‘inventing a better path to aligned ASI’ requires a much broader kind of sense-making that relies on taste, discernment, and understanding of the strategic landscape.
So, this path winds up being basically the same as §6.2.1. After all, figuring out the technical aspects of getting to Safe & Beneficial AGI is intertwined with figuring out the strategic aspects.
Sabotage, buy out, or talk sense into projects approaching AGI in a dangerous way? This seems worse than §6.2.1 on all three sets of constraints.
Something impressive enough to “wake up” the world to the imminent threat of ASI, thus spurring a global treaty? As discussed in §1.3, this post is premised on certain assumptions that would make it basically too late for a global treaty, no matter how “woken up” people are.[11] Don’t get me wrong, I really care about consensus, rights, boundaries, and all those other nice things. Good international treaties via global consensus would be lovely. But if that’s obviously not going to happen, then I’d rather try something else than roll over and accept human extinction. Human extinction is boundary-violating too.
6.2.3 The brain-like ASI wants to assess the overall strategic situation regarding ASI, and then solve the problem autonomously
This starts out in the same place as §6.2.1, in that the AGI is motivated to brainstorm what to do about the overall strategic situation, and in particular, what to do about the imminent arrival of ever-more-powerful AIs. But then it wants to solve the problem autonomously, rather than chat about its understanding with people. I expect that it would irreversibly escape control (if it was even caged up in the first place), and then go prevent rogue ASIs (via either overwhelming technological advantage or by taking some drastic and illegal action). The hope is that it would use its powers for something good and democratic like Long Reflection, or at least hands-off like a Nanny AI or whatever. Of course, all that is just a guess, what do I know, it would ultimately be up to the AGI.
Would this approach satisfy “Technical Alignment Constraints”? I think the challenge here is strictly harder than §6.2.1. Specifically, (1) there’s more of a distribution shift as the AGI acts autonomously in the world (e.g. developing new technologies, possibly self-improving), (2) we’re more sensitive to the AGI having a spot-on notion of goodness, since it can build whatever the future it wants, whereas in §6.2.1 we care a lot about its notion of goodness, but it’s mostly sharing insights rather than deciding on the whole future, we hope. (Unless it’s super-persuasive, or decides to escape control.)
(Misalignment here is obviously a single point of failure for the whole future, but that’s true in §6.2.1 too.)
I don’t know how to micromanage what kind of future the AGI would want to build. My current best plan would be to give it some modified form of human social instincts (e.g. no spite, more truth-seeking), and let it decide for itself, while we hope for the best.
Would this approach satisfy “Ethical, Societal, and Buy-in Constraints”? Maybe. A lot of people can get behind the idea of “make an AGI that wants a great future”, or “make an AGI with motivations resembling an unusually kind human”. The crazy stuff I was saying above (where the AGI will almost definitely escape human control and entrench itself as global dictator, but hopefully it’s a benevolent dictator and/or one who will eventually yield power to some better process) … that’s what I strongly expect to happen, but notably that doesn’t have to be what the AGI programmers expect to happen. If the programmers are mostly AGI risk denialists (as I expect them to be, see footnote above), they still may endorse a plan that sounds like “make an AGI that wants a great future”, which is what matters.
Would this approach satisfy “Strategic Constraints”? Yes, I’m pretty confident that a competent autonomous brain-like AGI, motivated to prevent any rogue ASI from arising anywhere on Earth perpetually, could make that happen.
Upshot: This proposal beats §6.2.1 on Strategic Constraints, but it’s worse on Technical Alignment Constraints. Pick your poison, decide your doom. Umm, all things considered, I think I currently like §6.2.1 better. But this one here is still very much worth thinking about, especially because if the §6.2.1 AI decides to take matters into its own hands, we wind up back here.
6.2.4 Corrigibility: The brain-like AGI wants to be a helpful assistant, helping people with whatever they’d be doing anyway
(I mean Paul-Christiano-style corrigibility.)
Pessimist: Nope, this fails the “strategic constraints”, because “whatever they’d be doing anyway” is not “trying to make the world resilient to ASI”.
Optimist: Says who? At least some people are trying to make the world resilient to ASI, right? Like, we are! Right now! Why are we not assuming that those “some people” are the ones with corrigible AGI assistants? In other words, you’re pointing out a possible failure mode, but not an inevitable failure mode, nor even one particularly related to technical alignment.
Pessimist: Oh c’mon. Doomers like us are an infinitesimal fraction of the population with infinitesimal resources.
Optimist: Well, when there’s brain-like AGI, the stakes and urgency will feel much higher, and correspondingly more people will be on board. Just look at how the US government has just now finally started taking LLMs seriously.
Pessimist: Brain-like AGI takeoff will be much sharper than LLMs, and the government is not doing anything useful on LLMs anyway.
Well anyway, there’s an even bigger problem, centered around the technical alignment constraints. If the AGI is motivated by immediate human approval, the result is sycophancy. If the AGI is motivated by eventual human approval, the result is brainwashing. If the AGI is motivated by its self-image as a helpful helper, modeled after how most humans think of helpfulness, you still get sycophancy, because it’s not widely seen as “kind” or “helpful” to be a disagreeable nerd who won’t just “agree to disagree” but keeps pressing and pressing when they feel like the other person is importantly mistaken. If the AGI is motivated to be that kind of truth-seeking disagreeable nerd, then we’re back in §6.2.1.
Optimist: Hmm, well I want to push back on the sycophancy part. The AGI can be sycophantic but also helpful. You just need human-checkable tasks.
Pessimist: I don’t think there’s any human-checkable tasks for an AGI to do that would satisfy the “strategic constraints”. Cf. discussion at this link and in §6.2.2.
Optimist: That link brings up debate, but dismisses it on the grounds that the prosecutor AGI, defender AGI, and judge AGI (if applicable) all want to subvert the setup and escape onto the internet. But if the AGIs are helpful-but-sycophantic rather than ruthless, that’s a different situation.
Pessimist: If the AGI wants the person to feel seen and helped, that’s different than wanting to win the debate. So the debate setup is screwed for a different reason.
Optimist: New topic: What about the fact that corrigibility is safer? In §6.2.1, we have the problem that the truth-seeking disagreeable nerd AGI can probably convince the human of anything, so we’re screwed if we mess it up.
Pessimist: We’re screwed if we mess up corrigibility too. There’s no getting around single points of failure.
Optimist: Umm, something about “corrigibility is a basin of attraction”?
Pessimist: I don’t see how that applies to brain-like AGI, and I think the whole “basin of attraction” thing has always been wishful thinking anyway.
Me, summing up my current take: Going for corrigibility seems strictly worse and harder than §6.2.1.
6.2.5 Something else?
I dunno.
7. A couple other random things from my notes
7.1 Two (related) dilemmas
7.1.1 “The visceral reaction updating dilemma”
(This is a snappier version of one of the core ideas in “Perils of under- vs over-sculpting AGI desires”.)
Suppose there’s a system that emits ground-truth labels for what’s good vs bad, or honest vs dishonest, or whatever else. These turn into “visceral reactions” in the AGI,[12] sorta on the “ought” side of the is/ought divide, and they influence what the AGI wants to do.
7.1.2 “The value drift dilemma”
7.1.3 Discussion
For both of these dilemmas, I find that both branches feel intuitively unacceptable.
So when I’m thinking about the problem, I keep messing up as follows: I’ll think about some approach, but then rule it out, on account of one branch of a dilemma. Then I’ll switch to liking some other approach, but then rule that one out too, on account of the other branch of the same dilemma. Back and forth.
I end up feeling stumped, and going around in circles, and feeling like this is an argument that a good solution to technical alignment simply does not exist.
But of course that’s dumb. Both of these dilemmas are just a fact of life, and they would be a fact of life even if AGI were fundamentally impossible (because they also apply to the future of the human world). I need to be more open-minded.
That still leaves open the question: which branch of these dilemmas should I be more open-minded to?
I think my current answer is basically:
7.2 I think I overstated the connection between “under-sculpting” and “non-ruthlessness”
I had a discussion in “Perils of under- vs over-sculpting AGI desires” §6.1: “The Omega-hates-aliens scenario” (you need to read that to understand this section).
What I wrote there isn’t wrong, but in my head I was incorrectly drawing a general intuitive connection between “under-sculpting” (a.k.a. pausing certain types of visceral updates, as in the second branch of the §7.1.1 “visceral reaction updating dilemma” above), and “non-ruthlessness” (in the sense of having a kinda normal-ish set of social intuitions). I now think that that’s not really a useful way to think about it. Under-sculpting per se doesn’t solve the “Omega-hates-aliens scenario” that I talked about in that post; rather, the trick is “pointing the reward function at the right thing”, where the right thing in this case is the belief that the aliens are happy. Under-sculpting by itself leads to things like phobias and risk aversion, not “niceness”. See also §3.2 above.
8. Next steps?
I am fond of the “disagreeable nerd AGI trying to figure out what to do next” intuition of §4.2 and §6.2.1 above, but what I wrote above is obviously very very far short of where we want to be (a fleshed-out plan with associated list of assumptions, and test protocols for de-risking each assumption, along with extra protections against s-risk, and on and on).
After months of (mostly) focused research leading up to this post, I now have a bunch of odds and ends to catch up on. But after that, I’m thinking that the biggest Venn-diagram overlap between “useful for further progress on this issue” and “immediately tractable by me” would be the project I described here, where I would try to wring some insights via trying to reconcile the voluminous literature on human personality variation with my ideas about innate social drives (and specifically, my belief that we can explain personality variation via different innate drives and reflexes having different strengths in different people).
Originally, that had just struck me as a fun thing to do, but now I see how it’s directly relevant. Specifically, here’s the elevator pitch:
This might change in the future, who knows, but at least as of today, the vast majority of people and groups that I would classify as “working towards something like brain-like AGI” seem to have a strong ideological commitment to promptly and openly publishing all their ideas and code, for better or worse. Examples include Astera (my own org), Jeff Hawkins, Yann LeCun, Rich Sutton, and practically everyone in academia.
There are many other social drives besides those four, and I think there are correspondingly many different hypothalamus circuits underlying them. For example, I think there are separate brain circuits related to each of: the “drive to feel feared”, play, a “drive to think about or interact with other people” (§5 here), loneliness (cf. Liu et al. 2025), love, lust, various moods, and so on. I basically don’t see any use for these in AGI alignment. For example, people sometimes talk about how AGIs need to love humanity, but insofar as that’s true (as opposed to being just sentimental nonsense), it doesn’t matter much to me, because I think the engineering principles that enable companionate love in the brain are not interestingly different from the engineering principles that enable Sympathy Reward and Approval Reward.
“Unified” is a bit of an exaggeration; I suspect that the top and bottom rows are orchestrated by different hypothalamus cell groups, even if they have similar mechanisms in important respects. However, the left vs right columns are truly “unified”—they naturally emerge as a pair, from the same innate circuits getting triggered by two different types of everyday situations, see here.
I’m mainly interested in how to set up the motivation system of a brain-like AGI, and specifically whether inserting something like the “compassion / spite circuit” would be a good idea. In that context, an easy and obvious thing to do would be: set the innate “friend (+) vs enemy (–) parameter” to be always positive. In that case, “Schadenfreude Reward” and “Provocation Reward” would never come up.
Is doing that a good idea? Maybe! As an example, I’m not worried about a spite-free AGI failing to fight bullies. Bullies can be fought based on means-end reasoning plus sympathy for the victims. Spite towards the bully is unnecessary.
A thornier issue is that, if we leave any major human emotion out of an AGI altogether, then it potentially messes up the “transient empathetic simulation” aspect of how the “compassion / spite circuit” works. Perhaps we might need spite etc. in our AGIs, alas, but set to very low levels, and/or maybe only temporarily during training. Or maybe we can leave it out altogether, and patch the problem with a non-biomimetic workaround, as discussed in §5.
As a weirder example, see my speculative proposal here that there was a gradual cultural transition in the centuries around 1300–800 BC in the Near East, before which realistic images would generally “seem alive” and trigger social instincts in adults, and after which realistic images generally wouldn’t do that, just as images feel inanimate to us today. Accordingly, Mesopotamian idolatry had a very distinct and weird-to-us character in the period before that transition; the statues were fed, dressed, brought for visits to other idols, etc.
As discussed in Neuroscience of human social instincts §6.3, I innately want Zoe’s approval in proportion to the physiological arousal I feel associated with thoughts of Zoe (assuming that I view Zoe as a friend not enemy). If Zoe frequently talks about something that I find inherently exciting, I think that should count.
It could maybe be done with a complete human connectome plus a ton of interpretative labor. With less neuroscience info, the interpretive labor gets harder, although I do still hope to squeeze some insight out of the personality literature, see §8.
As always, our design decisions could eventually get undone by AGI self-modification / making successors / etc., but I think all the plans I’m proposing would be reflectively stable, if they work at all. Unless I’m missing something. (I didn’t think about it too carefully.)
See Foom & Doom §1.8.7, and maybe also What does it take to defend the world against out-of-control AGIs?
This might change in the future, who knows, but at least as of today, the vast majority of people and groups that I would classify as “working towards something like brain-like AGI” are aggressively dismissive of the idea that human extinction from AGI is possible at all, or think it’s possible but easily solved via some plan that does not stand up to scrutiny. See e.g. my criticisms of Jeff Hawkins, Yann LeCun, Rich Sutton & David Silver, and Blake Richards. My own org, Astera, has a few safety-focused staff including me, which is great! But my esteemed coworkers trying to build brain-like AGI are very much not all on the same page as me about extinction risk. See e.g. my post here addressed to Dileep George, who was at Google at the time but is now Head of AI Research at Astera.
By the way, pretty please don’t read that previous paragraph, and then go found your own brain-like-AGI development project, on the theory that you’ll be really trying to make it safe and aligned, unlike those lousy other guys. That’s basically the origin story behind lots of groups trying to make AGI right now, from OpenAI to Groq (and Astera!), and AFAICT this proliferation of AGI development efforts has only made the situation worse. If we had a great brain-like-AGI technical alignment plan, testing plan, etc., and were bottlenecked only on the care with which that plan is implemented, then I might (or might not) feel differently, but we’re very far from that right now.
To review, the relevant assumptions were: at least one team already has brain-like AGI, many more are close behind, radical superintelligence is in close reach, AGI compute requirements are extremely low, and we do not already have in place a system of draconian global restrictions on who can own small numbers of GPUs or what they can do with them.
In the terminology of Intro series §6, I’m talking about updates to any of the “Thought Assessors”.
As Wei Dai put it in “Ontological Crisis in Humans” (2012): “I don't think loss of faith in God actually constitutes an ontological crisis, or if it does, certainly not a very severe one. An ontology consisting of Gods, Self, Other People, and Dumb Matter just isn't very different from one consisting of Self, Other People, and Dumb Matter (the latter could just be considered a special case of the former with quantity of Gods being 0), especially when you compare either ontology to one made of microscopic particles or even less familiar entities…”