This is a special post for quick takes by Tom Butterweich. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
What's going through your mind, vacuum pump?
.
.
.
Air, and air, and air again.
But less, and less, and less.
4Where exactly is human alignment implemented?
Echoing StanislavKrym, I would not-so-humbly suggest my own Neuroscience of human social instincts: a sketch, along with follow-ups Social drives 1: “Sympathy Reward”, from compassion to dehumanization and Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking :)
My impression, having lived in this world for a while now, is that the formation of an individual is not enough to align it with any kind of morality. Most people will defect to some degree when given the chance -- and defection pays, at least locally.
That seems way too strong. In modern WEIRD culture, it’s not economically beneficial to have children and pets, and to keep them happy and well-fed, but lots of people do anyway, and they don’t leave them to starve, nor torture them just to see what would happen, even if they’re very confident they could get away with it. (And it would be very easy indeed to get away with torturing and killing a companion animal.) That is some nonzero kind of morality.
1. Some of it is hard-wired and passed on through the genome and gene expression: reacting to a baby's cry, being afraid of the dark, ...
2. Some of it is internalised over a lifetime: loyalty to one's family, doing "the right thing" even when no one's watching (though you still get to tell the story), ...
I dispute the implication that a tendency to “internalize” things is not “hard-wired and passed on through the genome” (not sure if you meant to imply that). If you raise an undomesticated animal in a loving family, it will not “internalize” familial bonds. Internalization is a genetic mechanism and one that’s worth understanding; I talk about what controls the virtues that people internalize in §4 here. I’m confident in the big picture but there’s still some important AI-alignment-relevant details that I’m trying to understand better.
Cooperative AI, obviously, but also multi-level selection (unpopular in biology because it's hardly needed there, more useful for explaining culture; was discussed on LW mostly ages ago)
It’s true that if you build AIs that can internalize virtues based on what other AIs (or humans) want, then you need to think about cultural evolution. But I think that, if you have the nuts-and-bolts understanding of how to build AIs that can internalize virtues, then probably you can and should choose what virtues it internalizes in a more direct and engineered way that is pretty different from cultural evolution.
and the pragmatic approaches to trust networks proposed by the web3 people (closely related to alignment stuff in concept, but lacking mutual citations and shared terminology).
I think this is a dead end for reasons discussed in §5 of “6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa”.
1I think you think I was talking about people being “nice vs mean”, whereas I was actually talking about people being “norm-internalizers vs not”. Those are different axes: all four quadrants of the 2×2 exist.
Here’s a “negative” example to show how those two axes are different:
Suppose there’s a school where lots of people are proud racists, especially all the older kids, and all the most beautiful kids, and the captain of the football team and the head cheerleader, etc. Now some new kid goes to that school who previously didn’t have any opinion about racism. They’re probably going to wind up racist. And they won’t just wind up “acting racist as a self-aware ploy to gain respect”; rather, they’ll wind up really drinking the Kool-Aid of racism, and durably taking pride in how racist they are, and proudly acting racist even in situations where no one would ever find out. This is how I was using the term “internalization”.
If you put a paperclip maximizer kid at the same school, they would not wind up durably racist. They might or might not find it pragmatically useful to do racist things to impress their classmates, in situations where other people are watching, but they wouldn’t incorporate racism into their self-image. They don’t really have a self-image, at least not in the sense that we think of it. They’re just taking actions to get more paperclips.
I think if you put a sociopath kid in that school, they would be more like the paperclip maximizer in the sense of developing only a transactional and instrumental interest in performative racism, and not like the neurotypical kids who would incorporate performative racism into their proud self-image and “true identity”. You can find sociopaths talking about how they simply don’t have a “true identity”, at least not in the sense that we’re used to. They just have ways of acting—a collection of masks that they can put on where convenient, without a true self underneath.
It might be 95%/5% rather than 99%/1%, because it’s not just sociopaths, but whatever the exact number, I do think the overwhelming majority of humans are “norm-internalizers” or whatever we want to call them. Again, that’s different from them being nice. But it is a very important ingredient in how the world works.
international relations do a remarkable job at keeping order where constant war was the default
I strongly disagree that we should think of international relations as a situation of pure ruthless Machiavellian self-interest. Rather, in the post-WWII era, lots of people in lots of countries around the world adopted a norm that “wars of territorial conquest are bad”. Those people elected politicians who not only avoided wars of territorial conquest themselves, but also enforced that norm at a cost to themselves, by intervening in defense when they saw other countries starting wars of territorial conquest, which they reacted to with outrage and offense, not just a calculus of self-interest.
So that’s a great example of a stable (ish) equilibrium resting on a foundation of widespread internalization of a certain norm.
LLMs don't possess it (yet)?
I wasn’t expressing any opinion about LLMs.
I think there’s two ways to get a norm-internalizer AI.
The first is to note that norm-internalizing is an aspect of human behavior, so if we simply build an AI that wholesale copies human behavior, then we’ll get a norm-internalizer AI. I think that’s a good starting point for thinking about LLMs, with various caveats (and it’s becoming progressively less true as people crank up post-training RL).
The second is to figure out how humans do it under the hood, and make an AI that has the same mechanism. That’s the part I’ve been working on. It’s the only possible approach in the AI paradigm that I’m trying to prepare for, since the “wholesale copying of human behavior” like LLM pretraining is not a thing that works at all in the context of that paradigm.
(For more on what I’m working on and why, see RL & search is a terrifying way to build AGI (an FAQ).)
1