Alexander and Yudkowsky on AGI goals

Eliezer Yudkowsky

Facile answer: Why, that's just what the Soviets believed, this Skinner-box model of human psychology devoid of innate instincts, and they tried to build New Soviet Humans that way, and failed, which was an experimental test of their model that falsified it.

(UPDATE: I WROTE A BETTER DISCUSSION OF THIS TOPIC AT: Heritability, Behaviorism, and Within-Lifetime RL)

There’s a popular tendency to conflate the two ideas:

“we should think of humans as doing within-lifetime learning by RL”, versus
“we should think of humans as doing within-lifetime learning by RL, where the reward function is whatever parents and/or other authority figures want it to be”

The second is associated with behaviorism, and is IMO preposterous. Intrinsic motivation is a thing; in fact, it’s kinda the only thing! The reward function is in the person’s own head, although things happening in the outside world are some of the inputs to it. Thus parents have some influence on the rewards (just like everything else in the world has some influence on the rewards), but the influence is through many paths, some very indirect, and the net influence is not even necessarily in the direction that the parent imagines it to be (thus reverse psychology is a thing!). My read of behavioral genetics is that approximately nothing that parents do to kids (within the typical distribution) has much if any effect on what kinds of adults their kids will grow into.

(Note the disanalogy to AGI, where the programmers get to write the reward function however they want.)

(…Although there’s some analogy to AGI if we don’t have perfect interpretability of the AGI’s thoughts, which seems likely.)

But none of this is evidence that the first bullet point is wrong. I think the first bullet point is true and important.

Slightly less facile answer: Because people are better at detecting cheating, in problems isomorphic to the Wason Selection Task, than they are at performing the naked Wason Selection Task, the conventional explanation of which is that we have built-in cheater detectors. This is a case in point of how humans aren't blank slates and there's no reason to pretend we are.

IIUC the experiment being referred to here showed that people did poorly on a reasoning task related to the proposition “if a card shows an even number on one face, then its opposite face is red”, but did much better on the same reasoning task related the proposition “If you are drinking alcohol, then you must be over 18”. This was taken to be evidence that humans have an innate cognitive adaptation for cheater-detectors. I think a better explanation is that most people don’t have a deep understanding of IF-THEN, but rather have learned some heuristics that work well enough in the everyday situations where IF-THEN is normally used. But “if you are drinking alcohol, then you must be over 18” is a sensible story. You don’t need a good understanding of IF-THEN to triangulate what the rule is and why it’s being applied. By contrast, the experimental subjects have no particular prior beliefs for “if a card shows an even number on one face, then its opposite face is red”.

In the paper, Cosmides & Tooby purport to rule out “familiarity” as a factor by noting that people do poorly on “If a person goes to Boston, then he takes the subway” and “If a person eats hot chili peppers, then he will drink a cold beer.” But those examples miss the point. If I said to you “Hey I want to tell you something about drinking alcohol and people-under-18…”, then you could already guess what I’m gonna say before I say it. But if I said to you “Hey I want to tell you something about going to Boston and taking the subway”, your guess would be wrong. Boston is very walkable! The conditional in this latter case is not obvious like it is in the former case. In the latter case, you can’t lean on common sense, you have to actually understand how IF-THEN works.

So I would be interested in a Wason selection task experiment on the following proposition: “If the stove is hot, then I shouldn’t put my hand on it”. This is not cheater-detection—it’s your own hand!—but I’d bet that people would do as well as the drinking question. (Maybe it’s already been done. I think there’s a substantial literature on Wason Selection that I haven’t read.)

(As it turns out, I’m open-minded to the possibility that humans do have cognitive adaptations related to cheater-detection, even if I don’t think this Wason selection task thing provides evidence for that. I think that this adaptation (if it exists) would be implemented via the RL reward function, more-or-less. Long story, still a work in progress.)

Actual answer: Because the entire field of experimental psychology that's why.

This excerpt isn’t specific so it’s hard to respond, but I do think there’s a lot of garbage in experimental psychology (like every other field), and more specifically I believe that Eliezer has cited some papers in his old blog posts that are bad papers. (Also, even when experimental results are trustworthy, their interpretation can be wrong.) I have some general thoughts on the field of evolutionary psychology in Section 1 here.

[-]Quintin Pope3y294

I also just tried giving the Wason selection task to text-davinci-003 using the example from Wikipedia, and it didn't get the right answer once in 10 tries. I rephrased the example so it was talking about hands on hot stoves instead, and text-davinci-003 got it right 9/10 times.

The rephrasing:

You are shown a set of four cards placed on a table, each of which has a stove temperature on one side and a hand position on the other side. The visible faces of the cards show "hot stove", "cool stove", "hand on stove" and "hand off stove". Which card(s) must you turn over in order to check if anyone has their hand on a hot stove?

[-]Kaj_Sotala3yΩ112819

I was a bit surprised to see Eliezer invoke the Wason Selection Task. I'll admit that I haven't actually thought this through rigorously, but my sense was that modern machine learning had basically disproven the evpsych argument that those experimental results require the existence of a separate cheating-detection module. As well as generally calling the whole massive modularity thesis into severe question, since the kinds of results that evpsych used to explain using dedicated innate modules now look a lot more like something that could be produced with something like GPT.

... but again I never really thought this through explicitly, it was just a general shift of intuitions that happened over several years and maybe it's wrong.

[-]Garrett Baker3y10

GPT is likely highly modular itself. Most ML models that generalize well are.

[-]Kaj_Sotala3y1410

I haven't read the posts that you're referencing, but I would assume that GPT would exhibit learned modularity - modules that reflect the underlying structure of its training data - rather than innately encoded modularity. E.g. CLIP also ends up having a "Spiderman neuron" that activates when it sees features associated with Spiderman, so you could kind of say that there's a "Spiderman module", but nobody ever sat down to specifically write code that would ensure the emergence of a Spiderman module in CLIP.

Likewise, experimental results like the Wason Selection Task seem to me explainable as outcomes of within-lifetime learning that does end up creating a modular structure out of the data - without there needing to be any particular evolutionary hardwiring for it.

[-]Ilio3y52

Specifying the dataset is one way to ensure some collection of neurons will represent Spiderman specifically, even when it’s not on purpose. « Pay attention to face » sounds enough to make our dataset full of social information, maybe enough to ensure a cheating-detector module (most likely a distributed representation) emerges.

[-]Steven Byrnes3y84

I think that’s a different topic.

We’re talking about the evolved-modularity-vs-universal-learning-machine debate.

Suppose the universal-learning-machine side of the debate is correct. Then the genome builds a big within-lifetime learning algorithm, and this learning algorithm does gradient descent (or whatever other learning rule) and thus gradually builds a trained model in the animal’s brain as it gets older and wiser. It’s possible that this trained model will turn out to be modular. It’s also possible that it won’t. I don’t know which will happen—it’s an interesting question. Maybe I could find out the answer by reading that sequence you linked. But whatever the answer is, this question is not related to the evolved-modularity-vs-universal-learning-machine debate. This whole paragraph is universal-learning-machine either way, by assumption.

By contrast, the evolved modularity side of the debate would NOT look like the genome building a big within-lifetime learning algorithm in the first place. Rather it would look like the genome building an “intuitive biology” algorithm, and an “intuitive physics” algorithm, and an “intuitive human social relations” algorithm, and a vision-processing algorithm, and various other things, with all those algorithms also incorporating learning (somehow—the details here tend to be glossed over IMO).

[-]Quintin Pope3y252

It also seems worth noting that Language models show human-like content effects on reasoning, including on the Wason selection task.

[-]TurnTrout3y*Ω11207

Actual answer: Because the entire field of experimental psychology that's why.
This excerpt isn’t specific so it’s hard to respond, but I do think there’s a lot of garbage in experimental psychology (like every other field), and more specifically I believe that Eliezer has cited some papers in his old blog posts that are bad papers. (Also, even when experimental results are trustworthy, their interpretation can be wrong.) I have some general thoughts on the field of evolutionary psychology in Section 1 here.

Eliezer's reasoning is surprisingly weak here. It doesn't really interact with the strong mechanistic claims he's making ("Motivated reasoning is definitely built-in, but it's built-in in a way that very strongly bears the signature of 'What would be the easiest way to build this out of these parts we handily had lying around already'").

He just flatly states a lot of his beliefs as true:

the conventional explanation of which is that we have built-in cheater detectors. This is a case in point of how humans aren't blank slates and there's no reason to pretend we are.

Conventional explanations are often bogus, and in particular I expect this one to be bogus.

Here, Eliezer states his dubious-to-me stances as obviously True, without explaining how they actually distinguish between mechanistic hypotheses, or e.g. why he thinks he can get so many bits about human learning process hyperparameters from results like Wason (I thought it's hard to go from superficial behavioral results to statements about messy internals? & inferring "hard-coding" is extremely hard even for obvious-seeming candidates).

Similarly, in the summer (consulting my notes + best recollections here), he claimed ~"Evolution was able to make the (internal physiological reward schedule) (learned human values) mapping predictable because it spent lots of generations selecting for alignability on caring about proximate real-world quantities like conspecifics or food" and I asked "why do you think evolution had to tailor the reward system specifically to make this possible? what evidence has located this hypothesis?" and he said "I read a neuroscience textbook when I was 11?", and stared at me with raised eyebrows.

I just stared at him with a shocked face. I thought, surely we're talking about different things. How could that data have been strong evidence for that hypothesis? I didn't understand how could possibly neuroscience textbooks provide huge evidence for evolution having to select the reward->value mapping into its current properties.

I also wrote in my journal at the time:

EY said that we might be able to find a learning process with consistent inner / outer values relations if we spent huge amounts of compute to evolve such a learning process in silicon

Eliezer seems to attach some strange importance to the learning process being found by evolution, even though the learning initial conditions screen off evolution's influence.

I still don't understand that interaction. But I've had a few interactions like this with him, where he confidently states things, and then I ask him why he thinks that, and offers some unrelated-seeming evidence which doesn't -- AFAICT -- actually discriminate between hypotheses.

[-]Daniel V3yΩ452

You (correctly, I believe) distinguish between controlling the reward function and controlling the rewards. This is very important as reflected in your noting the disanalogy to AGI. So I'm a little puzzled by your association of the second bullet point (controlling the reward function, which parents have quite low but non-zero control over) with behaviorism (controlling the rewards, which parents have a lot of control over).

[-]Steven Byrnes3y*Ω352

UPDATE: I WROTE A BETTER DISCUSSION OF THIS TOPIC AT: Heritability, Behaviorism, and Within-Lifetime RL)

Hmm. I’m not sure it’s that important what is or isn’t “behaviorism”, and anyway I’m not an expert on that (I haven’t read original behaviorist writing, so maybe my understanding of “behaviorism” is a caricature by its critics). But anyway, I thought Scott & Eliezer were both interested in the question of what happens when the kid grows up and the parents are no longer around.

My comment above was a bit sloppy. Let me try again. Here are two stories:

“RL with continuous learning” story: The person has an internal reward function in their head, and over time they’ll settle into the patterns of thought & behavior that best tickle their internal reward function. If they spend a lot of time in the presence of their parents, they’ll gradually learn patterns of thought & behavior that best tickle their internal reward function in the presence of their parents. If they spend a lot of time hanging out with friends, they’ll gradually learn patterns of thought & behavior that best tickle their internal reward function when they’re hanging out with friends. As adults in society, they’ll gradually learn patterns of thought & behavior that best tickle their internal reward function as adults in society.

“RL learn-then-get-stuck” story: As Scott wrote in OP, “a child does something socially proscribed (eg steal). Their parents punish them. They learn some combination of "don't steal" and "don't get caught stealing". A few people (eg sociopaths) learn only "don't get caught stealing", but most of the rest of us get at least some genuine aversion to stealing that eventually generalizes into a real sense of ethics.” (And that “real sense of ethics” persists through adulthood.)

I think lots of evidence favors the first story over the second story, at least in humans (I don’t know much about non-human animals). Particularly: (1) heritability studies, (2) cultural shifts, (3) people’s ability to have kinda different personalities in different social contexts, like reverting to childhood roles / personalities when they visit family for the holidays. I don’t want to say that the second story never happens, but it seems to me to be an unusual edge case, like childhood phobias / trauma that persists into adulthood, whereas the first story is central.

That’s one topic, maybe the main one at issue here. Then a second topic is: even leaving aside what happens after the kid grows up, let’s zoom in on childhood. I wrote “If they spend a lot of time in the presence of their parents, they’ll gradually learn patterns of thought & behavior that best tickle their internal reward function in the presence of their parents.” In that context, my comment above was bringing up the fact that IMO parental control over rewards is pretty minimal, such that the “patterns of thought & behavior that best tickle the kid’s internal reward function in the presence of their parents” can be quite different from “the thoughts & behaviors that the parent wishes the kid would have”. I think this has a lot to do with the fact that the parent can’t see inside the kid’s head and issue positive rewards when the kid thinks docile & obedient thoughts, and negative rewards when the kid thinks defiant thoughts. If defiant thoughts are its own reward in the kid’s internal reward function, then the kid is getting a continuous laser-targeted stream of rewards for thinking defiant thoughts, potentially hundreds or thousands of times per day, whereas a parent’s ability to ground their kid or withhold dessert or whatever is comparatively rare and poorly-targeted.

[-]Orpheus163y4835

Thank you for releasing this dialogue-- lots of good object-level stuff here.

In addition, I think Scott showcased some excellent conversational moves. He seemed very good at prompting Yudkowsky well, noticing his own confusions, noticing when he needed to pause/reflect before continuing with a thread, and prioritizing between topics.

I hope that some of these skills are learnable. I expect the general discourse around alignment would be more productive if more people tried to emulate some of Scott's magic.

Some examples that stood out to me:

Acknowledging low familiarity in an area and requesting an explanation at an appropriate level:

Can you expand on sexual recombinant hill-climbing search vs. gradient descent relative to a loss function, keeping in mind that I'm very weak on my understanding of these kinds of algorithms and you might have to explain exactly why they're different in this way?

Acknowledging when he had made progress and the natural next step would be for him to think more (later) on his own:

Okay, I'm going to have to go over all my thoughts on this and update them manually now that I've deconfused that, so I'm going to abandon this topic for now and move on. Do you want to take a break or keep going?

Sending links instead of trying to explain concepts & deciding to move to a new thread (because he wanted to be time-efficient):

How do you feel about me sending you some links later, you can look at them and decide if this is still an interesting discussion, but for now we move on?

Acknowledging the purpose and impact of a "vague question":

I don't think it was a very laser-like consequentialist question, more a vague prompt to direct you into an area where I was slightly confused, and I think it succeeded.

Kudos to Scott. I think these strategies made the discussion more efficient and focused. Also impressive that he was able to do this in a context where he had much less domain knowledge than his conversational partner.

[-]Zack_M_Davis3y3718

I think everyone in the field would be incredibly impressed if they managed to hook up a pretrained GPT to an AlphaStar-for-Minecraft and get back out something that could talk about its strategies with human coplayers. I'd consider that a huge advance in alignment research [bolding mine —ZMD] [...] because of the level of transparency increase it would imply, that there was an AI system that could talk about its internally represented strategies, somehow. Maybe because somebody trained a system to describe outward Minecraft behaviors in English, and then trained another system to play Minecraft while describing in advance what behaviors it would exhibit later, using the first system's output as the labeler on the data.

Huh? Isn't this essentially what Meta's Cicero did for Diplomacy? (No one seemed to think of this as an alignment advance.)

[-]green_leaf3y3-1

Unless I'm missing something, Cicero can talk about its strategies, but only in the sense that its training resulted in its text usually saying such things about its strategies that it usually helps to win the game. Not in the sense that it would have some subpart that would truthfully and reliably report on whatever strategy the network actually has (I'd expect those two goals to contradict each other pretty often (or at least sometimes)).

[-]Quintin Pope3y62

I've heard that this is false. Though I haven't personally read the paper, so I can't comment with confidence.

[-]green_leaf3y30

Oh, I see. It seems like it doesn't work reliably though (the comment says it "doesn't lead to a fully honest agent").

[-]Raemon3y*Ω153021

Okay, no, I think I see the problem, which is that I'm failing to consider that evolutionary-learning and childhood-learning are happening at different times through different algorithms, whereas for AIs they're both happening in the same step by the same algorithm.

Is it actually the case that they're happening "in the same step" for the AI?

I agree with "the thing going on in AI is quite different from the collective learning going on in evolutionary-learning and childhood learning", and I think trying to reason from analogy here is probably generally not that useful. But, my sense is if I was going to map the the "evolutionary learning" bit to most ML stuff, the evolutionary bit is more like "the part where the engineers designed a new architecture / base network", and on one hand engineers are much smarter than evolution, but on the other hand they haven't had millions of years to do it.

[-]cfoster03y2518

I was surprised when I reached this portion of the transcript. As you said, the analogous process to "how evolution happens over genomes" would be "how AI research as a field develops different approaches". Then the analogous process to "how a human's learning process progresses given the innate structures (such-and-such area is wired to such-and-such other area, bias to attend to faces, etc.) & learning algorithms (plasticity rules, dopamine triggers, etc.) specified by their genes" is "how an AI's learning process progresses given the innate structures (network architectures, pretrained components, etc.) & learning algorithms (autoregressive prediction, TD-lambda, etc.) specified by their Pytorch codebase".

See this post from Steve Byrnes as a more fleshed out case along these lines.

I was especially confused when I got to the part where Scott says

Like, we're not going to run evolution in a way where we naturally get AI morality the same way we got human morality, but why can't we observe how evolution implemented human morality, and then try AIs that have the same implementation design?

and Eliezer responds

Not if it's based on anything remotely like the current paradigm, because nothing you do with a loss function and gradient descent over 100 quadrillion neurons, will result in an AI coming out the other end which looks like an evolved human with 7.5MB of brain-wiring information and a childhood.

Say what? AFAICT, the suggestion Scott was making was not that gradient descent would produce the correct 7.5MB of brain-wiring information, but rather that those 7.5MB would be contents written by us intentionally into the Pytorch repo that we plan to train the 100Q neuron network with. In the same way as we ordinarily write ourselves intentionally how many neurons are in each layer, and which parts of the network get which inputs, and what pretrained feature detectors we're using, and which components are frozen vs. trained by loss functions 1+2 vs. trained by loss function 1 only, and which conditions trigger how much reward, and how the model samples policy rollouts etc. etc.

[-]Steven Byrnes3y*2411

Strong agree. To pile on a bit, I think I’m confused about what Eliezer is imagining when he imagines the content of those 7.5MB.

I know what I’m imagining is in those 7.5MB: The within-lifetime learning part has several learning algorithms (and corresponding inference algorithms), neural network architectures, and (space- and time-dependent) hyperparameters. And the other part is calculating the reward function, calculating various other loss functions, and doing lots of odds and ends like regulating heart rate and executing various other innate reactions and reflexes. So for me, these are 7.5MB of more-or-less the same kinds of things that AI & ML people are used to putting into their GitHub repositories.

By contrast, Eliezer is imagining… I’m not sure. That evolution is kinda akin to pretraining, and the 7.5MB are more-or-less specifying millions of individual weights? That I went wrong by even mentioning learning algorithms in the first place? Something else??

[-]Chris_Leong3y1911

I wish Eliezer had been clearer on why we can’t produce an AI that internalises human morality with gradient descent. I agree gradient descent is not the same as a combination of evolutionary learning + within lifetime learning, but it wasn’t clear to me why this meant that no combination of training schedule and/or bias could produce something similar.

[-]Hoagy3y*136

Yeah agreed, this doesn't make sense to me.

There are probably just a few MB (wouldn't be surprised if it could be compressed into much less) of information which sets up the brain wiring. Somewhere within that information are the structures/biases that, when exposed to the training data of being a human in our world, gives us our altruism (and much else). It's a hard problem to understand these altruism-forming structures (which are not likely to be distinct things), replicate them in silica and make them robust even to large power differentials.

On the other hand, the human brain presumably has lots of wiring that pushes it towards selfishness and agenthood which we can hopefully just not replicate.

Either way, it seems that they could in theory be instantiated by the right process of trial and error - the question being whether the error (or misuse) gets us first.

Eliezer expects selfishness not to require any wiring once you select for a certain level of capability, meaning there's no permanent buffer to be gained by not implementing selfishness. The margin for error in this model is thus small, and very hard for us to find without perfect understanding or some huge process.

I agree with this argument for some unknown threshold of capability but it seems strange to phrase it as impossibility unless you're certain that the threshold is low, and even then it's a big smuggled assumption.

EDIT: Looking back on this comment, I guess it comes down to the crux that for systems powerful enough to be relevant to alignment, by virtue of their power or research capability, must be doing strong enough optimisation on some function that we should model them as agents acting to further that goal.

[-]Raemon3yΩ8152

I liked the point about "the reason GPT3 isn't consequentialist is that it doesn't find it's way to the same configuration when you perturb the starting conditions." I think I could have generated that definition of consequentialism, but would have trouble making the connection on-the-fly. (At least, I didn't successfully generate it in between reading Scott's confusion and Eliezer's explanation).

I feel like I now get it more crisply.

[-]peterbarnett3yΩ71310

So could an AI engineer create an AI blob of compute the same size as the brain, with its same structural parameters, feed it the same training data, and get the same result ("don't steal" rather than "don't get caught")?

There is a disconnect with this question.

I think Scott is asking “Supposing an AI engineer could create something that was effectively a copy of a human brain and the same training data, then could this thing learn the “don’t steal” instinct over the “don’t get caught” instinct?”
Eliezer is answering “Is an AI engineer able to create a copy of the human brain, provide it with the same training data a human got, and get the “don’t steal” instinct?”

[-]TurnTrout3yΩ5108

Yeah, this read really bizarrely to me. This is a good way of making sense of that section, maybe. But then I'm still confused why Scott concluded "oh I was just confused in this way" and then EY said "yup that's why you were confused", and I'm still like "nope Scott's question seems correctly placed; evolutionary history is indeed screened off by the runtime hyperparameterization and dataset."

[-]Lukas_Gloor3y128

Thanks for posting this!

I really liked Scott's first question in the section "Analogies to human moral development" and the discussion that ensued there.

I think Eliezer's reply at [14:21] is especially interesting. If I understand it correctly, he's saying that it was a (fortunate) coincidence about what sort of moves evolution had available and what the developmental constraints were at the time, that "build in empathy/pro-social emotions" was an easy way to make people better at earning social rewards from our environment. [And maybe a further argument here is that once we start climbing upward on the gradient towards more empathy, the strategy of "also simultaneously become better at lying and deceiving" no longer gives highest rewards, because there are tradeoffs where it's bad to have (automatically accessible, ever-present) pro-social emotions if you go for a manipulative and exploitative life-strategy.]

By contrast, probably the next part of the argument is that we have no strong reason to expect gradient updates in ML agents to stumble upon a similarly simple attractor as "increase your propensity to experience compassion or feel others' emotions when you're anyway already modeling others' behavior based on what you'd do yourself in their situation." And is this because gradient descent updates too many things at once and there aren't any developmental constraints that would make a simple trick like "dial up pro-social emotions" reliably more successful than alternatives that involve more deception? That seems somewhat plausible to me, but I have some lingering doubts of the form "isn't there a sense in which honesty is strictly easier than deception (related: entangled truths, contagious lies), so ML agents might just stumble upon it if we try to reward them for socially cooperative behavior?"

What's the argument against that? (I'm not arguing for a high probability of "alignment by default" – just against confidently estimating it at <10%.)

Somewhat related: In the context of Shard theory, I shared some speculative thoughts on developmental constraints arguably making it easier (comparative what things could be like if evolution had easier access to more of "mind-design space") to distinguish pro-social from anti-social phenotypes among humans. Mimicking some of these conditions (if we understood AI internals well-enough to steer things) could maybe be a promising component for alignment work?

[-]Yitz3y20

is this because gradient descent updates too many things at once and there aren't any developmental constraints that would make a simple trick like "dial up pro-social emotions" reliably more successful than alternatives that involve more deception? That seems somewhat plausible to me, but I have some lingering doubts of the form "isn't there a sense in which honesty is strictly easier than deception (related: entangled truths, contagious lies), so ML agents might just stumble upon it if we try to reward them for socially cooperative behavior?"
What's the argument against that? (I'm not arguing for a high probability of "alignment by default" – just against confidently estimating it at <10%.)

+1 on this question

[-]Said Achmiz3y114

A question that occurred to me when reading Eliezer’s answer to Scott’s question “Can you expand on sexual recombinant hill-climbing search vs. gradient descent relative to a loss function …”:

How sensitive is the logic of Eliezer’s answer to variations in the numbers he quotes?

For example, at one point in the explanation, Eliezer derives a number, 7.5 megabytes. Now let’s say that we learn that actually, this number should instead be not 7.5 MB, but 75 MB. (For whatever reason—maybe we make some new discovery in genomics, or maybe we find that Eliezer made an arithmetic error; either way, the number is found to be otherwise than Eliezer gives it.)

What effect does this have on the reasoning that Eliezer outlines? What if it’s 750 MB instead? 7.5 GB? 750 KB? 75 TB? etc.

(And likewise the other numbers involved, like “70 million neurons”, etc.)

[-]Raemon3yΩ7118

Facile answer: Why, that's just what the Soviets believed, this Skinner-box model of human psychology devoid of innate instincts, and they tried to build New Soviet Humans that way, and failed, which was an experimental test of their model that falsified it.

On one hand, I've heard a few things about blank-slate experiments that didn't work out, and I do lean towards "they basically don't work". But I... also bet not that many serious attempts actually happened, and that the people attempting them kinda sucked in obvious ways, and that you could do a lot better than however "well" the soviets did.

[-]rahulxyz3y82

GPT-2 does not - probably, very probably, but of course nobody on Earth knows what's actually going on in there - does not in itself do something that amounts to checking possible pathways through time/events/causality/environment to end up in a preferred destination class despite variation in where it starts out.
A blender may be very good at blending apples, that doesn't mean it has a goal of blending apples.
A blender that spit out oranges as unsatisfactory, pushed itself off the kitchen counter, stuck wires into electrical sockets in order to burn open your produce door, grabbed some apples, and blended those apples, on more than one occasion in different houses or with different starting conditions, would much more get me to say, "Well, that thing probably had some consequentialism-nature in it, about something that cashed out to blending apples" because it ended up at highly similar destinations from different starting points in a way that is improbable if nothing is navigating Time.

It doesn't seem crazy to me that a GPT type architecture with the "Stack More Layers" could eventually model the world well enough to simulate consequentialist plans - i.e given a prompt like:

"If you are a blender with legs in environment X, what would you do to blend apples?" and provide a continuation with a detailed plan like the above (and GPT4/5 etc with more compute giving slightly better plans - maybe eventually at a superhuman level)

It also seems like it could do this kind of consequentialist thinking without itself having any "goals" to pursue. I'm expecting the response to be one of the following, but I'm not sure which:

"Well, if it's already make consequentialist plans, surely it has some goals like maximizing the amount of text it generates etc., and will try to do whatever it can to ensure that (similar to the "consequentialist alphago" example in the conversation) instead of just letting itself be turned off.
A LLM / GPT will never be able to reliably output such plans with the current architecture or type of training data.

[-]quetzal_rainbow3y*10

Making hyperbole: very good random number generator sometimes can output numbers corresponding to some consequentialist plans, but it's not very useful as consequentialist.

Lowering level of hyperbole: LLMs trained to a superhuman level can produce consequentialist plans, but it can also produce many non-consequentialist useless plans. If you want it to reliably make good plans (better than human), you should apply some optimization pressure, like RLHF.

[-]robm3y10

There's a difference between "what would you do to blend apples" and "what would you do to unbox an AGI". It's not clear to me if it is just a difference of degree, or something deeper.

[-]Rohin Shah3yΩ670

Okay, then let me try to directly resolve my confusion. My current understanding is something like - in both humans and AIs, you have a blob of compute with certain structural parameters, and then you feed it training data. On this model, we've screened off evolution, the size of the genome, etc - all of that is going into the "with certain structural parameters" part of the blob of compute. So could an AI engineer create an AI blob of compute the same size as the brain, with its same structural parameters, feed it the same training data, and get the same result ("don't steal" rather than "don't get caught")?
[...]
Okay, no, I think I see the problem, which is that I'm failing to consider that evolutionary-learning and childhood-learning are happening at different times through different algorithms, whereas for AIs they're both happening in the same step by the same algorithm. Does that fit your model of what would produce the confusion I was going through above?

@Scott: can you elaborate on what the problem was? I thought the answer to your question was "tautologically yes" (where you have to be careful to have things like "training algorithm", "initial state", etc to be part of the "structural parameters") and I am confused what update you made and what you were previously confused about.

(And it seems several other commenters are confused too.)

I think everyone in the field would be incredibly impressed if they managed to hook up a pretrained GPT to an AlphaStar-for-Minecraft and get back out something that could talk about its strategies with human coplayers. I'd consider that a huge advance in alignment research - nowhere near the point where we all don't die, to be clear, but still hella impressive - because of the level of transparency increase it would imply, that there was an AI system that could talk about its internally represented strategies, somehow.

@Eliezer: For what it's worth, I think it's pretty plausible that we get something like this, and would be interested in betting on it. Though one important clarification: I mean that the AI's text statements appear to us to correspond with and predict its Minecraft behavior, not that the AI's statements reflect the true cognition that resulted in its behavior. (I'm much more uncertain about whether the latter would be the case, and it seems hard to bet on that anyway.)

(The rest of the "human-level at Minecraft but not human-level generality" section seemed roughly right to me.)

[-]Ben Pace3yΩ37-10

[Alexander][14:17]
Can you expand on sexual recombinant hill-climbing search vs. gradient descent relative to a loss function, keeping in mind that I'm very weak on my understanding of these kinds of algorithms and you might have to explain exactly why they're different in this way?
[Yudkowsky][14:21]
It's about the size of the information bottleneck. [followed by a 6 paragraph explanation]

It's sections like this that show me how many levels above me Eliezer is. When I read Scott's question I thought "I can see that these two algorithms are quite different but I don't have a good answer for how they're different", and then Eliezer not only had an answer, but a fully fleshed out mechanistic model of the crucial differences between the two that he could immediately explain clearly, succinctly, and persuasively, in 6 paragraphs. And he only spent 4 minutes writing it.

[-]lc3y3028

I would be more impressed if he had used the information bottleneck as a simple example of a varying training condition, instead of authoritatively declaring it The Difference, accompanied with its own just so story to explain discrepancies in implementation that haven't even been demonstrated. I'm not even sure the analogy is correct; is the 7.5MB storing training parameters or the python code?

[-]Rob Bensinger3yΩ14241

FYI, the timestamp is for the first Discord message. If the log broke out timestamps for every part of the message, it would look like this:

[2:21 PM]
It's about the size of the information bottleneck. The human genome is 3 billion base pairs drawn from 4 possibilities, so 750 megabytes. Let's say 90% of that is junk DNA, and 10% of what's left is neural wiring algorithms. So the code that wires a 100-trillion-synapse human brain is about 7.5 megabytes. Now an adult human contains a lot more information than this. Your spinal cord is about 70 million neurons so probably just your spinal cord has more information than this. That vastly greater amount of runtime info inside the adult organism grows out of the wiring algorithms as your brain learns to move around your muscles, and your eyes open and the retina wires itself and starts directing info on downward to more things that wire themselves, and you learn to read, and so on.
[2:22 PM]
Anything innate that makes reasoning about people out to cheat you, easier than reasoning about isomorphic simpler letters and numbers on cards, has to be packed into the 7.5MB, and gets there via a process where ultimately one random mutation happens at a time, even though lots of mutations are recombining and being selected on at a time.
[2:24 PM]
It's a very slow learning process. It takes hundreds or thousands of generations even for a pretty good mutation to fix itself in the population and become reliably available as a base for other mutations to build on. The entire organism is built out of copying errors that happened to work better than the things they were copied from. Everything is built out of everything else, the pieces that were already lying around for building other things.
[2:27 PM]
When you're building an organism that can potentially benefit from coordinating, trading, with other organisms very similar to itself, and accumulating favors and social capital over long time horizons - and your organism is already adapted to predict what other similar organisms will do, by forcing its own brain to operate in a special reflective mode where it pretends to be the other person's brain - then a very simple way of figuring out what other people will like, by way of figuring out how to do them favors, is to notice what your brain feels when it operates in the special mode of pretending to be the other person's brain.
[2:27 PM]
And one way you can get people who end up accumulating a bunch of social capital is by having people with at least some tendency in them - subject to various other forces and overrides, of course - to feel what they imagine somebody else feeling. If somebody else drops a rock on their foot, they wince.
[2:28 PM]
This is a way to solve a favor-accumulation problem by laying some extremely simple circuits down on top of a lot of earlier machinery.

[-]Ben Pace3yΩ340

That makes more sense.

[-]TurnTrout3yΩ220

Lol, cool. I tried the "4 minute" challenge (without having read EY's answer, but having read yours).

Hill-climbing search requires selecting on existing genetic variance on alleles already in the gene pool. If there isn’t a local mutation which changes the eventual fitness of the properties which that genotype unfolds into, then you won’t have selection pressure in that direction. On the other hand, gradient descent is updating live on a bunch of data in fast iterations which allow running modifications over the parameters themselves. It’s like being able to change a blueprint for a house, versus being able to be at the house in the day and direct repair-people.
The changes happen online, relative to the actual within-cognition goings-on of the agent (e.g. you see some cheese, go to the cheese, get a policy gradient and become more likely to do it again). Compare that to having to try out a bunch of existing tweaks to a cheese-bumping-into agent (e.g. make it learn faster early in life but then get sick and die later), where you can’t get detailed control over its responses to specific situations (you can just tweak the initial setup).
Gradient descent is just a fundamentally different operation. You aren’t selecting over learning processes which unfold into minds, trying out a finite but large gene pool of variants, and then choosing the most self-replicating; you are instead doing local parametric search over what changes outputs on the training data. But RL isn’t even differentiable, you aren’t running gradients through it directly. So there isn’t even an analogue of “training data” in the evolutionary regime.

I think I ended up optimizing for "actually get model onto the page in 4 minutes" and not for "explain in a way Scott would have understood."

[-]Thomas Kwa3y*Ω5106

FWIW this was basically cached for me, and if I were better at writing and had explained this ~10 times before like I expect Eliezer has, I'd be able to do about as well. So would Nate Soares or Buck or Quintin Pope (just to pick people in 3 different areas of alignment), and Quintin would also have substantive disagreements.

[-]Ben Pace3yΩ46-2

Fair enough. Nonetheless, I have had this experience many times with Eliezer, including when dialoguing with people with much more domain-experience than Scott.

[-]Said Achmiz3y20

Could you reconstruct the argument now, having seen it, and without having it in front of you?

[-]Ben Pace3y30

No.

[-]Said Achmiz3y32

Are you quite sure you understand it, then…?

[-]Ben Pace3y20

I'm not certain, but I'm fairly confident I follow the structure of the argument and how it fits into the conversation.

I don't mean to imply I achieved mastery myself from reading the passage, I'm saying that the writer seems to me (from this and other instances) to have a powerful understanding of the domain.

[-]Said Achmiz3y213

Yes, I understood what you meant. What I’m suggesting is that “seems” is precisely the operative word here.

Now, I obviously don’t know what “other instances” you have in mind, so I can’t comment on the validity of your overall impression. But judging just on the basis of this particular explanation, it seems to me that the degree to which it appears to convey a powerful understanding of the domain rather exceeds the degree to which it actually conveys a powerful understanding of the domain. (Note the word “conveys” there—I am not making a claim about the degree to which Eliezer actually understands the domain in question!)

In other words, if the argument that Eliezer gives was bad and wrong, how sure are you that you’d have noticed?

EDIT: For instance, how easily could you answer the question I asked in my top-level comment? Whatever answer you may give—is it the answer Eliezer would give, as well? If you think it is—how sure are you? Trying to answer these questions is one way to check whether you’ve really absorbed a coherent understanding of the matter, I think.

[-]Seth Herd10mo60Review for 2023 Review

I don't think this really qualifies for year's best. It's interesting if you think or have to explain to someone who thinks "just raise an RL mechanism in a human environment and it would come out aligned, right?" I'm surprised anyone thinks that, but here's a pretty good writeup of why you shouldn't.

The biggest portion is about why we shouldn't expect an AGI to become aligned by exposing an RL system to a human - like environment. A child, Alexander says, might be punished for stealing a cookie, and it could internalize the rule "don't get caught stealing" or the rule "don't steal". If the latter, they're aligned.

Yudkowsky says that's not how humans get aligned; we have social instincts designed in by evolution; it's not just plain RL learning. When asked, he says "Actual answer: Because the entire field of experimental psychology that's why."

Having worked in/adjacent to experimental psychology for 20 years or so, I think this is absolutely correct. It's very clear at this point that we have complex instincts that guide and shape our general learning. I would not expect an RL learner to come away with anything like a human value system if it was exposed to human society. I think this should be taken as a given. We are not blank slates, and there is voluminous direct and indirect evidence to that effect.

EY: "So the unfortunate answer to "How do you get humans again?" is "Rerun something a lot like Earth" which I think we both have moral objections about as something to do to sentients."

Also right, but the relevant question is how you get something close enough to humans to roughly share our ethical systems. I'm not even sure that's adequate; humans look a lot more aligned when they don't possess immense power. But maybe we could get something close enough and a bit better by leaving out some of the dangerous instincts like anger. This is Steve Byrnes' research agenda. I don't think we have long enough, since my timeline estimates are mostly short. But it could work.

There's some interesting stuff about the outlines of a mechanism for social instincts. I also endorse EYs summary of the neuroscience as almost certainly correct.

at 1600 YK refers to (I think) human values as something like trapped priors- I think this is right. An AGI would not be likely to get the same priors trapped in the same way, even if it did have similar instincts for prosocial behavior. I'm not sure a human would retain similar values if we lived even 300 years in varied environments and just kept thinking hard about our values.

The remainder of the post is on acausal trades as a route to human survival (e.g., an AGI saving us in case it meets another AGI that was aligned and willl retaliate against a murderous AGI on principal). This is not something I've wrapped my head around.

EY says: "Frankly, I mostly consider this to be a "leave it to MIRI, kids" question"

And I'm happy to do that if he says so. I also don't care to bet the future on a theory that only an insular group even thinks they understand, so I'm hoping we get way more legible and promising alignment plans (and I think we do, which is mostly what I write about).

[-]sig3y60

750 megabytes. Let’s say 90% of that is junk DNA, and 10% of what’s left is neural wiring algorithms. So the code that wires a 100-trillion-synapse human brain is about 7.5 megabytes

What am I missing here? 10% of 750MB is 75MB, not 7.5MB...

[-]Hoagy3y63

10% of what's left, ie of the 75MB of non-junk DNA, so 7.5MB.

fwiw 90% junk DNA seems unlikely, I thought it was largely found to influence gene expression, but then 10% being neural wiring seems high so may cancel to about my own guess.

[-]gwern3y81

You're probably thinking of the debate over ENCODE. It was a furious debate over what the ENCODE results meant, whether some mere chemical activity proved non-junkness, and whether they even measured the narrow chemical thing they claimed to measure and then based the interpretations on; I didn't follow it in detail, but my overall impression was that most people were not convinced by the ENCODE claims and continue to regard junk DNA as being pretty junky (or outright harmful, with all the retrotransposons and viruses lurking in it).

Genome synthesis may help answer this in the not too distant future: it's already been used to create 'minimal organism' bacteria genomes which are much smaller, and synthetic genomes without the 'junk DNA' are appealing because synthesis costs so much and you want to cut corners as much as possible, so proving empirically the junk DNA doesn't matter is obvious and valuable.

[-]Hoagy3y10

Ah interesting, - I'd not heard of ENCODE and wasn't trying to say that there's no such thing as DNA without function.

The way I remembered it was that 10% of DNA was coding, and then a sizeable proportion of the rest was promoters and introns and such, lots of which had fairly recently been reclaimed from 'junk' status. From that wiki, though, it seems that only 1-2% is actually coding.

In any case I'd overlooked the fact that even within genes there's not going to be sensitivity to every base pair.

I'd be super interested if there were any estimates of how many bits in the genome it would take to encode a bit of a neural wiring algorithm as expressed in minified code. I'd guess the DNA would be wildly inefficient and the size of neural wiring algos expressed in code would actually be much smaller than 7.5MB but then it's had a lot of time and pressure to maximise the information content so unsure.

[-]Davidmanheim3yΩ140

So the code that wires a 100-trillion-synapse human brain is about 7.5 megabytes. Now an adult human contains a lot more information than this.

Minor quibble which seems to have implications - "There is a consensus that there are roughly about 100 billion neurons total in the human brain. Each of these neurons can have up to 15,000 connections with other neurons via synapses"

My rough understanding is that babies' brains greatly increase how many synapses there are until age 2 or 3, then these are eliminated or become silent in older children and adults. But this implies that there's a ton of connections, and most of the conditioning and construction of the structure is environmental, not build into the structure via genetics.

[-]Ponder Stibbons3y20

I think this is a very good point.. Evolution has given humans the brain plasticity to create brain connectivity so that a predisposition for morality can be turned into a fully fledged sense of morality. There is, for sure, likely some basic structure in the brain that predisposes us to develop morality but I’d be of the view the crucial basic genes that control this structure are, firstly present in primates, and at least, other mammals, and, secondly, the mutations in these genes required to generate the morally inclined human brain, are far fewer than need be represented by 7.5 MB of information.

One thing both the genome and evolution have taught us is that huge complexity of function and purpose can be generated by a relatively small amount of seed information

[-]Raemon3yΩ131

Not really the main point, but, I would bet:

a) something pretty close to Minecraft will be an important testing ground for some kinds of alignment work.

b) Minecraft itself will probably get a lot of use in AI research as things advance (largely due to being one of the most popular videogames of all time), whether or not it's actually quite the right test-bed. (I think the right test-bed will probably be optimized more directly for ease-of-training).

I think it might be worth Eliezer playing a minecraft LAN party with some friends* for a weekend, so that the "what is minecraft?" question has a more true answer than the cobbled-together intuitions here, if for no other reason that having a clear handle on what people are talking about when they use Minecraft as an example. (But, to be fair, if my prediction bears out it'll be pretty easy to play Minecraft for a weekend later)

*the "with friends" part is extremely loadbearing. Solo minecraft is a different experience. Minecraft is interesting to me for basically being "real life, but lower resolution". If I got uploaded into Minecraft and trapped there forever I'd be sad to be missing some great things, but I think I'd have at least a weak form of most core human experiences, and this requires having other people around.

Minecraft is barely a "game". There is a rough "ascend tech tree and kill cooler monsters" that sort of maps onto Factorio + Skyrim, but the most interesting bits are:

build interesting buildings/structures out of legos
- this begins with "make an interesting house", or a sculpture, but then proceeds to "construct automated factory farms", "figure out ways to hack together flying machines that the minecraft physics engine technically allows but didn't intend", "make music", "build computers that can literally run minecraft". The game getting played here is basically the same thing real life society is playing (i.e. do ever-more-impressive things to keep from getting bored and signal your ally-able and mate-able status, etc)
figure out what resources you need to build the structures you are interested in
build logistical infrastructure and transportation
figure out how to trade with other players so they can get the tools they need to either build interesting structures or go on monster-killing-adventures
invent games to play in minecraft (i.e. capture the flag, parkour racing, etc)
if you're in a PvP server, figure out how to fight against and protect yourself from other players, who are intelligent adversaries who are looking for ways to exploit the game.
Often involves obeying vague social norms that accumulate in the game. It's typically cool to take some stuff from your neighbor's chest, but, not all of it. Sometimes there is mixed PvP where, like, it's okay to sometimes gank someone and take their stuff, but, not all the time.

Training an AI to actually do useful things in this context seems like it requires grappling some things that don't normally come up in games.

I recall some people in CHAI working on a minecraft AI that could help players do useful tasks the players wanted. This was a couple years ago and I assume the work didn't output anything particularly impressive, but I do think some variant of "do useful things without having the rest of the players vote to ban your bot from the game" gets at something alignment-relevant.

I do think most ways people will go about this will be RLFH-like and I don't expect them to scale to superintelligence, and not to be that useful for directly building a pivotal-act capable AGI.

[-]Rohin Shah3yΩ231

I recall some people in CHAI working on a minecraft AI that could help players do useful tasks the players wanted.

I think you're probably talking about my work. This is more of a long-term vision; it isn't doable (currently) at academic scales of compute. See also the "Advantages of BASALT" section of this post.

(Also I just generically default to Minecraft when I'm thinking of ML experiments that need to mimic some aspect of the real world, precisely because "the game getting played here is basically the same thing real life society is playing".)

LESSWRONG
LW

LESSWRONG
LW

179

Alexander and Yudkowsky on AGI goals

179

Ω 59

179

Ω 59

1. Analogies to human moral development

2. Consequentialism and generality

3. Acausal trade, and alignment research opportunities