I'm an AGI safety / AI alignment researcher in Boston with a particular focus on brain algorithms. Research Fellow at Astera. See https://sjbyrnes.com/agi.html for a summary of my research and sorted list of writing. Physicist by training. Email: steven.byrnes@gmail.com. Leave me anonymous feedback here. I’m also at: RSS feed, X/Twitter, Bluesky, Substack, LinkedIn, and more at my website.
Thanks. I feel like I want to treat “reward function design” and “AGI motivation design” as more different than you do, and I think your examples above are more about the latter. The reward function is highly relevant to the motivation, but they’re still different.
For example, “reward function design” calls for executable code, whereas “AGI motivation design” usually calls for natural-language descriptions. Or when math is involved, the math in practice usually glosses over tricky ontology identification stuff, like figuring out which latent variables in a potentially learned-from-scratch (randomly-initialized) world model correspond to a human, or a shutdown switch, or a human’s desires, or whatever.
I guess you’re saying that if you have a great “AGI motivation design” plan, and you have somehow operationalized this plan perfectly and completely in terms of executable code, then you can set that exact thing as the reward function, and hope that there’s no inner misalignment / goal misgeneralization. But that latter part is still tricky. …And also, if you’ve operationalized the motivation perfectly, why even have a reward function at all? Shouldn’t you just delete the part of your AI code that does reinforcement learning, and put the already-perfect motivation into the model-based planner or whatever?
Again I acknowledge that “reward function design” and “AGI motivation design” are not wholly unrelated. And that maybe I should read Rubi’s posts more carefully, thanks. Sorry if I’m misunderstanding what you’re saying.
there's some implication here that motivation and positive valence are the same thing?
[will reply to other part of your question later]
Thanks!!
you seem to assume that the cortex's modelling of one's own happiness is very similar to the cortex's modelling of thinking of happiness
I would say “overlaps” rather than “is similar to”. Think of it as vaguely like I-am-juggling versus you-are-juggling. Those are different thoughts, but they overlap, in that they both involve the “juggling” concept. That overlap is very necessary for e.g. recognizing that the same word “juggling” applies to both, and for transferring juggling-related ideas between myself and other people, which we are obviously very capable of doing.
you might argue that it's only the "concept of happiness", which I would agree is present in both scenarios, but it doesn't strike me why that in particular would be learned using this supervised mechanism.
The chain of events would be e.g.
(1) The Thought Generator (world-model) catalogs our own interoceptive feelings into emotion-concepts like "pleasure".
(2) The Thought Generator learns from experience that pleasure has something to do with smiling, e.g. during times where we feel pleasure and notice ourselves smile, or otherwise learn this obvious regularity in the world. This becomes a world-model (thought generator) semantic association “smile-concept” ↔ “pleasure-concept”.
(3) Often we’re paying attention to our own feelings, and then the “pleasure” emotion-concept is active if and only if our immediate interoceptive sensory inputs match “pleasure”. And these times, when we’re paying attention to our own feelings, are the only times where the pleasure Thought Assessor learning rate is nonzero. So the Thought Assessor learns that there’s a robust correlation between the “pleasure-concept” in the Thought Generator and the pleasure innate signal.
(4) Other times we’re NOT paying attention to our own immediate interoceptive sensory inputs, and then the emotion-concepts are “left hanging”, inactive regardless of what we’re feeling. But while they’re left hanging, they can INSTEAD be activated by semantic associations with other parts of our world-model. Then in such a moment, if I see someone smile, it activates smile-concept, which [via (2)] in turn weakly activates pleasure-concept, which in turn [via (3)] weakly activates the pleasure Thought Assessor. This is a candidate “transient empathetic simulation”. But remember, the learning rate of that Thought Assessor is zero whenever the emotion-concepts are “left hanging” like that. So the Thought Assessor won’t disconnect pleasure-concept.
Does that help? Sorry if I’m missing your point. …The above might be hard to follow without a diagram.
analyzing facial cues - in particular humans exhibit micro expressions
The theory that we have evolved direct responses to different facial reactions seems probably wrong to me (or at least, not the main explanation), for a couple reasons:
First, blind people seem to have normal social intuitions.
Second, I don’t think it’s plausible to simultaneously say that microexpressions immediately trigger important innate reactions, and that people are generally bad at consciously noticing microexpressions. When I think of other environmental things that immediately trigger innate reactions, I think of, like, balls flying at my face, big spiders, sudden noises, getting poked, foul smells, etc. We’re VERY good and fast at forming good conscious models of all those environmental things. So it doesn’t seem plausible to me that we could get metaphorically “poked” by microexpressions many times a day for years straight without ever developing a conscious awareness of those microexpressions.
So why do we have them if other people can't pick up on them
For my answer, see Lisa Feldman Barrett versus Paul Ekman on facial expressions & basic emotions. We have “innate behaviors” that impact the face, such as gagging, laughing, and Duchenne-smiling. We also have voluntary control of facial muscles, which we learn to deploy strategically for social signaling. When we use voluntary control to hide the signs of “innate behaviors”, the bit of “innate behavior” that slips through the cracks is a microexpression.
You might ask: why don’t our “innate behaviors” evolve to not impact the face, so that we can hide them better? Hard to say for sure. Probably part of it is that we are only sometimes trying to hide them. Some “innate behavior” facial manifestations might also have more direct adaptive utility (cf. §4.2 of that link). Part of it is probably that the hiding is good enough, because microexpressions are actually hard to notice.
Thanks!
Perhaps you do think that of me
My gut reaction is to cheer you on, but hmm, that might be more tribal affiliation than considered opinion. My considered opinion is: beats me, it’s kinda outside my wheelhouse. ¯\_(ツ)_/¯
most famous for her opinion that it is safe to drink alcohol during pregnancy
Emily Oster thinks that it is safe to drink sufficiently small amounts of alcohol during pregnancy, but super duper unsafe to drink a lot of alcohol during pregnancy. I think you should edit your comment to make that clearer. (Source: I read Expecting Better.)
(No opinion on whether she’s right.)
I feel like my starting-point definition of “reward function” is neither “constitutive” nor “evidential” but rather “whatever function occupies this particular slot in such-and-such RL algorithm”. And then you run this RL algorithm, and it gradually builds a trained agent / policy / whatever we want to call it. And we can discuss the CS question about how that trained agent relates to the thing in the “reward function” slot.
For example, after infinite time in a finite (and fully-explored) environment, most RL algorithms have the property that they will will produce a trained agent that takes actions which maximize the reward function (or the exponentially-discounted sum of future rewards or whatever).
More generally, all bets are off, and RL algorithms might or might not produce trained agents that are aware of the reward function at all, or that care about it, or that relate to it in any other way. These are all CS questions, and generally have answers that vary depending on the particulars of the RL algorithm.
Also, I think that, in the special case of the human brain RL algorithm with its reward function (innate drives like eating-when-hungry), a person’s feelings about their own innate drives are not a good match to either “constitutive” or “evidential”.
So if AGI somehow does have an Approval Reward mechanism, what will count as a relevant or valued approval reward signal? Would AGI see humans as not relevant (like birds -- real, embodied creatures with observable preferences that just don't matter to them), or not valued (out-group, non-valued reference class), and largely discount our approval in their reward systems? Would it see other AGI entities as relevant/valued?
I feel like this discussion can only happen in the context of a much more nuts-and-bolts plan for how this would work in an AGI. In particular, I think the AGI programmers would have various free parameters / intervention points in the code to play around with, some of which may be disanalogous to anything in human or animal brains. So we would need to list those intervention points and talk about what to do with them, and then think about possible failure modes, which might be related to exogenous or endogenous distribution shifts, AGI self-modification / making successors, etc. We definitely need this discussion but it wouldn’t fit in a comment thread.
The way I see it, "making solid services/products that work with high reliability" is solving a lot of the alignment problem.
Funny, I see "high reliability" as part of the problem rather than part of the solution. If a group is planning a coup against you, then your situation is better not worse if the members of this group all have dementia. And you can tell whether or not they have dementia by observing whether they’re competent and cooperative and productive before any coup has started.
If the system is not the kind of thing that could plot a coup even if it wanted to, then it’s irrelevant to the alignment problem, or at least to the most important part of the alignment problem. E.g. spreadsheet software and bulldozers likewise “do a lot of valuable work for us with very low risk”.
There’s a failure mode I described in “The Era of Experience” has an unsolved technical alignment problem:
Basically, I think we need more theoretical progress to find a parametrized space of possible reward functions, where at least some of the reward functions in the space lead to good AGIs that we should want to have around.
I agree that the ideal reward function may have adjustable parameters whose ideal settings are very difficult to predict without trial-and-error. For example, humans vary in how strong their different innate drives are, and pretty much all of those “parameter settings” lead to people getting really messed up psychologically if they’re on one extreme or the opposite extreme. And I wouldn’t know where to start in guessing exactly, quantitatively, where the happy medium is, except via empirical data.
So it would be very good to think carefully about test or optimization protocols for that part. (And that’s itself a terrifyingly hard problem, because there will inevitably be distribution shifts between the test environment and the real world. E.g. An AI could feel compassionate towards other AIs but indifferent towards humans.) We need to think about that, and we need the theoretical progress.