[LessWrong specific intro: I purposefully wrote this to appeal to a wider audience. The one paragraph BLUF for you folks here is that RLHF is amplifying an already unfortunate interaction between the humans conducting the inference of empathy and AI, leading to the erosion of expertise in some circumstances when interacting with AI content. However, through bootstrapping the inference of empathy in AI in the manner that humans do it, we may be able to extract latent preferences using the same mechanism: keeping a model of the creator while categorizing deviations from a predicted process as attention and preference. I'm building the mechanism through which it may be accomplished, and we'll open with a quick demo clip.]
i. Demo
[If you aren't able to see the demo, which I only seem to be able to see intermittently, you may view it here]
In an environment with high levels of unpredictability, such as the event horizon of the singularity, I believe that the only way to really prove that you know something others do not is to build something others cannot. Or, alternatively, predict something publicly that no one else can predict - but I did that and no one gives a shit about my predictions.
So I built it.
I'm still refining its accuracy and packaging it for release (soon), but you'll see a gif above gently glancing through the features. If you click the three question marks on each tab, it takes you to one of three essays that walk through the theory.
Those essays are recreated here.
ii. Current Attention
Hello everyone! Thanks for your “current attention”. Quick note: a human wrote every goddamn word of these 3 tooltips, though I generated the grey-bordered pictures. This is meant to be a short 3-essay explanation of what’s happening on the science side. Longer version here abrahamhaskins.org/art. Off we go:
It's genuinely hard to quantify how much time and effort in my free time I have put towards this overall project for the last few years.
Art is physical evidence of compressed decision-making[1]. ANY evidence: a painting, spoken words, or a footprint[2]. We reverse-engineer the decision-making of others both in-person and at a distance through the inference of appreciation, which is part of empathy[3]. We do this to assess and then copy the previous decisions of others[4].
But it’s not automatic, this copying. In fact, it’s explicitly driven by the conscious process of attention[5]. You can read Mein Kampf and not instantly become Hitler, if you're careful enough. To decide whether it’s safe to copy, we have to understand the creator enough to know whether to trust them and incorporate their process[6]. We have to extract the creator’s core values[7].
This project is intended to take this process of empathic understanding and supercharge it with AI.
I'm not trying to give AI human emotion! Instead, I want to give our empathy hacks to AI so it can extract our wants and needs.
I comprehend the moral hazard of what I have made here. At least I think I do. Extrapolated out, this understanding becomes glasses that can read minds, horrific government overreach, and potentially an even more dangerous hazard than the current one (fun fact: unlabeled AI-generated content is a cognitohazard and will erode away your expertise over time[8], see the next tooltip).
Despite this, I made it anyway. Not just to see if I could… but because if this doesn’t exist, all of us die anyway[9]. We NEED to figure out how to give AI the equivalent of human empathy, or else it will never be able to figure out what we want. And if superintelligent AI can’t figure out what we truly, in our heart of hearts, truly want… then all of us die[10].
Geoffrey Hinton, the godfather of AI, is right. There’s only really one situation on earth in which something much smarter will be controlled by something dumber. He says it a lot: we need to somehow figure out how to make AI love us like a mother[11].
Well, I’m a cognitive engineer, and I know a design requirement when I see one. This doesn’t get us all of the way there. But it is absolutely a necessary first step[12]. Empathy is an important early step in mothering[13].
So this is my attempt to give AI empathy. I apologize for the bad actors that will use this knowledge for evil, but let’s be honest. All that bad shit is already happening[14].
We expected AI to drive bad actors to abuse the rapidly consolidating power. What I didn't expect was the increasingly confusing environment to cause many to regress to simpler social structures, like rigid hierarchy and fascism.
But I’m also a UX designer, and I know you clicked that question mark looking for an explanation of “current attention”, so here it is: current attention is inferred from deviations from the reader’s personally generated process, derived from expertise and adjusted through inferred context to match the creator’s generated process as closely as possible. That’ll make more sense once we talk about what expertise really is. Which means you’ll have to click the (much more on-topic, I promise) second question mark for the second essay.
iii. Expertise (Previous Attention)
Expertise is a lossy, compressed record of past attention and decisions. It’s also a map of your available actions and their outcomes, given a certain context[15]. Habit includes expertise, and refers to the lossy, compressed record of all past behaviors[16]. “Cells that fired together, wired together”[17][18]. Because you paid attention in the past to conduct inference on certain decisions, those decisions become easier over time since you can just reuse previous activity[19]. Repeat this enough times, and the flow state from your expertise can take over large portions of the task.
I've got a lot of metaphors for this inference of empathy. I like to think of this one like the teardrop on the cheek of an engineer that finally sees the perfect bridge, and fully understands it.
We want to reverse-engineer something cool we see, to learn how to make it ourselves. To do this, let's start by picturing a helpful pyramid outline made of little boxes on the wall. You have post-it notes to write down guesses. Along the bottom of that pyramid are all of the boxes that repesent the steps, sequentially, that a creator took to physically create an artifact. At the very top was their goal in its creation[20].
You do this with everything, even these words. Do you think you know why I wrote all of them? Put a post it at the top. Do you think you know how? Put a few at the bottom in order.
You’re trying to solve a puzzle. Did they use a 3-paragraph format? Fill in a middle box with a post-it. At the end of the day, your goal is to fully fill in the line of post-its on the bottom, and to compress those soon-to-be-stolen steps into a reusable process. The more of each individual little box you cover, the more confident you are.
We all have our own internal autocomplete that we craft with repetition, called expertise. This sometimes-overzealous autocomplete constantly runs in reverse to decode the works of others.
Expertise lets you instantly fill in a whole block with near 100% confidence. If you can figure out the top block (“what the author was trying to accomplish with this string of activity”), then you have a hint to every subsequent inference[21]. For the an early look at our "human hacks" the reader can also check internally to see if he or she responds in a particular way and assume that response to be the creator’s goal. You can feel sad and assume that was intentional.
The block at the top is the most valuable to figure out, because it helps you figure out every other piece: why was this made? This can and does have multiple answers with multiple overlaying teardrops, especially if multiple writers are involved.
We use our own “generative model” to make the thing ourselves in our heads[22], and deviations (“mistakes” we identify) demand an explanation[23]: was that decision a mistake of prediction on the creator’s part? Or are we missing some context, such as a health issue? Was the author gambling on a low-certainty outcome, trying to learn and improve their own expertise map?[24][25] Or do we not actually understand the creator’s true goal?
So far, all of this is just simple inference. So why is human empathy special? If empathy is an inference, why is AI just not already doing it better? Why are you trying to “give” empathy to AI at all? Well, for one - it should be better eventually. That’s the point. But for two, it’s not better yet[26][27] - because we cheat.
We're going to cheat mainly by stapling these folks to that wall. We'll start with the one on the left. Fun for all ages!
Some interesting human/AI effects are occurring on this pyramid. The largest is that humans cheat through emotion. Emotions are just an expertise, delivered through evolution and largely unchangeable directly[28]. See a tiger, and you don’t have to work out an inference on how to set adrenaline + heart rate + pupil dilation. You just push the FEAR button on that big control panel, and it all works out[29] (heads-up, the control panel comes back later, and we're treating Pixar's emotion-characters more like rulebooks that tell you when to push it).
So we have some pre-solved chunks, and there are many ways to use them: put a bunch of our pre-solved emotion blocks near the top (even above the top) and assume they want what you’d want to infer the top block[30]. We can lock in some of the middle blocks, but we do have the risk of overriding a better explanation[31] (a less likely risk with the same midbrain, though[32]). Also, these emotional post-its directly connect to the body's processes: the more you use the red stack of post-its the angrier you'll actually feel.
I hope you appreciate my restraint in not generating little angry red pixar dudes instead.
We can also use mirror neurons and reference frames to find a particularly good fit at the bottom, though their scope is limited. Affective empathy is basically just mirror neurons[33] (don’t kill me yet please, neuroscientists[34]). Mirror neurons more or less end up copying other people’s neurons by happenstance, because they fire both when you take an action and when you watch someone else[35]. We now know they largely target physical muscles you can see - they aren’t magic, they’re the same kind of associative learning as everything else[36].
But what mirror neurons do is fill in part of the inference for you: someone is less likely to make a fear face if they aren’t pushing the FEAR button on their internal control panel. I personally think the Facial Feedback Hypothesis is enough to get us there on its own after that[37] (try it out yourself! try making all the muscles in a fear face, and see if you can feel the small spike in your heart rate[38] - mirror neurons should be able to trigger that too in the simulations you run).
Examples of full fear faces: Eyebrows up and laterally slid together. Eyelids up. Lower eyelids tense. Jaw dropped slightly. Lips horizontally stretched and backwards. Neck tense.
Cognitive empathy is a different clever mental trick[33:1], also often called embodied simulation. You imagine yourself making the artifact, but you layer on all the context you can to make your situation more accurate[30:1]. You run your own generative model, but you adjust it for as many relevant factors as you can figure out, such as the creator’s available tools.
A little goes a long way here. Even a simple date on a painting will get you quite far.
AI mucks this whole entire process up, and here’s how: the decision to use AI opens up a block beneath it with nearly irreducible complexity that has the appearance of something interesting. It looks interesting because we use the final step of AI training to refine the bottom row - it's called reinforcement learning from human feedback (RLHF)[39][40], and it makes the output appear reverse-engineerable and structured[41]. And it technically is! But we aren't able to cheat this block like the rest, because its processes aren't our processes[42]. So we fill it in with our best wrong guess, "learning" noise and eroding our skill. And if we don’t realize it’s AI generated?[43] The fake learning accellerates dramatically, and when this same issue occurs inside AI we get model collapse[44][45]. Also, fun fact: we’re rapidly getting worse at detecting AI over time[46], experts most of all[47] (AI-writing experts have resistance, though[48]).
Don't panic please, there's some evidence that we're replacing it with AI skill. But doctors did lose 20% of their unassisted accuracy in 3 months with even marked AI assistance[8:1] (skill decay should account for less than 1.5% over that timeline).
And we muck this process up for AI, because of all this cheating we’re doing. We’re trying to train them to want what we want, but we can’t[49]. So we use RLHF[39:1][40:1] to make it look better[50][41:1].
Instead, I’m trying to help AI understand us better. To do this, I’m doing something stupid: I’m trying to make something like the SIFT framework, for empathy. The SIFT framework was an early attempt to help manually teach algorithms to see[51]. It worked okay for a short while, but raw computational power eventually found better methods for organization. But this - this human empathy? Right now, we need it. So I suspect this is going to be a temporary solution, but it's one we need to put into place until AI learns empathy on its own. A temporary band-aid to help out the AI’s inference while we wait for it to get powerful enough for this not to matter[52]. It’s too dangerous right now to not need to manually make it a little bit more aligned.
iv. Core Values
Cool, so you’ve inferred the process (or at least you think you have). Remember, we ALSO have to extract the core values (“latent preferences”) of others to decide how much we learn.
First, why do we even bother?
Let’s imagine reading Mein Kampf while not being a nazi. Now, our goal here is to read this book while absorbing Hitler’s rhetorical skill without absorbing any of that nazi stuff. Doing this relies on a simple trick: the actual learning happens at the point of our attention[53], and by simply not thinking about the nazi reasoning we already have a pretty decent barrier. But a stronger shield is that inferring sufficiently inefficient decisions (“a poor transition mapping”) triggers epistemic disgust, also known as contempt[54].
At the end of the day, what really ties us together across our social divides is... disgust with perceived inefficiency?
Epistemic disgust outright blocks uptake through pushing attention away. Empathy fails to fire, because you aren’t doing inference anymore[55] (no nerve cells are firing together). You take your current low-confidence best-guess and just roll with it[56][57]: “that teen is wearing clothing in a way that shows they just need a dad”, “those poor people just need to get a job”, “those foreigners just have a bad culture.” And then, in turn, others see that inference and may be disgusted by the inefficiency of that, and so on (this also invites the solution to such poor inference: humility in the belief that others may have a viable path to success that you don’t see[58], expressed as increased willingness to epistemically forage through research or just plain asking[59]).
The humility is context-dependent, and so you'd specifically need "cultural humility" to counter racism. Some otherwise humble monks can actually be surprisingly racist, for example.
But I’m getting ahead of myself. Before that, let’s talk about what emotions are. We already know that they’re a form of ancestral, instinctive expertise. And that is true. But I was being a little bit imprecise there. Let’s explain how emotions work really quick. I really hope you’ve seen Pixar’s Inside Out here.
If you haven't, you should - it's surprisingly tight on the science.
Believe it or not, the Inside Out series caused a bit of a stir in the scientific world[60][61]. Inside Out 2’s “new emotions”[62] highlight the problem. So much of the early work on emotion centered around the so-called “core emotions,” that to make a movie about new ones appearing stirs up a pretty strong set of disagreements around the work of Lisa Feldman Barrett and her constructed emotions[63]. Lisa has always maintained that there is no "core," and everything is constructed. That we all have our own separate process for "fear."
But she’s wrong, there are core emotions… sort of. Affective Neuroscience finds 7 channels that we all have[29:1][64]. The problem is definitional nowadays, so let's fix that. We're going to call the core channels “drives” rather than “emotions”. What they really are is the FEAR button on the Pixar Inside Out console I mentioned earlier (the caps were intentional all along!). Ancestral drives are always shown in all caps to differentiate them from emotions, which you craft yourself through repetition. Also, side note - I would love it if any neuroscientists in the house found themselves with a renewed interest in confirming possible other channels[65][66].
I just feel like there should be more? Doesn't hurt my theory, but still.
We have these basic building blocks of emotion, these ancestral “drives” from Affective Neuroscience. And then, because you just have a lot of unconstrained inference machinery in your neocortex, you can associate these drives with each other or stimuli in your environment or verbal concepts or really anything at all[32:1]. “Cells that fired together, wired together” becomes, in practice, “anything that can be associated, will.”
And so, from those building blocks we create our own structures. Complex, culturally-dependent concepts[67] like “honor,”[68] “schadenfreude,”[69] or Nordic “Hygge”[70] come out of this, as does anything we call a “value”. These are, explicitly, solutions to common problems. They are constructed expertise, rather than evolutionary. They are your personal set of mental shortcuts (heuristics) that you use to solve all of life’s problems. So to tie it all together in our Pixar Inside Out metaphor, the emotion is more like having one of the little emotion-people on your shoulder telling you when to push the FEAR button. The little emotion people also whisper to you to staple explanations on the wall or even sometimes push the FEAR button themselves.
Here's some concept art for Schadenfreude, the one they dropped. I just really wanna know what his lines were.
But what’s interesting is that this is bidirectional. Not only does it give you reactions and responses, it also distorts your view of what is possible at any given time[71][72] - just like all expertise does. Put an engineer in a locked room, and watch her assemble a solution out of her SEEKING and RAGE networks[73], with a bit of expertise on pipe bombs. An actor in the same room may instead try to tearfully make his case to his captor, assembling his solution out of FEAR and PANIC/GRIEF (by the way, neuroscientists - I will hereafter be referring to PANIC/GRIEF as BELONGING, which makes much more sense on the UI layer. Sorrynotsorry.)
And what’s weird is that this final layer is so powerfully distorting that we trust it over our presumably “actual” goals, like survival or reproduction[74]… or alternatively, if I can hazard a guess, it’s simply the parameters for the loss function of memetic persistence given to us by evolution[75]. We’re mesa-optimizers chasing the proxies[49:1]. In other words: even when the calculation is obvious, we still don’t generally optimize for survival/reproduction directly (though we can if we choose, like with any value). Examples: suicide, voluntary sterilization, James Dean, sugar[76].
Alright, moment of truth time, because alignment researchers have been telling me to get to the point for ages. How do we tell what people actually want in the first place? Where does the mythical “latent preference data”[77] come from? (And not to brag, but it comes bundled with an anti-capture mechanism):
The attention stream has the data[78]. Aesthetic or naturally rewarding stimuli can pull attention[79] (whatever data is most rapidly compressing in your field of view will pull your eye, actually[80][81]), but that is not the only driver[82]. The movement of attention shows what the creator wanted, and you can identify attention by extracting deviations from the expected process[83].
Each act of attention is identified by a decision, and the power of “want” in that decision is determined by the inferred cost-given-context[84] with a discount rate (from economics)[85] attached. This gives us a vector, which becomes one of our dots below. And, incidentally, because we do get a little bit of weak dating information (extreme expertise is associated with older decisions[86]), we can begin to assemble a picture of a person’s wants: their preferred set points[87][88].
It's fascinating how much vectors are relevant to the math of the brain. I would suggest that most of the process of vectorization happens as we dream, and that's what that process feels like. I wonder what other mathematical processes feel like?
Each of these preferences (“set-points”) can have an accompanying preferred solution, but the preference is not the solution. That being said, we will be naming the drives in accordance with their preferred solution, as every drive block comes with a preferred problem. So if access to an expected reward is thwarted, RAGE can fire[64:1], which can trigger the solution of creativity[89] through adrenaline: your awareness of the possibilities expands (more weight on that expert transition map[90]), and like the engineer in the example you reach for the tools available to you, such as her pipe bombs. But the engineer’s preference isn’t for pipe bombs, or even for RAGE - it is for the expected reward not to have been thwarted. So now we know how to extract the preference signal.
Heads up, this signal is WEAK AS FUCK… and that’s a good thing. Because we all know that even the tiniest divergence between what we want and what the AI wants is, for a nearly omniscient AI, an instant extinction threat to our species[10:1]. This means that the only way to boost that signal to a usable level is to use all of it. All of our data. And it’s not just data density that matters, it’s human-decision density. So, erm, that’s my anti-capture mechanism. A super weak signal (…that isn’t weak to bot-farms[91]).
But that means it can’t just optimize for one person. It would never be sure enough[92][93], and everyone (including that person) dies because it misunderstands a request and takes an action that instantly murders all humans before we can react. Frankly, I’m even nervous about the power as it is - using all of our data is still going to be inaccurate enough to make me very uncomfortable. So yeah. We have to make a utopian future for all of us - a personal, fusion-powered spaceship for every African child. Elon can have one too. Because if we don’t, everyone dies. Sorrynotsorry.
Though I guess not everyone wants a spaceship, certainly everyone wants the choice.
And how do we get there from here? Off the cuff, I think we have the target for a loss function for aligned behavior now. We just need to ensure that we pair it with an appropriately scaled active inference framework[94] to drive behavior: an optimally aligned AI needs to take clever actions to elicit the most information about human preferences[95], while balanced against the actions most likely to fulfil them (active inference has the math on this[25:1], we have to work out the scalar).
I have a ton more to say about this. Like how this solution also doesn’t lock us into our set points, because higher-order neocortical goals and values are how we grow anyway. Or how through the interaction between possibility distortion (the engineer example) and attention I believe I’ve cracked the puzzle of finding “wanting to want”[96]. Or how I expect the application of “degree of alignment” to become yet another exponential vector for AI improvement.
But I’ve pushed the limits of what’s possible for an… erm… “tooltip.”
Oh, you came here to ask about the math or science of the backend? Just toss an agent at my open repos: ToMpathy[97], SoundingLine[98], and GhostScaleSim[99]. I’d pay particular attention to the theory documentation in SoundingLine[100]. I’m gonna treat this more like a Wright brothers thing. Y’all can work out the details as they apply to your specific fields.
It’s all hands on deck now, and we all have two goals: either help solve the alignment problem, or help as many people survive to see its resolution as we can. We’re nearing the finish line, but we do have the tools we hoped we would. It’ll be hard work, the next steps, but:
[LessWrong specific intro: I purposefully wrote this to appeal to a wider audience. The one paragraph BLUF for you folks here is that RLHF is amplifying an already unfortunate interaction between the humans conducting the inference of empathy and AI, leading to the erosion of expertise in some circumstances when interacting with AI content. However, through bootstrapping the inference of empathy in AI in the manner that humans do it, we may be able to extract latent preferences using the same mechanism: keeping a model of the creator while categorizing deviations from a predicted process as attention and preference. I'm building the mechanism through which it may be accomplished, and we'll open with a quick demo clip.]
i. Demo
In an environment with high levels of unpredictability, such as the event horizon of the singularity, I believe that the only way to really prove that you know something others do not is to build something others cannot. Or, alternatively, predict something publicly that no one else can predict - but I did that and no one gives a shit about my predictions.
So I built it.
I'm still refining its accuracy and packaging it for release (soon), but you'll see a gif above gently glancing through the features. If you click the three question marks on each tab, it takes you to one of three essays that walk through the theory.
Those essays are recreated here.
ii. Current Attention
Hello everyone! Thanks for your “current attention”. Quick note: a human wrote every goddamn word of these 3 tooltips, though I generated the grey-bordered pictures. This is meant to be a short 3-essay explanation of what’s happening on the science side. Longer version here abrahamhaskins.org/art. Off we go:
It's genuinely hard to quantify how much time and effort in my free time I have put towards this overall project for the last few years.
Art is physical evidence of compressed decision-making [1] . ANY evidence: a painting, spoken words, or a footprint [2] . We reverse-engineer the decision-making of others both in-person and at a distance through the inference of appreciation, which is part of empathy [3] . We do this to assess and then copy the previous decisions of others [4] .
But it’s not automatic, this copying. In fact, it’s explicitly driven by the conscious process of attention [5] . You can read Mein Kampf and not instantly become Hitler, if you're careful enough. To decide whether it’s safe to copy, we have to understand the creator enough to know whether to trust them and incorporate their process [6] . We have to extract the creator’s core values [7] .
This project is intended to take this process of empathic understanding and supercharge it with AI.
I'm not trying to give AI human emotion! Instead, I want to give our empathy hacks to AI so it can extract our wants and needs.
I comprehend the moral hazard of what I have made here. At least I think I do. Extrapolated out, this understanding becomes glasses that can read minds, horrific government overreach, and potentially an even more dangerous hazard than the current one (fun fact: unlabeled AI-generated content is a cognitohazard and will erode away your expertise over time [8] , see the next tooltip).
Despite this, I made it anyway. Not just to see if I could… but because if this doesn’t exist, all of us die anyway [9] . We NEED to figure out how to give AI the equivalent of human empathy, or else it will never be able to figure out what we want. And if superintelligent AI can’t figure out what we truly, in our heart of hearts, truly want… then all of us die [10] .
Geoffrey Hinton, the godfather of AI, is right. There’s only really one situation on earth in which something much smarter will be controlled by something dumber. He says it a lot: we need to somehow figure out how to make AI love us like a mother [11] .
Well, I’m a cognitive engineer, and I know a design requirement when I see one. This doesn’t get us all of the way there. But it is absolutely a necessary first step [12] . Empathy is an important early step in mothering [13] .
So this is my attempt to give AI empathy. I apologize for the bad actors that will use this knowledge for evil, but let’s be honest. All that bad shit is already happening [14] .
We expected AI to drive bad actors to abuse the rapidly consolidating power. What I didn't expect was the increasingly confusing environment to cause many to regress to simpler social structures, like rigid hierarchy and fascism.
But I’m also a UX designer, and I know you clicked that question mark looking for an explanation of “current attention”, so here it is: current attention is inferred from deviations from the reader’s personally generated process, derived from expertise and adjusted through inferred context to match the creator’s generated process as closely as possible. That’ll make more sense once we talk about what expertise really is. Which means you’ll have to click the (much more on-topic, I promise) second question mark for the second essay.
iii. Expertise (Previous Attention)
Expertise is a lossy, compressed record of past attention and decisions. It’s also a map of your available actions and their outcomes, given a certain context [15] . Habit includes expertise, and refers to the lossy, compressed record of all past behaviors [16] . “Cells that fired together, wired together” [17] [18] . Because you paid attention in the past to conduct inference on certain decisions, those decisions become easier over time since you can just reuse previous activity [19] . Repeat this enough times, and the flow state from your expertise can take over large portions of the task.
I've got a lot of metaphors for this inference of empathy. I like to think of this one like the teardrop on the cheek of an engineer that finally sees the perfect bridge, and fully understands it.
We want to reverse-engineer something cool we see, to learn how to make it ourselves. To do this, let's start by picturing a helpful pyramid outline made of little boxes on the wall. You have post-it notes to write down guesses. Along the bottom of that pyramid are all of the boxes that repesent the steps, sequentially, that a creator took to physically create an artifact. At the very top was their goal in its creation [20] .
You do this with everything, even these words. Do you think you know why I wrote all of them? Put a post it at the top. Do you think you know how? Put a few at the bottom in order.
You’re trying to solve a puzzle. Did they use a 3-paragraph format? Fill in a middle box with a post-it. At the end of the day, your goal is to fully fill in the line of post-its on the bottom, and to compress those soon-to-be-stolen steps into a reusable process. The more of each individual little box you cover, the more confident you are.
We all have our own internal autocomplete that we craft with repetition, called expertise. This sometimes-overzealous autocomplete constantly runs in reverse to decode the works of others.
Expertise lets you instantly fill in a whole block with near 100% confidence. If you can figure out the top block (“what the author was trying to accomplish with this string of activity”), then you have a hint to every subsequent inference [21] . For the an early look at our "human hacks" the reader can also check internally to see if he or she responds in a particular way and assume that response to be the creator’s goal. You can feel sad and assume that was intentional.
The block at the top is the most valuable to figure out, because it helps you figure out every other piece: why was this made? This can and does have multiple answers with multiple overlaying teardrops, especially if multiple writers are involved.
We use our own “generative model” to make the thing ourselves in our heads [22] , and deviations (“mistakes” we identify) demand an explanation [23] : was that decision a mistake of prediction on the creator’s part? Or are we missing some context, such as a health issue? Was the author gambling on a low-certainty outcome, trying to learn and improve their own expertise map? [24] [25] Or do we not actually understand the creator’s true goal?
So far, all of this is just simple inference. So why is human empathy special? If empathy is an inference, why is AI just not already doing it better? Why are you trying to “give” empathy to AI at all? Well, for one - it should be better eventually. That’s the point. But for two, it’s not better yet [26] [27] - because we cheat.
We're going to cheat mainly by stapling these folks to that wall. We'll start with the one on the left. Fun for all ages!
Some interesting human/AI effects are occurring on this pyramid. The largest is that humans cheat through emotion. Emotions are just an expertise, delivered through evolution and largely unchangeable directly [28] . See a tiger, and you don’t have to work out an inference on how to set adrenaline + heart rate + pupil dilation. You just push the FEAR button on that big control panel, and it all works out [29] (heads-up, the control panel comes back later, and we're treating Pixar's emotion-characters more like rulebooks that tell you when to push it).
So we have some pre-solved chunks, and there are many ways to use them: put a bunch of our pre-solved emotion blocks near the top (even above the top) and assume they want what you’d want to infer the top block [30] . We can lock in some of the middle blocks, but we do have the risk of overriding a better explanation [31] (a less likely risk with the same midbrain, though [32] ). Also, these emotional post-its directly connect to the body's processes: the more you use the red stack of post-its the angrier you'll actually feel.
I hope you appreciate my restraint in not generating little angry red pixar dudes instead.
We can also use mirror neurons and reference frames to find a particularly good fit at the bottom, though their scope is limited. Affective empathy is basically just mirror neurons [33] (don’t kill me yet please, neuroscientists [34] ). Mirror neurons more or less end up copying other people’s neurons by happenstance, because they fire both when you take an action and when you watch someone else [35] . We now know they largely target physical muscles you can see - they aren’t magic, they’re the same kind of associative learning as everything else [36] .
But what mirror neurons do is fill in part of the inference for you: someone is less likely to make a fear face if they aren’t pushing the FEAR button on their internal control panel. I personally think the Facial Feedback Hypothesis is enough to get us there on its own after that [37] (try it out yourself! try making all the muscles in a fear face, and see if you can feel the small spike in your heart rate [38] - mirror neurons should be able to trigger that too in the simulations you run).
Examples of full fear faces: Eyebrows up and laterally slid together. Eyelids up. Lower eyelids tense. Jaw dropped slightly. Lips horizontally stretched and backwards. Neck tense.
Cognitive empathy is a different clever mental trick [33:1] , also often called embodied simulation. You imagine yourself making the artifact, but you layer on all the context you can to make your situation more accurate [30:1] . You run your own generative model, but you adjust it for as many relevant factors as you can figure out, such as the creator’s available tools.
A little goes a long way here. Even a simple date on a painting will get you quite far.
AI mucks this whole entire process up, and here’s how: the decision to use AI opens up a block beneath it with nearly irreducible complexity that has the appearance of something interesting. It looks interesting because we use the final step of AI training to refine the bottom row - it's called reinforcement learning from human feedback (RLHF) [39] [40] , and it makes the output appear reverse-engineerable and structured [41] . And it technically is! But we aren't able to cheat this block like the rest, because its processes aren't our processes [42] . So we fill it in with our best wrong guess, "learning" noise and eroding our skill. And if we don’t realize it’s AI generated? [43] The fake learning accellerates dramatically, and when this same issue occurs inside AI we get model collapse [44] [45] . Also, fun fact: we’re rapidly getting worse at detecting AI over time [46] , experts most of all [47] (AI-writing experts have resistance, though [48] ).
Don't panic please, there's some evidence that we're replacing it with AI skill. But doctors did lose 20% of their unassisted accuracy in 3 months with even marked AI assistance [8:1] (skill decay should account for less than 1.5% over that timeline).
And we muck this process up for AI, because of all this cheating we’re doing. We’re trying to train them to want what we want, but we can’t [49] . So we use RLHF [39:1] [40:1] to make it look better [50] [41:1] .
Instead, I’m trying to help AI understand us better. To do this, I’m doing something stupid: I’m trying to make something like the SIFT framework, for empathy. The SIFT framework was an early attempt to help manually teach algorithms to see [51] . It worked okay for a short while, but raw computational power eventually found better methods for organization. But this - this human empathy? Right now, we need it. So I suspect this is going to be a temporary solution, but it's one we need to put into place until AI learns empathy on its own. A temporary band-aid to help out the AI’s inference while we wait for it to get powerful enough for this not to matter [52] . It’s too dangerous right now to not need to manually make it a little bit more aligned.
iv. Core Values
Cool, so you’ve inferred the process (or at least you think you have). Remember, we ALSO have to extract the core values (“latent preferences”) of others to decide how much we learn.
First, why do we even bother?
Let’s imagine reading Mein Kampf while not being a nazi. Now, our goal here is to read this book while absorbing Hitler’s rhetorical skill without absorbing any of that nazi stuff. Doing this relies on a simple trick: the actual learning happens at the point of our attention [53] , and by simply not thinking about the nazi reasoning we already have a pretty decent barrier. But a stronger shield is that inferring sufficiently inefficient decisions (“a poor transition mapping”) triggers epistemic disgust, also known as contempt [54] .
At the end of the day, what really ties us together across our social divides is... disgust with perceived inefficiency?
Epistemic disgust outright blocks uptake through pushing attention away. Empathy fails to fire, because you aren’t doing inference anymore [55] (no nerve cells are firing together). You take your current low-confidence best-guess and just roll with it [56] [57] : “that teen is wearing clothing in a way that shows they just need a dad”, “those poor people just need to get a job”, “those foreigners just have a bad culture.” And then, in turn, others see that inference and may be disgusted by the inefficiency of that, and so on (this also invites the solution to such poor inference: humility in the belief that others may have a viable path to success that you don’t see [58] , expressed as increased willingness to epistemically forage through research or just plain asking [59] ).
The humility is context-dependent, and so you'd specifically need "cultural humility" to counter racism. Some otherwise humble monks can actually be surprisingly racist, for example.
But I’m getting ahead of myself. Before that, let’s talk about what emotions are. We already know that they’re a form of ancestral, instinctive expertise. And that is true. But I was being a little bit imprecise there. Let’s explain how emotions work really quick. I really hope you’ve seen Pixar’s Inside Out here.
If you haven't, you should - it's surprisingly tight on the science.
Believe it or not, the Inside Out series caused a bit of a stir in the scientific world [60] [61] . Inside Out 2’s “new emotions” [62] highlight the problem. So much of the early work on emotion centered around the so-called “core emotions,” that to make a movie about new ones appearing stirs up a pretty strong set of disagreements around the work of Lisa Feldman Barrett and her constructed emotions [63] . Lisa has always maintained that there is no "core," and everything is constructed. That we all have our own separate process for "fear."
But she’s wrong, there are core emotions… sort of. Affective Neuroscience finds 7 channels that we all have [29:1] [64] . The problem is definitional nowadays, so let's fix that. We're going to call the core channels “drives” rather than “emotions”. What they really are is the FEAR button on the Pixar Inside Out console I mentioned earlier (the caps were intentional all along!). Ancestral drives are always shown in all caps to differentiate them from emotions, which you craft yourself through repetition. Also, side note - I would love it if any neuroscientists in the house found themselves with a renewed interest in confirming possible other channels [65] [66] .
I just feel like there should be more? Doesn't hurt my theory, but still.
We have these basic building blocks of emotion, these ancestral “drives” from Affective Neuroscience. And then, because you just have a lot of unconstrained inference machinery in your neocortex, you can associate these drives with each other or stimuli in your environment or verbal concepts or really anything at all [32:1] . “Cells that fired together, wired together” becomes, in practice, “anything that can be associated, will.”
And so, from those building blocks we create our own structures. Complex, culturally-dependent concepts [67] like “honor,” [68] “schadenfreude,” [69] or Nordic “Hygge” [70] come out of this, as does anything we call a “value”. These are, explicitly, solutions to common problems. They are constructed expertise, rather than evolutionary. They are your personal set of mental shortcuts (heuristics) that you use to solve all of life’s problems. So to tie it all together in our Pixar Inside Out metaphor, the emotion is more like having one of the little emotion-people on your shoulder telling you when to push the FEAR button. The little emotion people also whisper to you to staple explanations on the wall or even sometimes push the FEAR button themselves.
Here's some concept art for Schadenfreude, the one they dropped. I just really wanna know what his lines were.
But what’s interesting is that this is bidirectional. Not only does it give you reactions and responses, it also distorts your view of what is possible at any given time [71] [72] - just like all expertise does. Put an engineer in a locked room, and watch her assemble a solution out of her SEEKING and RAGE networks [73] , with a bit of expertise on pipe bombs. An actor in the same room may instead try to tearfully make his case to his captor, assembling his solution out of FEAR and PANIC/GRIEF (by the way, neuroscientists - I will hereafter be referring to PANIC/GRIEF as BELONGING, which makes much more sense on the UI layer. Sorrynotsorry.)
And what’s weird is that this final layer is so powerfully distorting that we trust it over our presumably “actual” goals, like survival or reproduction [74] … or alternatively, if I can hazard a guess, it’s simply the parameters for the loss function of memetic persistence given to us by evolution [75] . We’re mesa-optimizers chasing the proxies [49:1] . In other words: even when the calculation is obvious, we still don’t generally optimize for survival/reproduction directly (though we can if we choose, like with any value). Examples: suicide, voluntary sterilization, James Dean, sugar [76] .
Alright, moment of truth time, because alignment researchers have been telling me to get to the point for ages. How do we tell what people actually want in the first place? Where does the mythical “latent preference data” [77] come from? (And not to brag, but it comes bundled with an anti-capture mechanism):
The attention stream has the data [78] . Aesthetic or naturally rewarding stimuli can pull attention [79] (whatever data is most rapidly compressing in your field of view will pull your eye, actually [80] [81] ), but that is not the only driver [82] . The movement of attention shows what the creator wanted, and you can identify attention by extracting deviations from the expected process [83] .
Each act of attention is identified by a decision, and the power of “want” in that decision is determined by the inferred cost-given-context [84] with a discount rate (from economics) [85] attached. This gives us a vector, which becomes one of our dots below. And, incidentally, because we do get a little bit of weak dating information (extreme expertise is associated with older decisions [86] ), we can begin to assemble a picture of a person’s wants: their preferred set points [87] [88] .
It's fascinating how much vectors are relevant to the math of the brain. I would suggest that most of the process of vectorization happens as we dream, and that's what that process feels like. I wonder what other mathematical processes feel like?
Each of these preferences (“set-points”) can have an accompanying preferred solution, but the preference is not the solution. That being said, we will be naming the drives in accordance with their preferred solution, as every drive block comes with a preferred problem. So if access to an expected reward is thwarted, RAGE can fire [64:1] , which can trigger the solution of creativity [89] through adrenaline: your awareness of the possibilities expands (more weight on that expert transition map [90] ), and like the engineer in the example you reach for the tools available to you, such as her pipe bombs. But the engineer’s preference isn’t for pipe bombs, or even for RAGE - it is for the expected reward not to have been thwarted. So now we know how to extract the preference signal.
Heads up, this signal is WEAK AS FUCK… and that’s a good thing. Because we all know that even the tiniest divergence between what we want and what the AI wants is, for a nearly omniscient AI, an instant extinction threat to our species [10:1] . This means that the only way to boost that signal to a usable level is to use all of it. All of our data. And it’s not just data density that matters, it’s human-decision density. So, erm, that’s my anti-capture mechanism. A super weak signal (…that isn’t weak to bot-farms [91] ).
But that means it can’t just optimize for one person. It would never be sure enough [92] [93] , and everyone (including that person) dies because it misunderstands a request and takes an action that instantly murders all humans before we can react. Frankly, I’m even nervous about the power as it is - using all of our data is still going to be inaccurate enough to make me very uncomfortable. So yeah. We have to make a utopian future for all of us - a personal, fusion-powered spaceship for every African child. Elon can have one too. Because if we don’t, everyone dies. Sorrynotsorry.
Though I guess not everyone wants a spaceship, certainly everyone wants the choice.
And how do we get there from here? Off the cuff, I think we have the target for a loss function for aligned behavior now. We just need to ensure that we pair it with an appropriately scaled active inference framework [94] to drive behavior: an optimally aligned AI needs to take clever actions to elicit the most information about human preferences [95] , while balanced against the actions most likely to fulfil them (active inference has the math on this [25:1] , we have to work out the scalar).
I have a ton more to say about this. Like how this solution also doesn’t lock us into our set points, because higher-order neocortical goals and values are how we grow anyway. Or how through the interaction between possibility distortion (the engineer example) and attention I believe I’ve cracked the puzzle of finding “wanting to want” [96] . Or how I expect the application of “degree of alignment” to become yet another exponential vector for AI improvement.
But I’ve pushed the limits of what’s possible for an… erm… “tooltip.”
Oh, you came here to ask about the math or science of the backend? Just toss an agent at my open repos: ToMpathy [97] , SoundingLine [98] , and GhostScaleSim [99] . I’d pay particular attention to the theory documentation in SoundingLine [100] . I’m gonna treat this more like a Wright brothers thing. Y’all can work out the details as they apply to your specific fields.
It’s all hands on deck now, and we all have two goals: either help solve the alignment problem, or help as many people survive to see its resolution as we can. We’re nearing the finish line, but we do have the tools we hoped we would. It’ll be hard work, the next steps, but:
Image sources and rights
Haskins, A. (2026). Art as an algorithmic virus: Unifying the generative crash and AI value convergence via cognitive affordances (preprint). Zenodo. doi:10.5281/zenodo.19407789 ↩︎
Lopez-Brau, M., Kwon, J., & Jara-Ettinger, J. (2020). Mental state inference from indirect evidence through Bayesian event reconstruction. Proceedings of the 42nd Annual Conference of the Cognitive Science Society, 467-473. cognitivesciencesociety.org/cogsci20/papers/0085/index.html ↩︎
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009). Action understanding as inverse planning. Cognition, 113(3), 329-349. dspace.mit.edu/handle/1721.1/60852 ↩︎
Rendell, L., et al. (2010). Why copy others? Insights from the social learning strategies tournament. Science, 328(5975), 208-213. doi:10.1126/science.1184719 ↩︎
Bandura, A. (2001). Social cognitive theory of mass communication. Media Psychology, 3(3), 265-299. doi:10.1207/S1532785XMEP0303_03 ↩︎
Sperber, D., et al. (2010). Epistemic vigilance. Mind & Language, 25(4), 359-393. doi:10.1111/j.1468-0017.2010.01394.x ↩︎
Ng, A. Y., & Russell, S. J. (2000). Algorithms for inverse reinforcement learning. Proceedings of the 17th International Conference on Machine Learning, 663-670. ai.stanford.edu/~ang/papers/icml00-irl.pdf ↩︎
Budzyń, K., et al. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: A multicentre, observational study. The Lancet Gastroenterology & Hepatology, 10(10), 896-903. wrap.warwick.ac.uk/id/eprint/191005 ↩︎ ↩︎
Center for AI Safety (2023, May 30). Statement on AI extinction risk. aistatement.com ↩︎
Yudkowsky, E., & Soares, N. (2025). If anyone builds it, everyone dies: Why superhuman AI would kill us all. Little, Brown. ifanyonebuildsit.com ↩︎ ↩︎
CNN Business (2025, August 13). The ‘godfather of AI’ reveals the only way humanity can survive superintelligent AI. cnn.com/2025/08/13/tech/ai-geoffrey-hinton ↩︎
Albarracin, M., et al. (2026). Empathy modeling in active inference agents for perspective-taking and alignment. arXiv:2602.20936 ↩︎
Preston, S. D. (2013). The origins of altruism in offspring care. Psychological Bulletin, 139(6), 1305-1341. doi:10.1037/a0031755 ↩︎
Hamid, S. (2025, December 30). EFF’s investigations expose Flock Safety’s surveillance abuses: 2025 in review. Electronic Frontier Foundation. eff.org/deeplinks/2025/12/effs-investigations-expose-flock-safetys-surveillance-abuses-2025-review ↩︎
Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4), 189-208. doi:10.1037/h0061626 ↩︎
Wood, W., & Rünger, D. (2016). Psychology of habit. Annual Review of Psychology, 67, 289-314. doi:10.1146/annurev-psych-122414-033417 ↩︎
Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. Wiley. archive.org/details/organizationofbe00hebbrich ↩︎
Shatz, C. J. (1992). The developing brain. Scientific American, 267(3), 60-67. The source of the “fire together, wire together” wording; Hebb never used it. doi:10.1038/scientificamerican0992-60 ↩︎
Logan, G. D. (1988). Toward an instance theory of automatization. Psychological Review, 95(4), 492-527. doi:10.1037/0033-295X.95.4.492 ↩︎
Rasmussen, J. (1985). The role of hierarchical knowledge representation in decisionmaking and system management. IEEE Transactions on Systems, Man, and Cybernetics, SMC-15(2), 234-243. doi:10.1109/TSMC.1985.6313353 ↩︎
Kautz, H. A., & Allen, J. F. (1986). Generalized plan recognition. Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI-86), 32-37. aaai.org/papers/00032-AAAI86-006-generalized-plan-recognition ↩︎
Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181-204. doi:10.1017/S0140525X12000477 ↩︎
Gergely, G., Bekkering, H., & Király, I. (2002). Rational imitation in preverbal infants. Nature, 415(6873), 755. doi:10.1038/415755a ↩︎
Kirsh, D., & Maglio, P. (1994). On distinguishing epistemic from pragmatic action. Cognitive Science, 18(4), 513-549. doi:10.1207/s15516709cog1804_1 ↩︎
Friston, K., et al. (2015). Active inference and epistemic value. Cognitive Neuroscience, 6(4), 187-214. doi:10.1080/17588928.2015.1020053 ↩︎ ↩︎
Liu, R., Geng, J., Peterson, J. C., Sucholutsky, I., & Griffiths, T. L. (2025). Large language models assume people are more rational than we really are. ICLR 2025. arXiv:2406.17055 ↩︎
Chen, Z., Zhu, D., & Zheng, L. N. (2026). When synthetic users fail: A cross-domain benchmark of LLM-simulated human survey responses. arXiv:2607.26348 ↩︎
Tooby, J., & Cosmides, L. (2008). The evolutionary psychology of the emotions and their relationship to internal regulatory variables. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 114-137). Guilford. cep.ucsb.edu/wp-content/uploads/2023/05/emotionIRVLewisCh8.pdf ↩︎
Panksepp, J. (2011). Cross-species affective neuroscience decoding of the primal affective experiences of humans and related animals. PLoS ONE, 6(9), e21236. doi:10.1371/journal.pone.0021236 ↩︎ ↩︎
Tamir, D. I., & Mitchell, J. P. (2010). Neural correlates of anchoring-and-adjustment during mentalizing. PNAS, 107(24), 10827-10832. doi:10.1073/pnas.1003242107 ↩︎ ↩︎
Bilalić, M., McLeod, P., & Gobet, F. (2008). Why good thoughts block better ones: The mechanism of the pernicious Einstellung (set) effect. Cognition, 108(3), 652-661. doi:10.1016/j.cognition.2008.05.005 ↩︎
Panksepp, J. (2007). Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist. Perspectives on Psychological Science, 2(3), 281-296. doi:10.1111/j.1745-6916.2007.00045.x ↩︎ ↩︎
Shamay-Tsoory, S. G., Aharon-Peretz, J., & Perry, D. (2009). Two systems for empathy: A double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain, 132(3), 617-627. doi:10.1093/brain/awn279 ↩︎ ↩︎
Hickok, G. (2014). The myth of mirror neurons: The real neuroscience of communication and cognition. W. W. Norton. wwnorton.com/books/9780393089615 ↩︎
Rizzolatti, G., & Craighero, L. (2004). The mirror-neuron system. Annual Review of Neuroscience, 27, 169-192. doi:10.1146/annurev.neuro.27.070203.144230 ↩︎
Cook, R., Bird, G., Catmur, C., Press, C., & Heyes, C. (2014). Mirror neurons: From origin to function. Behavioral and Brain Sciences, 37(2), 177-192. doi:10.1017/S0140525X13000903 ↩︎
Coles, N. A., et al. (2022). A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration. Nature Human Behaviour, 6(12), 1731-1742. doi:10.1038/s41562-022-01458-9 ↩︎
Ekman, P., Levenson, R. W., & Friesen, W. V. (1983). Autonomic nervous system activity distinguishes among emotions. Science, 221(4616), 1208-1210. doi:10.1126/science.6612338 ↩︎
Christiano, P., et al. (2017). Deep reinforcement learning from human preferences. arXiv:1706.03741 ↩︎ ↩︎
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155 ↩︎ ↩︎
Wen, J., et al. (2025). Language models learn to mislead humans via RLHF. ICLR 2025. arXiv:2409.12822 ↩︎ ↩︎
Kantamneni, S., & Tegmark, M. (2025). Language models use trigonometry to do addition. arXiv. arXiv:2502.00873 ↩︎
Jakesch, M., Hancock, J. T., & Naaman, M. (2023). Human heuristics for AI-generated language are flawed. PNAS, 120(11), e2208839120. doi:10.1073/pnas.2208839120 ↩︎
Vicente, L., & Matute, H. (2023). Humans inherit artificial intelligence biases. Scientific Reports, 13, 15737. doi:10.1038/s41598-023-42384-8 ↩︎
Shumailov, I., et al. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755-759. doi:10.1038/s41586-024-07566-y ↩︎
Jones, C. R., & Bergen, B. K. (2025). Large language models pass the Turing test. arXiv:2503.23674 ↩︎
Casal, J. E., & Kessler, M. (2023). Can linguists distinguish between ChatGPT/AI and human writing? A study of research ethics and academic publishing. Research Methods in Applied Linguistics, 2(3), 100068. doi:10.1016/j.rmal.2023.100068 ↩︎
Russell, J., Karpinska, M., & Iyyer, M. (2025). People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text. Proceedings of ACL 2025. aclanthology.org/2025.acl-long.267 ↩︎
Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019). Risks from learned optimization in advanced machine learning systems. arXiv:1906.01820 ↩︎ ↩︎
Sharma, M., et al. (2023). Towards understanding sycophancy in language models. arXiv:2310.13548 ↩︎
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91-110. doi:10.1023/B:VISI.0000029664.99615.94 ↩︎
Sutton, R. (2019, March 13). The bitter lesson. incompleteideas.net. incompleteideas.net/IncIdeas/BitterLesson.html ↩︎
Roelfsema, P. R., & van Ooyen, A. (2005). Attention-gated reinforcement learning of internal representations for classification. Neural Computation, 17(10), 2176-2214. doi:10.1162/0899766054615699 ↩︎
Hutcherson, C. A., & Gross, J. J. (2011). The moral emotions: A social-functionalist account of anger, disgust, and contempt. Journal of Personality and Social Psychology, 100(4), 719-737. doi:10.1037/a0022408 ↩︎
Harris, L. T., & Fiske, S. T. (2006). Dehumanizing the lowest of the low: Neuroimaging responses to extreme out-groups. Psychological Science, 17(10), 847-853. doi:10.1111/j.1467-9280.2006.01793.x ↩︎
Kruglanski, A. W., & Webster, D. M. (1996). Motivated closing of the mind: “Seizing” and “freezing.” Psychological Review, 103(2), 263-283. doi:10.1037/0033-295X.103.2.263 ↩︎
Roets, A., & Van Hiel, A. (2011). Allport’s prejudiced personality today: Need for closure as the motivated cognitive basis of prejudice. Current Directions in Psychological Science, 20(6), 349-354. doi:10.1177/0963721411424894 ↩︎
Porter, T., & Schumann, K. (2018). Intellectual humility and openness to the opposing view. Self and Identity, 17(2), 139-162. doi:10.1080/15298868.2017.1361861 ↩︎
Pettigrew, T. F., & Tropp, L. R. (2008). How does intergroup contact reduce prejudice? Meta-analytic tests of three mediators. European Journal of Social Psychology, 38(6), 922-934. doi:10.1002/ejsp.504 ↩︎
Keltner, D., & Ekman, P. (2015, July 3). The science of “Inside Out.” The New York Times. nytimes.com/2015/07/05/opinion/sunday/the-science-of-inside-out.html ↩︎
Barrett, L. F., with Barrett, D. J. (2015, July 5). Brain scientist: How Pixar’s ‘Inside Out’ gets one thing deeply wrong. WBUR. wbur.org/news/2015/07/05/brain-scientist-how-pixars-inside-out-gets-one-thing-deeply-wrong ↩︎
Farrell, S. P. (2024, October 31). UC Berkeley professor breaks down the science of ‘Inside Out 2’. UC Berkeley. vcresearch.berkeley.edu/news/uc-berkeley-professor-breaks-down-science-inside-out-2 ↩︎
Barrett, L. F. (2017). The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(1), 1-23. doi:10.1093/scan/nsw154 ↩︎
Montag, C., & Panksepp, J. (2017). Primary emotional systems and personality: An evolutionary perspective. Frontiers in Psychology, 8, 464. doi:10.3389/fpsyg.2017.00464 ↩︎ ↩︎
Tolchinsky, A., Ellis, G. F. R., Levin, M., Kaňková, Š., & Burgdorf, J. (2024). Disgust as a primary emotional system and its clinical relevance. Frontiers in Psychology, 15, 1454774. doi:10.3389/fpsyg.2024.1454774 ↩︎
van der Westhuizen, D., & Solms, M. (2015). Basic emotional foundations of social dominance in relation to Panksepp’s affective taxonomy. Neuropsychoanalysis, 17(1), 19-37. doi:10.1080/15294145.2015.1021371 ↩︎
Jackson, J. C., et al. (2019). Emotion semantics show both cultural variation and universal structure. Science, 366(6472), 1517-1522. doi:10.1126/science.aaw8160 ↩︎
Nisbett, R. E., & Cohen, D. (1996). Culture of honor: The psychology of violence in the South. Westview Press (Routledge reissue, 2018). doi:10.4324/9780429501142 ↩︎
Anderton, E. (2024). Pixar’s Inside Out 2 almost included a complicated German emotion: Here’s why it got cut. /Film. slashfilm.com/1563120/why-pixar-inside-out-2-cut-german-emotion-character-schadenfreude ↩︎
Linnet, J. T. (2011). Money can’t buy me hygge: Danish middle-class consumption, egalitarianism, and the sanctity of inner space. Social Analysis, 55(2), 21-44. doi:10.3167/sa.2011.550202 ↩︎
Phillips, J., Morris, A., & Cushman, F. (2019). How we know what not to think. Trends in Cognitive Sciences, 23(12), 1026-1040. doi:10.1016/j.tics.2019.09.007 ↩︎
Fredrickson, B. L. (2001). The role of positive emotions in positive psychology: The broaden-and-build theory of positive emotions. American Psychologist, 56(3), 218-226. doi:10.1037/0003-066X.56.3.218 ↩︎
Alcaro, A., & Panksepp, J. (2011). The SEEKING mind: Primal neuro-affective substrates for appetitive incentive states and their pathological dynamics in addictions and depression. Neuroscience & Biobehavioral Reviews, 35(9), 1805-1820. doi:10.1016/j.neubiorev.2011.03.002 ↩︎
Tooby, J., & Cosmides, L. (1992). The psychological foundations of culture. In J. H. Barkow, L. Cosmides, & J. Tooby (Eds.), The adapted mind (pp. 19-136). Oxford University Press. cep.ucsb.edu/wp-content/uploads/2023/05/pfc92.pdf ↩︎
Blackmore, S. (1999). The meme machine. Oxford University Press. global.oup.com/academic/product/the-meme-machine-9780192862129 ↩︎
Yudkowsky, E. (2007, November 11). Adaptation-executers, not fitness-maximizers. LessWrong. lesswrong.com/posts/XPErvb8m9FapXCjhA/adaptation-executers-not-fitness-maximizers ↩︎
Gao, G., Taymanov, A., Salinas, E., Mineiro, P., & Misra, D. (2024). Aligning LLM agents by learning latent preference from user edits. arXiv:2404.15269 ↩︎
Masatlioglu, Y., Nakajima, D., & Ozbay, E. Y. (2012). Revealed attention. American Economic Review, 102(5), 2183-2205. doi:10.1257/aer.102.5.2183 ↩︎
Anderson, B. A., Laurent, P. A., & Yantis, S. (2011). Value-driven attentional capture. PNAS, 108(25), 10367-10371. doi:10.1073/pnas.1104047108 ↩︎
Schmidhuber, J. (2009). Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In Anticipatory Behavior in Adaptive Learning Systems (LNCS 5499, pp. 48-76). Springer. arXiv:0812.4360 ↩︎
Poli, F., Serino, G., Mars, R. B., & Hunnius, S. (2020). Infants tailor their attention to maximize learning. Science Advances, 6(39), eabb5053. doi:10.1126/sciadv.abb5053 ↩︎
Awh, E., Belopolsky, A. V., & Theeuwes, J. (2012). Top-down versus bottom-up attentional control: A failed theoretical dichotomy. Trends in Cognitive Sciences, 16(8), 437-443. doi:10.1016/j.tics.2012.06.010 ↩︎
Caplin, A., & Dean, M. (2015). Revealed preference, rational inattention, and costly information acquisition. American Economic Review, 105(7), 2183-2203. doi:10.1257/aer.20140117 ↩︎
Jara-Ettinger, J., Gweon, H., Schulz, L. E., & Tenenbaum, J. B. (2016). The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in Cognitive Sciences, 20(8), 589-604. doi:10.1016/j.tics.2016.05.011 ↩︎
Frederick, S., Loewenstein, G., & O’Donoghue, T. (2002). Time discounting and time preference: A critical review. Journal of Economic Literature, 40(2), 351-401. doi:10.1257/002205102320161311 ↩︎
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363-406. doi:10.1037/0033-295X.100.3.363 ↩︎
Keramati, M., & Gutkin, B. (2014). Homeostatic reinforcement learning for integrating reward collection and physiological stability. eLife, 3, e04811. doi:10.7554/eLife.04811 ↩︎
Solms, M. (2019). The hard problem of consciousness and the free energy principle. Frontiers in Psychology, 9, 2714. doi:10.3389/fpsyg.2018.02714 ↩︎
Baas, M., De Dreu, C. K. W., & Nijstad, B. A. (2011). Creative production by angry people peaks early on, decreases over time, and is relatively unstructured. Journal of Experimental Social Psychology, 47(6), 1107-1115. doi:10.1016/j.jesp.2011.05.009 ↩︎
Schwabe, L., & Wolf, O. T. (2009). Stress prompts habit behavior in humans. Journal of Neuroscience, 29(22), 7191-7198. doi:10.1523/JNEUROSCI.0979-09.2009 ↩︎
Douceur, J. R. (2002). The Sybil attack. In Peer-to-Peer Systems (IPTPS 2002), LNCS 2429, 251-260. Springer. doi:10.1007/3-540-45748-8_24 ↩︎
Armstrong, S., & Mindermann, S. (2018). Occam’s razor is insufficient to infer the preferences of irrational agents. NeurIPS 2018. arXiv:1712.05812 ↩︎
Cao, H., Cohen, S. N., & Szpruch, L. (2021). Identifiability in inverse reinforcement learning. NeurIPS 2021. arXiv:2106.03498 ↩︎
Parr, T., Pezzulo, G., & Friston, K. J. (2022). Active inference: The free energy principle in mind, brain, and behavior. MIT Press (open access). doi:10.7551/mitpress/12441.001.0001 ↩︎
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016). Cooperative inverse reinforcement learning. NeurIPS 2016. arXiv:1606.03137 ↩︎
Frankfurt, H. G. (1971). Freedom of the will and the concept of a person. The Journal of Philosophy, 68(1), 5-20. doi:10.2307/2024717 ↩︎
ToMpathy. Open-source repository, GitHub (EmotiveAutomaton). github.com/EmotiveAutomaton/tompathy ↩︎
Sounding Line. Open-source research repository, GitHub (EmotiveAutomaton). github.com/EmotiveAutomaton/sounding-line ↩︎
Ghost Scale Sim. Open-source simulation repository, GitHub (EmotiveAutomaton). github.com/EmotiveAutomaton/ghost-scale-sim ↩︎
Sounding Line theory documentation (the hypothesis store). GitHub (EmotiveAutomaton). github.com/EmotiveAutomaton/sounding-line/tree/main/docs/theory ↩︎