Related: Not for the Sake of Happiness (Alone), Value is Fragile, Fake Fake Utility Functions, You cannot be mistaken about (not) wanting to wirehead, Utilons vs. Hedons, Are wireheads happy?
When someone tells me that all human action is motivated by the desire for pleasure, or that we can solve the Friendly AI problem by programming a machine superintelligence to maximize pleasure, I use a two-step argument to persuade them that things are more complicated than that.
First, I present them with a variation on Nozick's experience machine,1 something like this:
Suppose that an advanced team of neuroscientists and computer scientists could hook your brain up to a machine that gave you maximal, beyond-orgasmic pleasure for the rest of an abnormally long life. Then they will blast you and the pleasure machine into deep space at near light-speed so that you could never be interfered with. Would you let them do this for you?
Most people say they wouldn't choose the pleasure machine. They begin to realize that even though they usually experience pleasure when they get what they desired, they want more than just pleasure. They also want to visit Costa Rica and have good sex and help their loved ones succeed.
But we can be mistaken when inferring our desires from such intuitions, so I follow this up with some neuroscience.
Wanting and liking
It turns out that the neural pathways for 'wanting' and 'liking' are separate, but overlap quite a bit. This explains why we usually experience pleasure when we get what we want, and thus are tempted to think that all we desire is pleasure. It also explains why we sometimes don't experience pleasure when we get what we want, and why we wouldn't plug in to the pleasure machine.
How do we know this? We now have objective measures of wanting and liking (desire and pleasure), and these processes do not always occur together.
One objective measure of liking is 'liking expressions.' Human infants, primates, and rats exhibit homologous facial reactions to pleasant and unpleasant tastes.2 For example, both rats and human infants display rhythmic lip-licking movements when presented with sugary water, and both rats and human infants display a gaping reaction and mouth-wipes when presented with bitter water.3
Moreover, these animal liking expressions change in ways analogous to changes in human subjective pleasure. Food is more pleasurable to us when we are hungry, and sweet tastes elicit more liking expressions in rats when they are hungry than when they are full.4 Similarly, both rats and humans respond to intense doses of salt (more concentrated than in seawater) with mouth gapes and other aversive reactions, and humans report subjective displeasure. But if humans or rats are depleted of salt, both humans and rats react instead with liking expressions (lip-licking), and humans report subjective pleasure.5
Luckily, these liking and disliking expressions share a common evolutionary history, and use the same brain structures in rats, primates, and humans. Thus, fMRI scans have uncovered to some degree the neural correlates of pleasure, giving us another objective measure of pleasure.6
As for wanting, research has revealed that dopamine is necessary for wanting but not for liking, and that dopamine largely causes wanting.7
Now we are ready to explain how we know that we do not desire pleasure alone.
First, one can experience pleasure even if dopamine-generating structures have been destroyed or depleted.8 Chocolate milk still tastes just as pleasurable despite the severe reduction of dopamine neurons in patients suffering from Parkinson's disease,9 and the pleasure of amphetamine and cocaine persists throughout the use of dopamine-blocking drugs or dietary-induced dopamine depletion — even while these same treatments do suppress the wanting of amphetamine and cocaine.10
Second, elevation of dopamine causes an increase in wanting, but does not cause an increase in liking (when the goal is obtained). For example, mice with raised dopamine levels work harder and resist distractions more (compared to mice with normal dopamine levels) to obtain sweet food rewards, but they don't exhibit stronger liking reactions when they obtain the rewards.11 In humans, drug-induced dopamine increases correlate well with subjective ratings of 'wanting' to take more of the drug, but not with ratings of 'liking' that drug.12 In these cases, it becomes clear that we want some things besides the pleasure that usually results when we get what we want.
Indeed, it appears that mammals can come to want something that they have never before experienced pleasure when getting. In one study,13 researchers observed the neural correlates of wanting while feeding rats intense doses of salt during their very first time in a state of salt-depletion. That is, the rats had never before experienced intense doses of salt as pleasurable (because they had never been salt-depleted before), and yet they wanted salt the very first time they encountered it in a salt-depleted state.
But why are liking and wanting so commingled that we might confuse the two, or think that the only thing we desire is pleasure? It may be because the two different signals are literally commingled on the same neurons. Resarchers explain:
Multiplexed signals commingle in a manner akin to how wire and optical communication systems carry telephone or computer data signals from multiple telephone conversations, email communications, and internet web traffic over a single wire. Just as the different signals can be resolved at their destination by receivers that decode appropriately, we believe that multiple reward signals [liking, wanting, and learning] can be packed into the activity of single ventral pallidal neurons in much the same way, for potential unpacking downstream.
......we have observed a single neuron to encode all three signals... at various moments or in different ways (Smith et al., 2007; Tindell et al., 2005).14
In the last decade, neuroscience has confirmed what intuition could only suggest: that we desire more than pleasure. We act not for the sake of pleasure alone. We cannot solve the Friendly AI problem just by programming an AI to maximize pleasure.
1 Nozick (1974), pp. 44-45.
2 Steiner (1973); Steiner et al (2001).
3 Grill & Berridge (1985); Grill & Norgren (1978).
4 Berridge (2000).
5 Berridge et al. (1984); Schulkin (1991); Tindell et al. (2006).
6 Berridge (2009).
7 Berridge (2007); Robinson & Berridge (2003).
8 Berridge & Robinson (1998); Berridge et al. (1989); Pecina et al. (1997).
9 Sienkiewicz-Jarosz et al. (2005).
10 Brauer et al. (2001); Brauer & de Wit (1997); Leyton (2009); Leyton et al. (2005).
11 Cagniard et al. (2006); Pecina et al. (2003); Tindell et al. (2005); Wyvell & Berridge (2000).
12 Evans et al. (2006); Leyton et al. (2002).
13 Tindell et al. (2009).
13 Aldridge & Berridge (2009). See Smith et al. (2011) for more recent details on commingling.
Aldridge & Berridge (2009). Neural coding of pleasure: 'rose-tinted glasses' of the ventral pallidum. In Kringelbach & Berridge (eds.), Pleasures of the brain (pp. 62-73). Oxford University Press.
Berridge (2000). Measuring hedonic impact in animals and infants: Microstructure of affective taste reactivity patterns. Neuroscience and Biobehavioral Reviews, 24: 173-198.
Berridge (2007). The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology, 191: 391-431.
Berridge (2009). ‘Liking’ and ‘wanting’ food rewards: Brain substrates and roles in eating disorders. Physiology & Behavior, 97: 537-550.
Berridge, Flynn, Schulkin, & Grill (1984). Sodium depletion enhances salt palatability in rats. Behavioral Neuroscience, 98: 652-660.
Berridge, Venier, & Robinson (1989). Taste reactivity analysis of 6-hydroxydopamine-induced aphagia: Implications for arousal and anhedonia hypotheses of dopamine function. Behavioral Neuroscience, 103: 36-45.
Berridge & Robinson (1998). What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience? Brain Research Reviews, 28: 309-369.
Brauer, Cramblett, Paxton, & Rose (2001). Haloperidol reduces smoking of both nicotine-containing and denicotinized cigarettes. Psychopharmacology, 159: 31-37.
Brauer & de Wit (1997). High dose pimozide does not block amphetamine-induced euphoria in normal volunteers. Pharmacology Biochemistry & Behavior, 56: 265-272.
Cagniard, Beeler, Britt, McGehee, Marinelli, & Zhuang (2006). Dopamine scales performance in the absence of new learning. Neuron, 51: 541-547.
Evans, Pavese, Lawrence, Tai, Appel, Doder, Brooks, Lees, & Piccini (2006). Compulsive drug use linked to sensitized ventral striatal dopamine transmission. Annals of Neurology, 59: 852-858.
Grill & Berridge (1985). Taste reactivity as a measure of the neural control of palatability. In Epstein & Sprague (eds.), Progress in Psychobiology and Physiological Psychology, Vol 2 (pp. 1-6). Academic Press.
Grill & Norgren (1978). The taste reactivity test II: Mimetic responses to gustatory stimuli in chronic thalamic and chronic decerebrate rats. Brain Research, 143: 263-279.
Leyton, Boileau, Benkelfat, Diksic, Baker, & Dagher (2002). Amphetamine-induced increases in extracellular dopamine, drug wanting, and novelty seeking: a PET/[11C]raclopride study in healthy men. Neuropsychopharmacology, 27: 1027-1035.
Leyton, Casey, Delaney, Kolivakis, & Benkelfat (2005). Cocaine craving, euphoria, and self-administration: a preliminary study of the effect of catecholamine precursor depletion. Behavioral Neuroscience, 119: 1619-1627.
Leyton (2009). The neurobiology of desire: Dopamine and the regulation of mood and motivational states in humans. In Kringelbach & Berridge (eds.), Pleasures of the brain (pp. 222-243). Oxford University Press.
Nozick (1974). Anarchy, State, and Utopia. Basic Books.
Pecina, Berridge, & Parker (1997). Pimozide does not shift palatibility: Separation of anhedonia from sensorimotor suppression by taste reactivity.Pharmacology Biochemistry and Behavior, 58: 801-811.
Pecina, Cagniard, Berridge, Aldridge, & Zhuang (2003). Hyperdopaminergic mutant mice have higher 'wanting' but not 'liking' for sweet rewards. The Journal of Neuroscience, 23: 9395-9402.
Robinson & Berridge (2003). Addiction. Annual Review of Psychology, 54: 25-53.
Schulkin (1991). Sodium Hunger: the Search for a Salty Taste. Cambridge University Press.
Sienkiewicz-Jarosz, Scinska, Kuran, Ryglewicz, Rogowski, Wrobel, Korkosz, Kukwa, Kostowski, & Bienkowski (2005). Taste responses in patients with Parkinson's disease. Journal of Neurology, Neurosurgery, & Psychiatry, 76: 40-46.
Smith, Berridge, & Aldridge (2007). Ventral pallidal neurons distinguish 'liking' and 'wanting' elevations caused by opioids versus dopamine in nucleus acumbens. Program No. 310.5, 2007 Neuroscience Meeting Planner. San Diego, CA: Society for Neuroscience.
Smith, Berridge, & Aldridge (2011). Disentangling pleasure from incentive salience and learning signals in brain reward circuitry. Proceedings of the National Academy of Sciences PNAS Plus, 108: 1-10.
Steiner (1973). The gustofacial response: Observation on normal and anecephalic newborn infants. Symposium on Oral Sensation and Perception, 4: 254-278.
Steiner, Glaser, Hawillo, & Berridge (2001). Comparative expression of hedonic impact: affective reactions to taste by human infants and other primates.Neuroscience and Biobehavioral Reviews, 25: 53-74.
Tindell, Berridge, Zhang, Pecina, & Aldridge (2005). Ventral pallidal neurons code incentive motivation: Amplification by mesolimbic sensitization and amphetamine. European Journal of Neuroscience, 22: 2617-2634.
Tindell, Smith, Pecina, Berridge, & Aldridge (2006). Ventral pallidum firing codes hedonic reward: When a bad taste turns good. Journal of Neurophysiology, 96: 2399-2409.
Tindell, Smith, Berridge, & Aldridge (2009). Dynamic computation of incentive salience: 'wanting' what was never 'liked'. The Journal of Neuroscience, 29: 12220-12228.
Wyvell & Berridge (2000). Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward: Enhancement of reward 'wanting' without enhanced 'liking' or response reinforcement. Journal of Neuroscience, 20: 8122-8130.
Possibly because the word "machine" is sneaking in connotations that lead to the observed conclusion: we picture something like a morphine pump, or something perhaps only slightly less primitive.
What if we interpret "machine" to mean "a very large computer running a polis under a fun-theoretically optimal set of rules" and "hook up your brain" to mean "upload"?
Both you and Eliezer seem to be replying to this argument:
People only intrinsically desire pleasure.
An FAI should maximize whatever people intrinsically desire.
Therefore, an FAI should maximize pleasure.
I am convinced that this argument fails for the reasons you cite. But who is making that argument? Is this supposed to be the best argument for hedonistic utilitarianism?
I wonder how much taking these facts into account helps. The error that gets people round up to simplistic goals such as "maximize pleasure" could just be replayed at a more sophisticated level, where they'd say "maximize neural correlates of wanting" or something like that, and move to the next simplest thing that their current understanding of neuroscience doesn't authoritatively forbid.
Sure. And then I write a separate post to deal with that one. :)
There are also more general debunkings of all such 'simple algorithm for friendly ai' proposals, but I think it helps to give very concrete examples of how particular proposed solutions fail.
Actually, they claim to also want those other things. Your post strongly implies that this claim is justified, when it seems much more plausible to me to just assume they instead also want a dopamine hit. In other words, to properly wirehead them, Nozick's machine would not just stimulate one part (liking), but two (+wanting) or three (+learning).
So I don't really see the point of this article. If I take it at face-value (FAI won't just optimize the liking response), then it's true, but obvious. However, the subtext seems to be that FAI will have to care about a whole bunch of values and desires (like travels, sex, companionship, status and so on), but that doesn't follow from your arguments.
It seems to me that you are forcing your conclusions in a certain direction that you so far haven't justified. Maybe that's just because this is an early post and you'll still get to the real meat (I assume that is the case and will wait), but I'm uncomfortable with the way it looks right now.
I don't understand what everything after the Nozick's experience machine scenario is for. That is, the rest of the post doesn't seem to support the idea that we intrinsically value things other than pleasure. It tells us why sometimes what we think will give us pleasure is wrong (wanting more of the drug), and why some things give us pleasure even though we didn't know they would (salt when salt-deprived)... but none of this means that pleasure isn't our objective.
Once we know about salt, we'll want it for the pleasure. We would also normally stop wanting something that turns out not to give us pleasure, as I stopped wanting to eat several chocolate bars at a time. It might not work like this in the drug example, though, that's true; but does this mean that there's something about the drug experience we desire besides the pleasure, or are our brains being "fooled" by the dopamine?
I think the latter. There is no aspect wanted, just the wanting itself.... possibly because dopamine is usually associated with - supposed to signal - pleasure?
(I think once anyone got in your Nozick's machine variation, they would change their minds and not want to get out. We think we'd exper... (read more)
I'd get in Nozick's machine for the wireheading. I figure it's likely enough that I'm in a simulation anyway, and his simulation can be better than my current one. I figure I'm atypical though.
I got an experience machine in the basement that supplies you with loads and loads of that marvelously distinct feeling of "the process of painting the picture" and "seeing how your decisions and plans are getting implemented in a semi-physical world around you". Your actions will have a perfectly accurate impact on your surroundings and you will have loads of that feeling of control and importance that you presumably believe is so important for your happiness.
Hey Lukeprog, thanks for your article.
I take it you have read the new book "Pleasures of the Brain" by Kringelbach & Berridge? I've got the book here but haven't yet had the time/urge to read it bookend to bookend. From what I've glimpsed while superficially thumbing through it however, it's basically your article in book-format. Although I believe I remember that they also give "learning" the same attention as they give to liking and wanting, as one of your last quotations hints at.
It's quite a fascinating thought, that the "virtue" of curiosity which some people display is simply because they get a major kick out of learning new things - as I suspect most people here do.
Anyway, I've never quite bought the experience machine argument for two reasons:
1) As we probably all know: what people say they WOULD DO is next to worthless. Look at the hypnotic pull of World of Warcraft. It's easy to resist for me, but there may very well be possible virtual realities, that strike my tastes in such an irresistible manner that my willpower will be powerless and I'd prefer the virtual reality over this one. I may feel quite guilty about it, but it may be so h... (read more)
Also related: Are Wireheads Happy?, You cannot be mistaken about (not) wanting to wirehead.
This sounds to me like a word game. It depends on what the initial intention for 'pleasure' is. If you say the device gives 'maximal pleasure' meaning to point to a cloud of good-stuffs and then you later use a more precise meaning for pleasure that is an incomplete model of the good-stuffs, you are then talking of different things.
The meaningful thought experiment for me is whether I would use a box that maximized pleasure\wanting\desire\happiness\whatever-is-going-on-at-the-best-moments-of-life while completely separating me as an actor or participant fr... (read more)
We have an experience machine at our disposal if we could figure out the API. Ever have a lucid dream?
Duh - otherwise sexual reproduction in mammals would be a non-starter.
When people say "pleasure" in this context, they usually just mean to refer to whatever it is that the human brain likes internally. To then say that people don't just like pleasure - since they also like happiness/bliss - or whatever - seems to be rather missing the point.
As for the rather distinct claim that people want external success, not just internal hedonistic pleasure signals - that seems to depend on the person under consideration. Few want their pleasure to end with them being fired, and running out of drug money (though we do still ... (read more)
Disclaimer: I don't think that maximizing pleasure is an FAI solution; however, I didn't find your arguments against it convincing.
With regards to the experience machine, further study has found that people's responses are generally due to status quo bias; a more recent study found that a slight majority of people would prefer to remain in the simulation.
With regards to the distinction between desire and pleasure: well, yes, but you seem to be just assuming that our desires are what ought to be satisfied/maximized instead of pleasure; I would assume that m... (read more)
While the article shows with neat scientific references that it is possible to want something that we don't end up liking, this is irrelevant to the problem of value in ethics, or in AI. You could as well say without any scientific studies that a child may want to put their hand in the fire and end up not liking the experience. It is well possible to want something by mistake. But it is not possible to like something by mistake, as far as I know. Differently from wanting, "liking" is valuable in itself.
Wanting is a bad thing according to Epicurus... (read more)
Intuitively, this feels true. I rarely do things based on how much pleasure they bring me. Some of my decisions are indirectly linked to future pleasure, or other people's pleasure, i.e. choosing to work 6 am shifts instead of sleeping in because then I won't be poor, or doing things I don't really want to but said I would do because other people are relying on me and their plans will be messed up if I don't do it, and I wouldn't want them to do that to me... Actually, when I think about it, an awful lot of my actions have more to do with other people's pleasure than with my own, something which the pleasure machine doesn't fulfill. In fact, I would worry that a pleasure machine would distract me from helping others.
I feel like I am missing something. You separated pleasure from wanting.
I don't see how this backs up your point though. Unless the machine offered is a desire-fulfilling machine and not a pleasure machine.
If it is a pleasure machine, giving pleasure regardless of the state of wanting, why would we turn it down? You said we usually want more than just pleasure, because getting what we want doesn't always give us pleasure. If wanting and pleasure are different, then of course this makes sense.
But saying we want more than pleasure? That doesn't make sense. Y... (read more)
Maybe you're hyperbolically discounting that future pleasure and it's outweighed by the temporary displeasure caused by agreeing to something abhorrent? ;)
Oddly, those two arguments end up cancelling out for me.
You explained how pleasure from our natural environment "caps out" past a certain threshold - I can't eat infinity sugar and derive infinity pleasure. So, obviously, my instinctive evaluation is that if I get wire-headed, I'll eventually get sick of it and want something else!
Equally, your research shows that we're not always perfect at evaluating what I want. Therefore, I'd have an instinctive aversion to wire-heading because I might have guessed wrong, and it's obviously very difficult to ... (read more)
BBC article: "The science of addiction: Do you always like the things you want?", Edmonds.
IAWYC, but would like to hear more about why you think the last sentence is supported by the previous sentence. I don't see an easy argument from "X is a terminal value for many people" to "X should be promoted by the FAI." Are you supposing a sort of idealized desire fulfilment view about value? That's fine--it's a sensible enough view. I just wouldn't have thought it so obvious that it would be a good idea to go around invisibly assuming it.
Is their meaningful data on thalamic stimulators with erodic side-effects? (See entry #1 here: http://www.cracked.com/blog/5-disturbing-ways-the-human-body-will-evolve-in-the-future/ ). Cracked gives the addictive potential of an accidental orgasm switch plenty of attention while citing just two examples (it's a comedy site after all), but have other cases been studied? I'm not convinced this couldn't be done intentionally with current models of the brain.
Well that was easy. In my (limited) experience most people making such claims do not really anticipate being pleasure-maximized, and thus can claim to want this without problems. It's only "real" ways of maximizing pleasure that they care about, so you need to find a "real" counterexample.
That said, I have less experience with such people than you, I guess, so I may be atypical in this regard.
I notice that I'm a bit confused, especially when reading, "programming a machine superintelligence to maximize pleasure." What would this mean?
It also seems like some arguments are going on in the comments about the definition of "like", "pleasure", "desire" etc. I'm tempted ask everyone to pull out the taboo game on these words here.
A helpful direction I see this article pointing toward, though, is how we personally evaluate an AI's behavior. Of course, by no means does an AI have to mimic human internal workings 1... (read more)
I have a different argument for why people don't want to wirehead themselves. It's easiest to think about if you imagine a self-modifying AI. Let's suppose that this particular AI is a paperclip maximizer. Its utility function is equal to the number of paperclips in the world.
Now, this AI is self-modifying, so of course it can wirehead itself. In other words it can replace its utility function of "U = number of paperclips" with a different utility function "U = VERYLARGENUMBER". Suppose the AI is considering whether to make this change.... (read more)
I've thought of a bit of intuition here, maybe someone will benefit by it or be kind enough to critique it;
Say you took two (sufficiently identical) copies of that person C1 and C2, and exposed C1 to the wirehead situation (by plugging them in) and showed C2 what was happening to C1.
It seems likely that C1 would want to remain in the situation and C2 would want to remove C1 from the wirehead device. This seems to be the case even if the wirehead machine doesn't raise dopamine levels very much and thus the user does not become dependent on it.
However, even ... (read more)
How about a machine that maximizes your own concept of pleasure and makes you believe that it is probably not a machine simulation (or thinks that machine simulation is an irrelevant argument)?
Going for what you "want" is merely going for what you like the thought of. To like the thought of something is to like something (in this case the "something" that you like is the thought of something; a thought is also something). This means that wanting cannot happen unless there is liking that creates the wanting. So, of wanting and liking, liking is the only thing that can ever independently make us make any choice we make. Wanting which is not entirely contingent on liking never makes us make any decisions, because there is no suc... (read more)
Either this conclusion contradicts the whole point of the article, or I don't understand what is meant by the various terms "desire", "want", "pleasure", etc. If pleasure is "that which we like", then yes we can solve FAI by programming an AI to maximize pleasure.
The mis... (read more)
Well, no wonder. The way the hypothetical scenario is presented evokes a giant array of ways it could go wrong.
What if the pleasure machine doesn't work? What if it fails after a month? What if it works for 100 years, followed by 1,000 years of loneliness and suffering?
Staying on Earth sounds a lot safer.
Suppose the person you are ask... (read more)
The pleasure machine argument is flawed for a number of reasons:
1) It assumes that, despite having never been inside the pleasure machine, but having lots of experience of the world outside of it, you could make an unbiased decision about whether to enter the pleasure machine or not. It's like asking someone if he would move all his money from a bank he knows a lot about to a bank he knows basically nothing about and that is merely claimed to make him richer than his current bank. I'm sure that if someone would build a machine that, after I stepped into it... (read more)
While humans may not be maximizing pleasure they are certainly maximizing some utility function which can be characterized. Human concerns can then be programmed to optimize this function in your FAI.