A discussion with Stefan (cheers, didn't get your email, please message me) during the European Community Weekend Berlin fleshed out an idea I had toyed around with for some time:
If a UFAI can wreak havoc by driving simple goals to extremes then so should driving human desires to extremes cause problems. And we should already see this.
Actually we do.
We know that just following our instincts on eating (sugar, fat) is unhealthy. We know that stimulating our pleasure centers more or less directly (drugs) is dangerous. We know that playing certain games can lead to comparable addiction. And the recognition of this has led to a large number of more or less fine-tuned anti-memes e.g. dieting, early drug prevention, helplines. These memes steering us away from such behaviors were selected for because they provided aggregate benefits to the (members of) social (sub) systems they are present in.
Many of these memes have become so self-evident we don't recognize them as such. Some are essential parts of highly complex social systems. What is the general pattern? Did we catch all the critical cases? Are the existing memes well-suited for the task?How are they related. Many are probably deeply woven into our culture and traditions.
Did we miss any anti-memes?
This last question really is at the core of this post. I think we lack some necessary memes keeping new exploitations of our desires in check. Some new ones result from our society a) having developed the capacity to exploit them and b) the scientific knowledge to know how to do this.
To structure our desires and their exploitation I started with a list which I'd now like to present and ask for discussion and addition to. This list basically takes our complexity of value and looks what happens if one factor is singled out and optimized for too much (where what too much means remains to be discussed).
- Extreme Status
- Extreme Consumption
- Extreme Food Consumption
- Extreme Satisfaction of Desire for Shelter
- Extreme Curiosity
- Generic Information
- Extreme Empathy
- Extreme Hate
- Extreme Fear
- Extreme Love/Romance
- Extreme Sex Drive
- Extreme Awe
- Extreme Pleasure from other Aspects
And before going in detail you might ask yourself how each of these might be driven to extremes - at least compared to what our ancestors would think as described in the Eutopia post.
The craving for power and the reward felt when achieving power must be one of the oldest and has brought about the most complex social structures (we developed from feudal system to democracy after all) and memes to keep it in check. This effectively results in a very high power ladder. The Peter Principle can easily lead to permanent disappointment because the craving is never satisfied despite continuous effort. Nonetheless it appears that there is not much risk here from further exploitation by advancing technology. Or?
Homework question: How can social power systems ensure the feeling of achieving power?
Bringing something into ones possession is rewarding in itself independent of the usefulness of the owned stuff. This can be most easily be seen in the different types of collecting hobbies, messy people and shopping addiction. There is a whole branch of science devoted to the satisfaction of the pleasure from consumption: Neuromarketing. But consumption is well-integrated into our society. Controlling consumption is part of market economy. Fine here it seems.
Extreme Food Consumption
Our evolutionary mostly unchecked desires for sugar and fat and probably a lot of other historically scarce substances has got us - after getting out of starvation - straight into collective obesity. This is well-known. There are lots of anti-memes circulating. E.g. diets. Have a look at Lifestyle interventions for rather helpful memes.
Extreme Satisfaction of Desire for Shelter
This is probably the least of our troubles. Luxury houses probably lead to problems in conjunction with status only.
I will split this into scientific curiosity, games and other.
Curiosity, despite not feeling like an emotion, is nonetheless a strong emotion once basic desires are satisfied. It is what gave us science. Curiosity drives lots of smart people into the sciences. But curiosity alone has no inherent direction. You may arrive at not very useful topics. And then there is a very high knowledge ladder like in the power/status case potentially denying you satisfaction during school and university (and later). But at least this has structured itself in a way to be mostly beneficial to society.
This is another thing. I know game addicts. I don't mean the ones in gambling joints (though this obviously also falls into the category of exploiting desires). I mean WoW and the like. Games are optimized to extract as much pleasure and by extension money out of you as possible. They do so to a large degree by appealing to curiosity, but also status (semi-virtual), ownership (virtual) and probably a lot more. Probably requires a separate post to take this fully apart.
If you are a curios person you can quickly burn lots of time in the knowledge depth of the WWW. This can be in the wake of procrastination but also start out as free time of study. See also this xkcd or this.
This was actually one key point I discussed on the Berlin event. Empathy is an emotion which lets you empathize with beings that are important to you.
"Being-like + important"
That is the pattern your brain locks on (oversimplification; but I hope you basically agree). It doesn't care what kind of being. That's the job of your higher faculties to decide. It doesn't care why that being is important. That's for other emotions to establish.
It's mostly other dear human beings. It can be lifestock. It can be animals. Pets. I could tell you a story of a fly that was the important being in question. For some people even things can become being-like.
Hofstadter proposed levels of consciousness (an illustration of the concept is online here. Sorry can't find a better link) and suggested that you empathize with everything above a certain level. But I disagree. There is no level. It's just your being+important pattern.
Why do I tell you this? Because empathy drives us to form ethical systems. We form ethical systems that (seem to) match our empathy. And then we apply the ethical system. And if the ethical system derives some (unexpected) result we don't question it (very strongly) because after all empathy can't be wrong or can't it? But it should be checked whether empathy can be led astray as easily as any other emotion.
So if you start from a notion that all humans are important beings (because via positive reinforcement you learned that no human beings in your environment are enemies) then you can empathise will all of those and you arrive at an ethical system that treats all humans as equal (even those far away, even in other universes, just because you never encountered negatives).
Please please note that this is descriptive not normative. I'm unclear myself what this exactly means for my ethical system.
If you additionally learn that all animals are important beings because you are exposed to environmentalism, own a personal pet, never experience the historic normality of butchering animals (but e.g. only perceive pictures of this in the mental frame of violence against animals by persons pictured as brutal) than you might arrive at the point where all (higher) animals are important.
This can be raised as needed to include insects and roots (e.g. Jainism) or even plants.
This of course is in contrast to the other end of the spectrum where only members of the same tribe are 'important beings'.
Interestingly this kind of empathy satisfaction exists since many centuries - but mostly in the context of religious systems. The resulting food restrictions could be aligned with food availability. The key point here is that today where the overall system of memes may not (yet) be as balanced as the system at that time (though one might wonder what dynamics existed before e.g. the Jainism stabilized).
I don't really have a good model for this, but I guess that terrorism falls into this area. This is kept mostly in check by existing structures obviously. In reason as memetic immune disorder this is actually given as an example.
Here examples are also harder to come up with, but all cases of over-protection or loss-aversion probably qualify.
This is especially strong regarding children. See this article for a meme trying to counter that and allow a bit of danger for a lot of other utility. For a lot of our fears there exist a whole market of therapy on the one hand and protection devices and services on the other hand.
The fear of death if let run free leads to exploitation by life insurance (which looks beneficial) on the one end to after-life memes (where I'd include cryonics) on the other end.
Now we slowly come to some obvious points. Your desire for relationships and company should be easily exploitable. Especially nowadays as people don't any longer run across each other but are mostly distracted by all the other exploitations of their attention. After all attention is a scarce resource. Now that you are robbed of this natural satisfaction it can again be satisfied with specific services by the dating industry in the form of dating sites and speed dating and so on.
As this is already structuring itself anew and as this is no new problem (in earlier times people probably didn't have much time for dating for other reasons) there should be enough structures to resolve this. Whether this process makes people overall happier is another question.
Extreme Sex Drive
Not much to say here. This desire has been exploited the longest and just because we now have contraceptives, free online porn (which doesn't seem to have detrimental effects), casual sex sites, legalized prostitution (again) and lots of sex toys this probably doesn't alter things a lot. Because it is intimately related to reproduction it migh cause surprising selection pressures though (a separate post suggests itself).
Awe is an emotion. Thus it should be exploitable albeit I'd guess less so than the other. Nonetheless a few areas quick come to mind:
- Awe for things/places: Impressive building e.g. churches for proselytizing, marketing and propaganda. Art for selling art (often coupled with curiosity). And tourism.
- Awe for persons: Just look at popular celebrities and super stars and the fandom surrounding them. This is heavily marketed and the fans follow this blindly not knowing that they have been hooked on one of their emotions.
- Awe for both: Apple
Extreme pleasure from other Aspects
These randomly come to mind:
- Music: Did you notice how many people hear music all the time from their devices?
- Physical Exertion: Now that we aren't any longer regularly exerted from physical work we are prone to have the satisfaction of physical exertion be gamed for in e.g. body building and professional sports.
- Thrill: As we no longer any need for training to cope with threats we instead satisfy this part of curiosity in dangers in amusement parks and possibly extreme sports.
- Collective Identity: I'd think that group-thinking and patriotism has been exploited on the one occasion or other.
Homework: Name two examples occurring in the wake of the internet.
This closes the list of exploitable desires for now.
In this way (scientific) reason is a memetic immune disorder. Coming up with new doable/marketable/organizable ideas dealing with desired in an environment of quick production and communication cycles leads to inhuman structures. Those will mendel themselves out - but maybe we'd like not to wait for that.
We are already living it a Siren World.
The market economy cannot other than make use of this with e.g. Neuromarketing (see also the Forbes article). And the mean thing is: This may only make you want it but not gain pleasure from it. Fact is: There are markets in everything so also in every desire we have.
Markets in this way are worse than an FAI so I have to wonder whether we actually need an FAI to solve the problems we are currently running into...
To avoid a fatalistic end I suggest that knowing about these memes provides you with a toolkit to build your own counter-memes and open the discussion for this.