Big, if true.
EDIT: I got tricked into making this joke when the article had no text, just a headline, so now I look stupider than I really am.
+1 to the content. I've been meaning to write something up about how I think this is the primary source of biases for humans - if you go through the bias sequences, they all seem to be things related to this.
Meta: This post feels like lesswrong "repeating itself" to me. I would have liked it to have a "related posts on less wrong" section, or etc, to avoid repeating discussion.
It seems to me that Kaj is saying that he didn't know this after reading the Sequences. In light of that, it's not repetition. It's very possible that this post still didn't convey the content that he was trying to convey to you, the tacit knowledge drawn from books and life outside of LW. To me, the post seems novel, but also vulnerable to not being taken literally. I feel that if it was taken literally and then exceptions were enumerated, elucidating patterns would appear. In light of that, I'll start in a new post.
This reminds me of Donald Kinder's research that shows people do not vote primarily on self-interest as one might naively expect. It seems that people tend to ask instead "What would someone like me do?" when they vote, with this question likely occurring implicitly.
Ah, crap. If I'd noticed this sooner, I might not have made the post I just made, which is somewhat redundant. But perhaps if anybody wants a more mechanical model of how these things Kaj points out can come to be, they might like reading the post on Goodhart and microhedonics.
Thanks, Kaj.
No, it's an important but counterintuitive topic, and counterintuitive topics need a lot of explanations from many different angles before people really internalize how they actually work. :) Plus, going into more mechanistic detail rather than just doing my vague handwaving-plus-a-million-links is super-valuable. Thank you for writing your post!
I feel like one of the most important lessons I’ve had about How the World Works, which has taken quite a bit of time to sink in, is:
In general, neither organizations nor individual people do the thing that their supposed role says they should do. Rather they tend to do the things that align with their incentives (which may sometimes be economic, but even more often they are social and psychological). If you want to really change things, you have to change people’s incentives.
But I feel like I’ve had to gradually piece this together from a variety of places, over a long time; I’ve never read anything that would have laid down the whole picture. I remember that Freakonomics had a few chapters about how incentives cause unexpected behavior, but that was mostly about economic incentives, which are just a small part of the whole picture. And it didn’t really focus on the “nothing in the world works the way you’d naively expect” thing; as I recall, it was presented more as a curiosity.
On the other hand, Robin Hanson has had a lot of stuff about “X is not about Y“, but that has mostly been framed in terms of prestige and signaling, which is the kind of stuff that’s certainly an important part of the whole picture (the psychological kind of incentives), but again just a part of the picture. (However, his upcoming book goes into a lot more detail on why and how the publicly-stated motives for human or organizational behavior aren’t actually the true motives.)
And then in social/evolutionary/moral psychology there’s a bunch of stuff about social-psychological incentives, of how we’re motivated to denounce outgroups and form bonds with our ingroups; and how it can be socially costly to have accurate beliefs about outgroups and defend them to your ingroup, whereas it would be much more rewarding to just spread inaccuracies or outright lies about how terrible the outgroups are, and thus increase your own social standing. And how even well-meaning ideologies will by default get hijacked by these kinds of dynamics and become something quite different from what they claimed to be.
But again, that’s just one piece of the whole story. And you can find more isolated pieces of the whole story scattered around in a variety of articles and books, also stuff like the iron law of oligarchy, rational irrationality, public choice theory, etc etc. But no grand synthesis.
There’s also a relevant strand of this in the psychology of motivation/procrastination/habit-formation, on why people keep putting off various things that they claim they want to do, but then don’t. And how small things can reshape people’s behavior, like if somebody ends up as a much more healthy eater just because they don’t happen to have a fast food restaurant conveniently near their route home from work. Which isn’t necessarily so much about incentives themselves, but an important building block in understanding why our behavior tends to be so strongly shaped by things that are entirely separate from consciously-set goals.
Additionally, the things that do drive human behavior are often things like maintaining a self-concept, seeking feelings of connection, autonomy and competence, maintaining status, enforcing various moral intuitions, etc., things that only loosely align one’s behavior with one’s stated goals. Often people may not even realize what exactly it is that they are trying to achieve with their behavior.
“Experiental pica” is a misdirected craving for something that doesn’t actually fulfill the need behind the craving. The term originally comes from a condition where people with a mineral deficiency start eating things like ice, which don’t actually help with the deficiency. Recently I’ve been shifting towards the perspective that, to a first approximation, roughly everything that people do is pica for some deeper desire, with that deeper desire being something like social connection, feeling safe and accepted, or having a feeling of autonomy or competence. That is, most of the things that people will give as reasons for why they are doing something will actually miss the mark, and also that many people are engaging in things that are actually relatively inefficient ways of achieving their true desires, such as pursuing career success when the real goal is social connection. (This doesn’t mean that the underlying desire would never be fulfilled, just that it gets fulfilled less often than it would if people were aware of their true desires.)