David Friedman is awesome. I came to the comments to give a different Friedman explanation for one generator of economic rationality from a different Friedman book than "strangepoop" did :-)

In "Law's Order" (which sort of explores how laws that ignore incentives or produce bad incentives tend to be predictably suboptimal) Friedman points out that much of how people decide what to do is based on people finding someone who seems to be "winning" at something and copy them.

(This take is sort of friendly to your "selectionist #3" option but explored in more detail, and applied in more contexts than to simply explain "bad things".)

Friedman doesn't use the term "mimesis", but this is an extremely long-lived academic keyword with many people who have embellished and refined related theories. For example, Peter Thiel has a mild obsession with Rene Girard who was obsessed with a specific theory of mimesis and how it causes human communities to work in predictable ways. If you want the extremely pragmatic layman's version of the basic mimetic theory, it is simply "monkey see, monkey do" :-P

If you adopt mimesis as THE core process which causes human rationality (which it might well not be, but it is interesting to think of a generator of pragmatically correct beliefs in isolation, to see what its weaknesses are and then look for those weaknesses as signatures of the generator in action), it predicts that no new things in the human behavioral range become seriously optimized in a widespread way until AFTER at least one (maybe many) rounds of behavioral mimetic selection on less optimized random human behavioral exploration, where an audience can watch who succeeds and who fails and copy the winners over and over.

The very strong form of this theory (that it is the ONLY thing) is quite bleak and probably false in general, however some locally applied "strong mimesis" theories might be accurate descriptions of how SOME humans select from among various options in SOME parts of real life where optimized behavior is seen but hard to mechanistically explain in other ways.

Friedman pretty much needed to bring up a form of "economic rationality" in his book because a common debating point regarding criminal law in modern times is that incentives have nothing to do with, for example, criminal law, because criminals are mostly not very book smart, and often haven't even looked up (much less remembered) the number of years of punishment that any given crime might carry, and so "can't be affected by such numbers".

(Note the contrast to LW's standard inspirational theorizing about a theoretically derived life plan... around here actively encouraging people to look up numbers before making major life decisions is common.)

Friedman's larger point is that, for example, if burglary is profitable (perhaps punished by a $50 fine, even when the burglar has already sold their loot for $1500), then a child who has an uncle who has figured out this weird/rare trick and makes a living burgling homes will see an uncle who is rich and has a nice life and gives lavish presents at Christmas and donates a lot to the church and is friends with the pastor... That kid will be likely to mimic that uncle without looking up any laws or anything.

Over a long period of time (assuming no change to the laws) the same dynamic in the minds of many children could lead to perhaps 5% of the economy becoming semi-respected burglars, though it would be easy to imagine that another 30% of the private economy would end up focused on mitigating the harms caused by burglary to burglary victims?

(Friedman does not apply the mimesis model to financial crimes, or risky banking practices. However that's definitely something this theory of behavioral causation leads me to think about. Also, advertising seems to me like it might be a situation where harming random strangers in a specific way counts as technically legal, where the perpetration and harm mitigation of the act have both become huge parts of our economy.)

This theory probably under-determines the precise punishments that should be applied for a given crime, but as a heuristic it probably helps constrain punishment sizes to avoid punishments that are hilariously too small. It suggests that any punishment is too small which allow there to exist a "viable life strategy" that includes committing a crime over and over and then treating the punishment as a mere cost of business.

If you sent burglars to prison for "life without parole" on first offenses, mimesis theory predicts that it would put an end to burglary within a generation or four, but the costs of such a policy might well be higher than the benefits.

(Also, as Friedman himself pointed out over and over in various ways, incentives matter! If, hypothetically, burglary and murder are BOTH punished with "life without parole on first offense" AND murdering someone makes you less likely to be caught as a burglar, then murder/burglary is the crime that might be mimetically generated as a pair of crimes that are mimetically viable when only one of them is not viable... If someone was trying to use data science to tune all the punishments to suppress anti-social mimesis, they should really be tuning ALL the punishments and keeping careful and accurate track of the social costs of every anti-social act as part of the larger model.)

In reality, it does seem to me that mimesis is a BIG source of valid and useful rationality for getting along in life, especially for humans who never enter Piaget's "Stage 4" and start applying formal operational reasoning to some things. It works "good enough" a lot of the time that I could imagine it being a core part of any organism's epistemic repertoire?

Indeed, entire cultures seem to exist where the bulk of humans lack formal operational reasoning. For example, anthropologists who study such things often find that traditional farmers (which was basically ALL farmers, prior to the enlightenment) with very clever farming practices don't actually know how or why their farming practices work. They just "do what everyone has always done", and it basically works...

One keyword that offers another path here is one Piaget himself coined: "genetic epistemology". This wasn't meant in the sense of DNA, but rather in the sense of "generative", like "where and how is knowledge generated". I think stage 4 reasoning might be one real kind of generator (see: science and technology), but I think it is not anything like the most common generator, neither among humans nor among other animals.

Connection to Alignment

One of the main arguments in AI risk goes something like:

• AI is likely to be a utility maximizer (or goal-directed in some other sense)
• Goodhart, instrumental convergence, etc make powerful goal-directed agents dangerous by default

One common answer to this is "ok, how about we make AI which isn't goal-directed"?

Unconscious Economics says: selection effects will often create the same effect as goal-directedness, even if we're trying to build a non-goal-directed AI.

Discussions around CAIS are one obvious application. Paul's "you get what you measure" failure-mode is another. A less-obvious application which I've personally run into recently: one strategy to deal with inner optimizers is to design learning algorithms which specifically avoid regions of parameter space in which the trained system will perform optimization. The Unconscious Economics argument says that this won't actually avoid the risk: selection effects from the outer optimizer will push the trained system to misbehave in exactly the same ways, even without an inner optimizer.

Connection to the Economics Literature

During the past year I've found and read a bit more of the formal economics literature related to selection-effect-driven economics.

The most notable work seems to be Nelson and Winter's "An Evolutionary Theory of Economic Change", from 1982. It was a book-length attempt to provide a mathematical foundation for microeconomics grounded in selection effects, rather than assuming utility-maximizing agents from the get-go. Reading through that book, it's pretty clear why the perspective hasn't taken over economics: Nelson and Winter's models are not very good. Some of the larger shortcomings:

• They limit themselves to competition between firms, and their models contain details which limit their generalization to other kinds of agents
• They use a "static" notion of equilibrium (i.e. all agents are individually unchanging), rather than a "dynamic" notion (i.e. distribution of agents is unchanging)
• They seem to lack the mathematical skills to prove properties of reasonably general models; instead they rely heavily on simulation

I do not see any of these problems as substantial barriers to a selection-based theory; it's just that Nelson and Winter did not have the mathematical chops to make it happen, and nobody better seems to have come along since.

I think it's a common notion that if you were just good enough you wouldn't respond to incentives. I used to think it more myself. It's a key element to realize that the system will still create these outcomes even if nobody is consciously choosing to fall prey to them, and that to 'avoid incentives' in a system like that you'd need to actually model what the incentives are and what outcome they're systematically choosing, via selection effects, by correctly optimizing according to feedback in a complex domain, and so on. One cannot have model-free integrity.

So it feels to me like a very fundamental insight, written up well. (It's also related I think to some of the discussion about lying vs unconscious bias between Scott and Zack and Jessica and so on.)

Here’s an insight I had about how incentives work in practice, that I’ve not seen explained in an econ textbook/course.

There are at least three ways in which incentives affect behaviour: 1) via consciously motivating agents, 2) via unconsciously reinforcing certain behaviour, and 3) via selection effects. I think perhaps 2) and probably 3) are more important, but much less talked about.

Jon Elster distinguishes these three different ways in Explaining Social Behavior. He first draws a distinction between 1-2 ("reinforcement") on the one hand, and 3 ("selection"), on the other. He then draws a further distinction between 1 ("conscious rational choice") and 2 ("unintentional choice"). Here are the relevant excerpts from ch. 11 (emphasis in the original; I have added numbers in square brackets to make the correspondence between your distinctions and his more conspicuous):

In this chapter, I discuss explanations of actions in terms of their objective consequences... There are two main ways in which this can happen: by reinforcement [1-2] and by selection [3]... If the consequences of given behavior are pleasant or rewarding, we tend to engage in it more often; if they are unpleasant or punishing it will occur less often. The underlying mechanism could simply be conscious rational choice [1], if we notice the pleasant or unpleasant consequences and decide to act in the future so as to repeat or avoid repeating the experience. Often, however, the reinforcement can happen without intentional choice [2].

In order to apply economic reasoning in the real world, this is an indispensable concept, and this post is my go-to link for it.

Author here: I think this post could use a bunch of improvements. It spends a bunch of time on tangential things (e.g. the discussion of Inadequacy and why this doesn't come through in textbooks, spending a while initially setting up a view to then tear down). 

But really what would be nice is to have it do a much better job at delivering the core insight. This is currently just done in two bullets + one exercise for the reader. 

Even more important would be to include JenniferRM's comment which adds a core mechanism (something like "cultural learning").

Overall, though, I still stand by the importance of the underlying concept; and think it's a crucial part of the toolkit required to apply economic thinking in practice.   

Just stumbled on an old econ paper specifically about this topic: Alchian's "Uncertainty, Evolution and Economic Theory". I came to it via this essay on Alchian, which mentions that a lot of Alchian's material was just standard thinking from the pre-WWII era which was forgotten by the more macro-focused Keynesian economics paradigm (which dominated from roughly WWII until the Lucas Critique). Through that lens, it's kind of surprising that people haven't revisited economic selection pressure much in the context of modern microfoundations - most current work on microfoundations still assumes rational utility-maximizing agents. It wouldn't surprise me if there's some interesting theory to be discovered there, although it would probably require building a very different mathematical foundation from what's currently used.

Interesting. I had always assumed that the focus in economics was on 1 because 2 and 3 aren't particularly controllable; it is very easy to say "I want X, and will pay $Y" but it is much harder to impose an unconscious reinforcement regime or a selection effect.

'Unconscious' is also a way I would describe some aspects of learning, which is an avenue into behavior effects, even without being noticed, or noticed much.

I have a different hypothesis for the "people aren't like that!" response. It's about signalling high status in order to be given high status. If I claim that "people aren't bad where I come from", it signals that I'm somehow not used to being treated badly, which is evidence that I'm not treated badly, which is evidence that mechanisms for preventing bad behavior are already in place.

This isn't just a random idea, this is introspectively the reason that I keep insisting that people really aren't bad. It's a sermon. An invitation to good people and a threat to bad ones.

The one who gets bullied is the one that openly behaves like they're already being bullied.