This is a linkpost for equilibria.club
Diversity and complexity - Scott E. Page.
It is time for the first book review on Equilibria Club! The book is filled to the brim with models and ideas about diversity and complexity, each of which could normally deserve their own blog post. I will try to highlight some of the most valuable insights, especially those using economic and political examples, while still leaving enough of a cliffhanger for the interested readers.
The author's previous books, of which one is about diversity and one about complexity, may have allowed him to see the interesting relations between the two topics.
Throughout the book, various technical definitions of diversity and complexity are used. Each definition or measure captures a different aspect of our understanding. Ultimately, the author mentions various benefits which diversity offers.
The first major insight of the book is that diversity can both create and stabilize a market. The first example is both fun and enlightning - at least to my economic theorist mind:
"Imagine an exchange market—a bazaar in which people bring wheelbarrows of goods to trade. This example demonstrates how diversity can reduce volatility in a system and also produce complexity. In an exchange market, diversity can enter in three ways: (1) in what the agents bring to buy and sell, their endowments; (2) in the agents’ preferences for the different goods; and (3) in the ways the agents adapt to information, specifically prices."
"If the market had no diversity, not much would happen. If everyone had identical endowments and preferences, then no one would have any reason to trade. So, we need diversity on at least one of these dimensions just to make the market come to life. Let’s add diversity to both endowments and preferences so that agents bring different goods to market and desire different bundles of goods as well. In such a market, we need some mechanism for prices to form. Following standard economics, let’s assume that there exists a market maker, who calls out prices with the intent of producing equilibrium trades."
"Once we introduce the market maker, we have to take into account how agents respond to prices. Let’s start by assuming no diversity. If all of the agents react in the same way, then prices will be volatile. They’ll jump all over the place. This volatility results from everyone reacting in the same way to a price that’s too low, resulting in a massive increase in demand and a similar rise in price. Gintis (2007) shows that diversity in the learning rules reduces this volatility."
A similar logic plays out every day in the tiny world of bees:
"They want to maintain a comfortable temperature in the hive. Bees have an internal mechanism that determines when the hive is too hot or too cold. When it’s too hot, they fan out. When it’s too cool, they huddle together. A hive of genetically identical bees will all get hot and cold at the same temperature. [...] If the bees have different temperatures at which they get hot and cold, that is, if they have variance in their temperature thresholds, then these fluctuations become less severe."
Pretty cool huh? However, both of the above example make use of negative feedback loops; in other words: increased participation by the market players or the bees leads to diminishing returns. Other models which are built on positive feedback loops actually show that variation can introduce chaotic behavior:
"With positive feedbacks, the opposite occurs: variation in thresholds leads to an increase in the probability of large events. This observation can be made more formal with Granovetter’s (1978) riot model."
Besides influencing stability, diversity (when defined as a property displaying high variance) can help to increase robustness or fitness. While stability is about a system staying in its place, robustness is about a system being able to adapt to a variety of circumstances.
Fisher’s theory of natural selection relates the rate of increase in fitness of an organism to its genetic variance. In simple terms: the more variance a particular population has, the more opportunities exists for organisms to adapt. By using the "Price equation" in which we ignore mutations, "Fisher's fundamental theorem" can be derived which states that the change in average fitness equals variance of fitness:
For a more thorough derivation of these formula's, I recommend you to read the book! A final note on the usefulness of variation:
"The idea that fitness increases with variance proves useful as a departure point for thinking about variation as a form of search. This logic can be made more formal (Weitzman 1979). [...] as the number of searches increases, what determines the value of the best solution found is weight in the upper tail of the distribution."
A new categorization to add to your mental model: fundamental versus emergent diversity! This distinction reminded me a bit of the difference between essential and accidental complexity introduced in the classic 1986 essay No Silver Bullet by Fred Brooks; which has some theoretical basis but seems useless in practice. Let's have a look at the types of diversity and complexity by looking at the Game of Life:
"The rules for the Game of Life are deceptively simple. A dead agent comes to life if it has exactly three live neighbors; otherwise it stays dead. A live agent remains alive if and only if it has two or three live neighbors. Otherwise, it dies, either of boredom (fewer than two live neighbors) or of suffocation (more than three live neighbors). The Game of Life can produce complex patterns including blinkers that flip back and forth, gliders that float across the torus, and even pulsing glider guns that spit out gliders at regular intervals."
"If the Game of Life doesn’t include much diversity, how can it produce complexity? Two answers: large numbers of parts and interdependence that produces emergent structures. First, the large number answer: a long string of zeros and ones proves a sufficiently rich space to support complexity, just as a long string of DNA can contain the instructions for life. Complexity requires little diversity in the parts, provided there are enough parts. With enough zeros and ones it’s possible to say anything. Second, the parts are interdependent. The sum, the component consisting of the parts, can be more complex than the parts themselves. What is a brain but a collection of spatially situated simple parts that interact according to rules? In the brain, the rules depend on chemistry and physics, whereas in the Game of Life, the rules depend on logic, but in both cases simple parts produce complexity."
"Here, then, is the take away: fundamental diversity is not required for complexity. Emergent diversity is. The Game of Life produces complexity through the interactions of diverse interacting parts, but those parts are not the cells. The relevant parts are the emergent structures, like the gliders. These exist on a higher level."
Given that reality has a surprising amount of detail, which interacts in unpredictable ways, I suspect that we can assume accidental complexity and emergent diversity to be the main determinants which we should take into account in our models.
In the first point above, we saw that diversity can have a different impact in the presence of positive versus negative feedback loops.
We already know that diversity increases robustness in a changing environment, but there is also a relation between amount of change in the environment and amount of useful diversity.
"This insight—that the level of variation should track the rate of disturbances—leads to the question: can an entity within a complex system locate this optimal level of variation? Yes. In fact, it’s relatively easy. To see how, we can return to our landscape model and think of the level of variation as the feature that adapts. Given a rate of disturbance, there exists an optimal level of variation. Except in rare cases, the closer the level of variation is to the optimal level, the better the population will perform on average. Therefore, the “variation landscape” isn’t rugged. It’s single-peaked, and easily scaled."
"In working through this logic, I’ve taken the rate of disturbances as exogenous—as occurring outside the system. In complex systems, the rate of disturbance to a landscape would be endogenous—it would depend on how fast other entities adapt and respond. Therefore, it would also depend on the levels of variation in other species. Whether levels of variation settle down or vary over time depends on the complex system and the path it takes. In either case, what’s important to keep in mind is that for any type of entity, the appropriate level of variation will eventually emerge from the system. Moreover, that level will tend to track the rate at which the system churns."
Moreover:
"Variation can also act as a signal in complex systems. Consider an ecological system that is undergoing a phase transition, such as a lake becoming eutrophic or a grassland moving toward desertification. During the phase transition, the fitness landscape for species will shift. That shift in the landscape may transform what was a peak into a flat spot on the fitness landscape. This implies the potential for an increase in variation prior to a major change in the system."
Various chapters in the book start with 'Diversity's inescapable benefits', which makes me wonder how many arguments were collected because of a happy death spiral or mood affiliation. Mind you, most of the book is actually very nuanced; describing neutral or context-dependent impacts of diversity. The book even starts with clear examples of how hard it can be to infer relationships about diversity:
"Assembly implies that the level of diversity in a system has survived some winnowing process. This winnowing creates three problems for empirical tests of the effects of diversity. I refer to these as the problem of multiple causes, the sample problem, and selection (squared) bias. I cover each in turn."
This still doesn't warrant for suddenly calling the benefits 'inescapable'. To give one example, the author mentions "diminishing returns to type" but fails to mention its big brother "increasing returns to scale", which are both valid and widespread phenomena.
Moreover diversity is just not a very hard and well-defined concept in science (yet):
"Moving back to the more general discussion of what is a type, it is probably not too much of a reach to say that the definition of types depends upon the question being asked. A candy store that sells thousands of types of candy can be thought of as selling either one type of good—candy—or, if we distinguish among the many brands and varieties, thousands of types. Whether we differentiate between a Clark Bar and a Butterfinger depends on whether we’re interested in the diversity of individual choices (in which case we do) or how economic diversity drives macro-economic growth (in which case we do not)."
As diversity has some potential benefits in environments with negative feedback loops (diminishing marginal returns) and high variance, could we somehow subsidize diversity or tax a lack of it?
Let's first look at diversifying financial capital ownership. However, individual market participants are already heavily incentivized to diversify their assets. As a result; large financial intermediaries are now investing people's savings into nicely diversified groups of asset classes; giving those intermediaries lots and lots of market power. Weyl and Posner argue for regulation to enforce diversity: major institutional investors should not be able to invest in more than one company in a particular sector/vertical.
Policies which enforce diversity of human capital are already mainstream, though there is a lot of ground to cover here as well. As humans have strong tendencies to avoid diversity, I suspect that this debate will continue for a long time.
When explaining concepts; there is always a question of how much to explain by analogy (which is useful given that the majority of human brain activity is likely to be cache lookups) versus how much to explain from first principles. I think the author did a great job at doing both, presenting both examples and equations from a wide array of scientific fields. One of the author's other books, Computational Models in Political Economy, is going on my shortlist for future reviews.
The various examples show a number of potentially stabilizing or adapting impacts of diversity, which may prove helpful when building models of the world or trying to evaluate your next course of action. If you like this material, you'll probably also like Algorithms To Live By, which is filled with more equations and models to provide insights for daily life!