Group selection might seem like an odd topic for a LessWrong post. Yet a google seach for "group selection" site:lesswrong.com turns up 345 results.
Just the power and generality of the concept of evolution is enough to justify posts on it here. In addition, the impact group selection could have on the analysis of social structure, government, politics, and the architecture of self-modifying artificial intelligences is hard to over-estimate. David Sloan Wilson wrote that "group selection is arguably the single most important concept for understanding the nature of politics from an evolutionary perspective." (You should read his complete article here - it's a much more thorough debunking of the debunking of group selection than this post, although I'm not convinced his interpretation of kin selection is sensible.) And I will argue that it has particular relevance to the study of rationality.
Eliezer's earlier post The Tragedy of Group Selectionism dismisses group selection, based on a mathematical model by Henry Harpending and Alan Rogers. That model is, however, fatally flawed: It studies the fixation of altruistic vs. selfish genes within groups of fixed size. The groups never go extinct. But group selection happens when groups are selected against. The math used to argue against group selection assumes from the outset that group selection does not occur. (This is also true of Maynard Smith's famous haystack model.)
(That post is still valuable; its main purpose is to argue that math trumps wishes and aesthetics. Empirical data, however, trumps mathematical models.)
Nitpicking digression on definitions
"Group selection" is one of those tricky phrases that doesn't mean what it means. Denotationally, group selectiond means selection at the level of a group. Connotationally, though, group selectionc means selection for altruistic genes at the level of a group. This is because, historically, group selection was posited to explain genetic adaptations that are hard to explain using individual selection.
group selectionn, selection at the group level for traits that are neutral or harmful at the level of the individual, or that don't even exist within the individual genome, should also be considered. group selectionc is a subset of group selectionn is a subset of group selectiond. If group-level selection occurs at all, then traits of the group that are not genetic traits, including cultural knowledge, must be considered. That makes a huge difference. Human history is full of group selectionn. Every time one group with better technology or social organization pushes another group off of its land, that's at least group selectionn.
If you want to model evolution thoroughly, and selection of groups occurs, then you need to model group selectiond to get your predictions to match reality, even if group selection occurs entirely as a result of non-group selectionc genetic traits that provide advantages to individuals. But people reject group selectiond on the basis of arguments leveled against group selectionc.
A case study of group selectionc: Nightshades
But I'm not backing off from saying that group selection can explain (some) altruism. Edward Wilson has been threatening for several years to write a book showing that group selection is more important than kin selection for generating altruism in ants. He doesn't seem to have published the book, but you can read his article about it. (Short version: Group selection is especially important in ants because ant colonies, which are small groups, engage in constant warfare with each other.)
And this brings me to the reason for writing this post now. Last week's Science contained an article by Emma Goldberg et al. with the most clear-cut demonstration of group selection that I have yet seen (summarized here). It concerns flowering plants of the nightshade family (Solanaceae). They descend from plants that evolved self-incompatibility (SI) about 90 million years ago. SI plants can't pollinate themselves. This is a puzzling trait. Sexual reproduction in itself is puzzling enough; but once a species is sexual, individual selection should drive out SI in favor of self-compatibility (SC), the ability to self-pollinate. SC gives individuals a great reproductive advantage - it means their offspring can contain 100% of their genes, rather than only 50%. The advantage given by SC is much greater than the supposed advantage of asexual over sexual reproduction: SC plants can both leave their own cloned offspring, and foist their genes onto the offspring of their neighbors at no additional cost to themselves. SC also makes survival of their genes much more likely when a single plant is isolated far from others of its species; this, in turn, makes spreading over geographical areas easier.
And yet, SI is a complex, multi-gene mechanism that evolved to prevent SC. Why did it evolve?
The authors looked at a phylogenetic tree of 998 species of Solanaceae. In this tree, SI keeps devolving into SC. Being an SC mutant in an SI species is the best of both worlds. You get to pollinate yourself, and exploit your altruistic SI neighbors. When some members of an SI species go SC, we expect those SC genes to eventually become fixed. And once a Solanaceae species loses SI and becomes SC, it never re-evolves SI. This has been going on for 36 million years. So why are so many species of Solanaceae still SI?
Let sI = speciation rate for SI; eI = extinction rate for SI; rI = net rate of species diversification = sI - eI. Likewise, rC is the net rate of species diversification for SC species. qIC is the rate of transition from SI to SC. SI will be lost completely if sI - eI = rI < rC + qIC = (sC - eC) + qIC.
The data shows that sC > sI, but eC >> eI, enough so that rI > rC + qIC. In English: Self-pollinators speciate and diversify more rapidly than SI species do, as we expect because SC provides an individual advantage. Once self-pollinators evolve in an SI species, these exploiters out-compete their altruistic SI neighbors until the entire species becomes SC. However, SC species go extinct more often than SI species. This is thought to be because SI makes a species less-likely to fixate deleterious genes (makes it more evolvable, in other words). Individual selection favors SC; but species selection favors SI more than enough to balance this out. Notice that gene-based group selection at the species level is mathematically more difficult than group selection at the tribal (or ant colony) level (ADDED: unless there is genetic flow between groups at the tribal/colony level).
So let's stop "accusing" people of invoking group selection. Group selection is real.
Group selection is especially relevant to rationality because, in an evolving system, if we use the definition "Rationalists win," "winning" applies to the unit of selection. In my painfully long post Only humans can have human values, the sections "Instincts, algorithms, preferences, and beliefs are artificial categories" and "Bias, heuristic, or preference?" argue that the boundary between an organism's biases and values is an artificial analytic distinction. Similarly, if group selection happens in people, then our discussion of rationality and values is overly focused on the rationality and values of individuals, when group dynamics are part of what produces rational (winning) group behavior.
Even if you still don't believe in group selectionc, you should accept that group selectionn may allow information to drift back and forth, in a fitness-neutral way, from being stored in genomes, to being culturally transmitted. And that makes it necessary, when talking about rationality in a normative way, to consider the rationality of the group, and not just the rationality of its individuals.
(This is related to my unpopular essay Rationalists lose when others choose. When the unit of selection is the group, rather than the individual, the "choice" is made on the basis of benefit to the group, rather than benefit to the individual. This will prefer "irrational" individuals who terminally (perhaps unconsciously) value benefits to the group, and not just benefits to themselves, over "rational" self-interest.)
group selectionn makes the Prisoner's Dilemma and tragedies of the commons smaller problems. But it raises a new problem: Is the individual the wrong place to put some of our collective rationality? Since humans have evolved in groups for a long time, the default assumption is that attributes, such as our rationality, are already optimized for the unit of selection.
Less generally, if the group has already evolved to place some of our rationality into the group, what will happen if we try to instill it into the individuals? Since group selection is real, we can expect to find situations where making individuals more rational upsets the evolutionary equilibrium and makes the group win less. Under what circumstances will making individuals more rational interact badly with group dynamics, and make our group less rational (= win less)? This will probably occur in circumstances involving individual altruism. But if the locus of group rationality can drift from individual genes to cultural knowledge, it may also occur in situations not involving altruism.
Postscript: The long-term necessity of war
If group selection is partly responsible for human altruism, this means that world peace may increase selfishness. Konrad Lorenz made a subset of this claim in On Aggression (1966): He claimed that the more effective each individual's killing tools are, the more necessary empathy is, to keep members of the group from killing each other; then invoked group selection. (This seems to me to apply a lot to canines and not much to felines.) If group selection works best with small groups, the switch from tribes to nation-states may have already begun this process. I do not, however, notice markedly greater altruism in tribal groups than in nation-states.