Institutions have been suffering a massive brain drain ever since the private sector shot way ahead at providing opportunities for smart people.

Think of any highly capable person who could contribute high-quality socially-valuable work to an important institution like the New York Times, CDC, Federal Highway Administration, city counsel, etc. What's the highest-paying, highest-fun career opportunity for such a person? Today, it's probably the private sector.

Institutions can't pay much because they don't have feedback loops that can properly value an individual's contribution. For example, if you work for the CDC and are largely the one responsible for saving 100,000 lives, you probably won't get a meaningful raise or even much status boost, compared to someone who just thrives on office politics and doesn't save any net lives.

In past decades, the private sector had the same problem as institutions: It was unable to attribute disproportionate value to most people's work. So in past decades, a typical smart person could have a competitive job offer from an institution. In that scenario, they might pick the institution because their passion for a certain type of work, and the pride of doing it well, and the pride of public service, on top of the competitive compensation and promotion opportunities, was the most attractive career option.

But now we're in a decades-long trend where the private sector has shot way ahead of institutions in its ability to offer smart people a good job. There are many rapidly-scaling tech(-enabled) companies, and it's increasingly common for the top 10% of workers to contribute 1,000%+ as much value as the median worker in their field, and companies are increasingly better at making higher-paying job offers to people based on their level of capability.

We see institutions do increasingly stupid things because the kind of smart people who used to be there are instead doing private-sector stuff.

The coordination problem of "fixing institutions" reduces to the coordination problem of designing institutions whose pay scale is calibrated to the amount of social good that smart people do when working there, relative to private sector jobs. The past gave us this scenario accidentally, but no such luck in the present.

Here's a take on why it's gotten harder to form and maintain beneficial institutions and social structures, using my favourite lens: Memetics.

(I mentioned this briefly on the group call, but I missed part of the model before, and can expose it to more people here)

• Enough people have a preference for hosting memes which are broadly prosocial, build a legacy, etc, and for co-operating with others who have this kind of preference to create self-sustaining cultures, so long as you get founding right and people can generally distinguish prosocial memes from others
• People over time have a roughly stable personal ability to distinguish memes that have good effects from ones more optimised for spreading at the cost of important norms
• Technologies have progressively lowered barriers to communication, allowing
  • More rapid evolution of memes which are better at fooling the distinguishing mechanisms while actually being hyper-optimised for self-promotion and disabling parts of the social environment's memetic immune system (e.g. fear of call-out/cancel culture reducing the ability of subcultures to reject it and related memes, even when there would normally be antibodies reacting to a meme's effects).
  • These highly optimised memes to be more rapidly spread into local cultures. A stable system need a certain dose of a virus or memeplex to become infected rather than petering out against the local background culture. A system gets a founding dose of alien memes much more often when new members maintain stronger connections to their old tribe (reinforce old memetic patterns), is broadly more exposed to flows of memes, and when likeminded people can more easily find each other to form a locally reinforcing group.
• This means that on an individual level people end up less able to sort valuable thought patterns like old NYT's culture from viral patterns, and on a cultural level are less able to maintain "cell walls" against memeplexes competing for space in their substrate's brains.
• I also think there some stuff around having hyper-optimised memes delivered by a hyper-optimised Out to Get You attention economy cutting into the free mindspace which is important for distinguishing prosocial from self-promoting memes, so maybe we're also getting worse at that at the same time as the job gets harder in several ways.

Attempting to blindsight the answer: 

In the past I imagine that people were usually trying to 'be a serious person'. And that's still true. But somehow being a serious person is now faker. And I think maybe its because they're being a very scared serious person. Somehow they're a lot more vulnerable from every direction. Or there's a lot more directions they're vulnerable from.

I don’t have a clear hypothesis, so I will instead write down a few pretty disparate observations that feel related.

• One thing I was surprised by around the Petrov Day Red Button was people saying they could not imagine that someone would actually care about the ritual. This felt like it was relevant to understanding why culture is hard to build and maintain.
• I was fairly unhappy with my graduation ceremony at University. All of our families were sat above each other in a tall circular room, while a man in formal robes said that the students had grown into adults through their work at the university. I feel like the parents and university staff were coordinating around a world where this narrative was true, and that people would respect us because we had a degree. But I didn’t grow because of my university, I grew in spite of it, and people no longer trust that a degree means much. I feel like the parents and the man in the centre of the room all thought it did, and I imagine once they were correct.
• I’ve been thinking about internet subcultures other than LessWrong and Effective Altruism, and thinking about how to help them out more. I think one thing that’s common is that the people trying to lead in building the infrastructure are not the people who started the culture but just people who are mood-affiliated and like the initial people. I also feel like such people are much more scared to act out what’s distinctive about their subculture, and thus normalize it a lot more.
• Robin had a good recent post about how basically every country in the world coordinated to execute the same regulations around COVID – no experiments, no variolation, etc. This feels really alarming to me, like this fact should be central in my understanding of the world, whereas at the minute it feels like a thing I’m ignoring (whilst nonetheless continuing to anticipate).
• I have an anticipation whereby if you want to be part of the popular discourse yet not simply ‘play your role’, you have to walk through fields of people saying awful things about you. I felt like OpenAI did it with their work on release practices, where most of the public dialogue was not ’staggered release versus open science’ but was more like ‘is OpenAI fundamentally immoral for doing this or not‘. I feel like Musk does it constantly, and I think that Musk not letting this get to him this is part of what allows his basic successes with Tesla and SpaceX to be part of the discussion.
• Dave Chappelle has been very successful and given respect and prestige while nonetheless ignoring all the pressures on him to bend the knee, and instead just being funny. Which is strange. It stands out to me that he lives in the same small town in Ohio that he grew up in, raises his family there and knows everyone there, rather than having moved to one of the major coastal cities for his work like LA or New York.

If I were to try to make things up this minute. I’d say I feel like there aren’t enough heroes, and that the main times most of my heroes are brought up in popular culture is to attack and destroy their reputation, and that the internet makes things very contextless and ignores people’s history and positive reputation. So I feel like the internet has made people think there are no good people to look up to, and this makes it harder to trust new people.

I asked just the title of this post to someone near me, who first laughed and said “ha ha not possible,” and when I said “no, really”, they came back with “community”. I asked for more details and it went something like this:

Community is the everyday practice of negotiating a positive outcome with people who aren’t just like you. When you do this regularly with people around you, you learn that they are people and that they have your back. Think churches, block parties, school boards. When community is our primary source of human interaction, we build this muscle of cooperation-by-default because that’s the social expectation, and because successful cooperation has positive feedback cycles that produce immediate returns.

We suck at this today: our communities are online, national and personalized. There’s no longer a forcing function to be nice to / learn to communicate with our neighbors.

One hypothesis for why it has gotten harder to form institutional cultures (I am assuming here that it has):

I’ll call this the “Geeks, MOPS, and sociopaths” model. Under this model (put forward e.g. in the essay of Ben Hoffman’s that I linked above), it has somehow become easier and more common for people to successfully ape the appearances of an institutional culture, while not truly being true to it (and so, while betraying it in longer-term or harder-to-trace ways).

In the example of the NYT, this could occur in several ways:

• People getting jobs within the NYT who believe less sincerely in the old journalistic ethic (though they perhaps believe in looking like they believe in whatever is popular);
• Alternative press outlets (Washington Post, or whoever) arising that believe less sincerely in journalistic ethics (or anything like this) than the NYT, but who parasitize the “kind of like journalistic ethics” brand by aping its appearance to readers;
• Leadership of the NYT being more interested in bending the NYT’s brand (and its internal culture/ethics) to however current people today happen to be evaluating which newspaper to trust, in ways that boost those leaders’ personal [$/prestige/political power] but that harm the longer-term legacy of the institution (because future people, who are under the sway of different fads, won’t see it this way).

Related argument: the 4-hour documentary / propaganda film “Century of the Self” argues that the dispersion of game theory (“it’s virtuous to think about my self-interest and e.g. defect in prisoners’ dilemmas”) and of marketing/focus groups/“public relations” (“my brand can figure out how other people are making sense of the world on a pre-conscious level, by using techniques similar to Gendlin’s Focusing on them, and can thereby figure out how to be perceived as having a certain ethic/culture/institution by hacking their detectors”) led to more of this sort of aping, and replaced institutional cultures that might’ve helped past people do real work with LARPing and “lifestyles.”

This is also quite related to Goodhardt’s law. But under this hypothesis, dynamics have somehow changed dynamics so that [individuals/organizations who are trying to appear to have virtues] are able to successfully fool the detectors of [individuals/organizations that are are trying to detect whether they have virtues]. It does not explain why that would have changed.

I want to lay out a basic hypothesis which claims that the main variable is political will. Important variables feeding into this are corruption and social capital.

Resources sort of feeding into this model (some more sort-of than others):

• The Dictator's Handbook (this one is for real the source of most of the intuitions)
• Moral Mazes, Elephant in the Brain (these aren't actually things I have read, but feel relevant)
• The Moral Economy: Why Good Incentives are No Substitute for Good Citizens
• Bowling Alone

Basic claim: quality institutions come from "political will". Political will is basically like coordination-juice. I will claim that political will comes from two places:

• Social Capital: the distributed trust that people have in each other.
• Explicit Structure: the actual political arrangement.

The question of where social capital comes from is an open one, but a few important factors:

• Television seems to have struck a significant blow to social capital, for the simple reason that it caused people to stay home in the evenings rather than go out.
• Free market capitalism and high social capital seem quite correlated.
• Measures of government corruption seem like a good proxy for low social capital.

As far as explicit structure goes, there are two basic sources of "political will":

• Benevolent dictators. These do exist, but it seems like it's not something you can count on.
• Much more often, the political will to create solid institutions comes from large-coalition dynamics.

Large-Coalition Dynamics

To those who have not read The Dictator's Handbook, I highly recommend it. You can get the basic model from the CPG Grey video The Rules for Rulers.

The basic idea is that you can put governments on a spectrum from democracy to dictatorship by asking how many people does the leader need to keep happy in order to stay in power. This is the "coalition". A very small coalition makes a government autocratic, and a very large coalition makes it democratic (regardless of the formal government structure). (Note that even within a democracy, the number of supporters required for a party to maintain power varies greatly depending on the voting method and the structure of the government.)

A ruler always has an incentive to minimize the size of the winning coalition, in order to minimize the number of people who need to be satisfied. This maximizes the amount the ruler can take for their own interests (be they benevolent or otherwise), and also maximizes the longevity of their rule (because it's easier to keep the coalition satisfied, and therefore, easier to keep it stable). Strong rulers will therefore tend to cut down the size of their coalition in a wide variety of ways.

However, the most important rule for a ruler is to keep their coalition satisfied. So although the ruler may want to cut the coalition down, whether they can get away with this depends largely on whether the coalition wants it, too.

The Dictator's Handbook argues that there is a turning point between small and large coalitions. Small coalitions generally want to become smaller, so long as they can maintain power: less people to split the pie between. Small coalitions will essentially loot the public welfare for their personal gains, restrict freedoms to prevent revolt, and shut down all parts of the economy except what they personally benefit from (through over-taxation and other means).

Large coalitions, on the other hand, rely more on public-good rewards. They want to grow the economy, create good infrastructure, etc. Large coalitions usually recognize that the best way to do this is to grow the coalition further, which helps ensure that the leader (who still wants a smaller coalition) won't boot anyone out, and puts more pressure on the leader to increase freedoms, invest in public works, etc.

So that's why dictatorship isn't the only stable attractor; democracy is also a stable attractor. (Even though the democratic leaders will keep looking for ways to reduce coalition size.)

This is, on this theory, the source of stable, cooperative institutions: the large-coalition attractor.

On this theory, the main interventions are those which decrease corruption, increase the size of the coalition, and increase social capital.

At the town hall, you remarked that you had made a list of many explanations, each of which looked apparently likely to be true when viewed in isolation, but none of which seemed obviously more compelling than the others. I attempted the same exercise, and got a similar result. Looking at my list and thinking about it, I'm not convinced there is necessarily a center; some of the hypotheses look like they plausibly could be, but the alternative is that there are a bunch of mechanisms which push to higher and lower levels of corruption, which mostly wind up cancelling out, but that some of the mechanisms are positive-feedback loops.

Here's my list of hypotheses, clustered in a few groupings. This is somewhere half-way between "developed theories that I fully believe" and this babble challenge.

Institutions Which Corrode Others in Proportion to their Own Corrosion

News media. The judiciary. Higher education. Unions. Grantmakers. Each of these institutions (or categories of institutions) has a surface area through which it influences other institutions, and is influenced by them in return; that influence might be positive or negative, depending on how well functioning it is. News media and the judiciary influence the behavior or other institutions through scandal and liability, respectively; that distortion can be positive or negative.

Loss of Slack

The parts of an institutional culture that trade off short- and long-term incentives rely on the institution and the people within it having slack. For a number of reasons, both people and organizations seem to have less slack than they used to. There are a few big well-known ones affecting individuals: student loans, housing prices, and health care prices, in particular. But the ones affecting institutions and person-instutition relationships are more interesting:

Financialization: It is fairly common for an investor to take an organization with a secure income stream and a lot of slack, and use financial instruments to transform it into a precarious organization plus money elsewhere. The prototypical instance of this is the leveraged buyout, which seems to have first started being a thing in the 1980s.

No More Company Men (/Women): Decades ago, and still in some countries, the relationship between an employee and a corporation was a long term and loyal one; people would spend most of their career inside a single organization. This implies a low risk of being fired, a long time horizon in their relationship with the company, and a lot of time in which to absorb a company's culture.

Loss of Ability to Filter

In order to function, an institution needs to recruit people filtered for both competence and alignment, which can be done either by recruiters assessing people directly, or by delegating and judging based on work history or formal credentials. Most large institutions have shifted heavily towards the latter, while the formal-credential-giving institutions have shifted away from selecting on intelligence and towards selecting on time-expenditure and conscientiousness. As a result, many institutions seem to have filled up with people who have papers that say they're qualified, who nevertheless aren't.

Scalability Problems

Some norms are enforced in a distributed way, where the enforcement only works if the violation is known widely enough, eg boycotts in response to corporate misbehavior. If the population doubles, the number of corporations doubles, and the per-capita number of boycott targets stays constant, then the total influence of boycotts halves. Similar effects exist internally when scaling up institutions, when increasing the number of participants in a market, and when increasing the complexity of regulation.

A noteworthy part of the experience of hanging out in any sort of niche forum is that US's major institutions have a recurring cast of scandals which never seem to be resolved, and which the relevant people don't show much awareness of. In 2015, the Congressional reaction to police brutality and discrimination didn't look like disagreement, it looked like not having enough bandwidth to investigate or think clearly about it. This is what we should expect to happen more and more to fixed-size institutions as the world gets more complex.

A related problem is that as the influence of an institution grows, the rewards for capturing or corrupting it also grow, and the growth between offense and defense is not symmetric. An example of this is the relationship between the PR and newspaper industries; the ratio of PR resources to investigative-reporting resources has grown drastically, and this fact seems like a natural consequence of economic growth.

Adversarial Action

Foreign intelligence agencies are actively working to reduce unity within the US.

A history of corruption in powerful institutions created a cultural backlash against institutional power in general, which isn't selective enough.

Institutions are easier to sabotage than they are to create, so scaling up the world disfavors them.

Decreased Attention Spans

When I was growing up, it was a standard aphorism that "television rots your brain"--ie, that consuming too much media messes people up in some way. I recall a more specific claim, which was that television decreases attention span -- ie, people who watch a lot of television have more scattered attention. People now say similar things about social media, and I think the effect is the same: people tend to spend less consecutive seconds on each thought, and have a harder time dealing with long inferential distances.

Many of the major problems look like leaders are being too miserly with cognition; there's a correct model of the problem which has some inferential distance, and a competing model of the problem which is simpler but wrong, and we find leaders acting as though they believe the latter. Eg the "landlords are greedy" model (simple, wrong) vs the "prices are high when housing construction is inhibited by regulation" model (correct, but more complex).

Assorted Other Hypotheses

Exposure advertising induces resistance to a class of messaging which includes both advertising and organizational ideology.

Business schools are teaching MBAs a strategy that doesn't rely on understanding the culture of an institution, and so they go on to destroy the local culture wherever they go.

The same thing that's causing an obesity crisis, also changes people's psychology in a way that makes them hard to build institutions out of.

Cynicism is a self-fulfilling prophecy; believing that an institution is bad makes the people within it stop trying, and the good people stop going there.

Many organizations relied on an implicit hierarchy in which older people were higher ranked and less numerous; the decrease in birth rate broke this.

Con artists are more skilled than they used to be, because TV/social media/something else made it easier for them to practice, and most organizations are being captured by them.

I think your causal options are missing an important one - "the invisible hand".  

Many people independently value the nice thing, and they altruistically decide to put their own efforts toward creating/maintaining the nice thing?

The first half is fine, but replace "altruistically" with "selfishly".  Someone wants the thing, and thinks many others want the thing.  They figure out how to make a living by providing the thing to everyone.  For non-exclusionary things (lighthouses being the classic example, although most of them were actually privately funded), the providers figure out how to get taxpayers or civic organizations to pay them.  

I like your 0-3 framework, and my answer for the decline is primarily about the following in #1:

• breakdown in long-term rewards.  The larger cultural institutions of stability and consumer expectations have shifted to short-term evaluations.  Institutions that seek long-term rewards, at the expense of short-term survival, don't get either one.

One can certainly argue that traditional values are biased and unpleasant, and encouraging radical change is the right thing to do.  But it remains true that this harms institutions which had optimized for long-term value in the previous environment.  And to the extent that there's uncertainty in the stability of the future (cultural) environment, it's hard to see how new institutions can correctly optimize for the long-term.

I've put together some guesses about what's important for US competence as a nation, loosely based on ideas from WEIRDest People and Where is my Flying Car?.

Human societies likely default to small groups that fragment (due to disagreements) if they grow much above 20 people.

Over the past 10 millennia or so, it has become common to use extended ties of kinship to scale up to the Dunbar number, and sometimes well beyond that.

Western civilization scaled up to unprecedented levels of trust and cooperation via a set of fairly new cultural features: moral universalism, use of impartial rules rather than contextual particularism, the expectation of supernatural punishment for undetected crimes, more emphasis on analytical thinking, and more positive-sum thinking.

The US has been partly held together by a shared religion, whose teachings promote trust between co-religionists, and which also encourage treating others as potential converts.

Shared enemies (Nazis, Communism, maybe briefly Islam) created additional, but temporary, boosts to cooperation within the English speaking world. Some of the polarization we've recently experienced is just a return to patterns that were previously common. If that were most of what's going wrong, I'd be fairly optimistic about the future of the US.

Over the past few decades, the US has experienced a decline in religion (or a least in a shared religion).

Science got too aggressive about demanding that we disbelieve any knowledge beyond what scientific journals would publish. That eroded beliefs in supernatural phenomena, and also eroded beliefs in the religion(s) that helped to promote large-scale trust and cooperation.

Science didn't succeed in replacing religion with something more rigorous. Instead, new quasi-religions sprouted (e.g. Green fundamentalism, and the cult of Trump). They're optimized more for features such as compatibility with forager instincts, than the ability to promote prosperity.

In contrast, the Protestant religion was selected in part for its ability to foster cooperation between distant strangers.

I'm concerned that many US problems of the past few decades (including The Great Stagnation) can only be solved by something like a return to being a Christian nation. The tension between Science and Christianity seems strong enough that it's hard to see how that is feasible.

Another problem is that democracy has morphed from a tool, to a goal in itself.

That has undercut the authority of political parties. They used to have near total control over what candidates were on the ballot. Then, starting around 1970, there was a massive shift toward direct voter control over who each party nominated.

That made it harder to hold any institution accountable for political results. Perhaps it's not a coincidence that this trend started around the same time as The Great Stagnation. It seems correlated with reduced trust in authority in general, although I can't tell whether this is a cause or effect.

WEIRDest People also claims that WEIRD cultures have produced lower testosterone levels, via monogamy. That's important, since it reduces impulsivity, reduces competitive urges, and increases positive-sum thinking.

But testosterone seems to have decreased over the past few decades, so the US ought to be in a better position than most societies to rebuild institutions.

I'm maybe 90% confident that the US still has enough competence to postpone collapse and civil war for a few decades.

It seems like there should be some research on how companies, nonprofits, etc. scale up past the Dunbar number. But I'm unclear whether it's relevant to groups as big as the US, but WEIRDest People has led me to expect that the optimal approach for a group of 300 million people will be rather different from the optimal approach for 200,000 people.

Vinay Gupta, in Cutting Through Spiritual Colonialism, and Venkatesh Rao, in The Gervais Principle, paint a picture where the routine operation and maintenance of life and organizations generates some sort of pollution (focusing mostly on the intrapersonal and interpersonal varieties), and an important function of institutions is basically doing the 'plumbing work' of routing the pollution away from where it does noticeable damage to where it doesn't do noticeable damage. I don't think I fully endorse this lens, but it seems like it resonates moderately well, and combines with trends in a few unsettling ways.

In centuries past, it was common to have communities that cared very strongly about whether or not insiders were treated fairly, but perceived the rest of the world as "fair game" to be fleeced as much as you could get away with; now the 'expanding moral circle' seems more common (while obviously not universal), in a way that makes the 'plumbing work' harder to do. [If life requires aggression, and you have fewer legal targets, this increases the friction life has to work against.] 

It seems like our credit-allocation mechanisms have become weirdly unbalanced, where it's both easier to evade responsibility / delete your identity and start over / impact many people who will never know it was you who impacted them, and simultaneously it's easier to discover crimes, put things on permanent records, and rally the attention of thousands and millions to direct at wrongdoers. The new way that they operate seems to have empowered Social Desirability Bias; once we might have imagined the Very Serious People leading the crowd, and now it seems the crowds are leading the Very Serious People.

This is also one of the ways that I think about the 'crisis in confidence'; see Revolt of the Public for more details, but my basic take is that experts have always been uncertain and incorrect and yet portrayed themselves as certain and correct as part of their role's bargain with broader society. Overconfidence helps experts serve their function of reassuring and coordinating the public, and part of the 'plumbing work' is marginalizing dissent and keeping it constrained to private congregations of experts. But with expanded flow of information, we both have more expertise as a society, and more memory of expert mistakes, and more virulent memes spreading distrust in experts. This feels like the sort of thing where we get lots of short-term benefits in correcting the expert opinion, but also long-term costs in that we lose the ability to coordinate because of expertise.

[Feynman in an autobiography describes his father, who makes uniforms, pointing out that uniforms are manufactured / Feynman shouldn't reflexively trust people because of their uniforms, which seems like great advice for Feynman, but not great advice for everyone in society; the social technology of respecting uniforms does actually do a lot of useful work!]

Very interesting question, Anna.  In fact, it covers so much territory and is so deep I'm not sure where to start.  

To one of your core questions, "the difficulty we are having lately in forming/sustaining institutional cultures (especially, ones adequate to get much done)" - Yuval Levin has studied and written extensively on this.  

One of Levin's core diagnoses is that there is a new tendency of people to use institutions as platforms to launch/improve their own popularity rather than letting the institution form and guide them.  

Think of the relatively new phenomenon of new star journalists using, say the NYT, as the platform to leverage and boost their own brand and career independent of the NYT.  Or, say a politician who uses the R or D party as a platform for a media following rather than the old way of slowly working your way up the ranks of a party and gaining influence and prestige within the confines of the organizational norms.  

At the risk of blaming social media for yet another social ill, I do think SM enables this "platform using" behavior.  In today's world, a young upstart politician can get her own following on Twitter or FB, breaking through the limitations of the old party organization.  

===

Consider also larger societal and cultural trends that have been in play for decades now. On average Americans are much wealthier and have more opportunities than in past generations.  One of the side effects of higher wealth is people don't need or desire to be part of institutions as much as before.  Fifty years ago if someone was down on their luck, short on money, getting help from their local church or VFW Club would have been more common.  

Relatedly and additionally, a surge in government benefits also reduces the need and incentives to rely on those local institutions.  Why bother with the church or the Lion's Club if state and local gov. is providing checks no (or few) questions asked.  

According to Strauss–Howe generational theory, history is made of cycling Saecula each divided into four generational Turnings: the High, the Awakening, the Unraveling, and the Crisis, in that order, which tend to last approximately 20 years while the next social generation of people advances in age and takes over their role in society. Each Turning is provoked by the flaws of the last.

We are now approaching the end of the current Crisis Turning (since about 2005). The Crisis is the Turning when institutions are at their nadir, due to their long Unraveling in the previous Turning (1982–2004), when individualism was at its peak. Institutions will be destroyed and rebuilt for the next Turning. The current Millennial Saeculum will end in approximately 2025 and usher in the next Saeculum. The coming High will be when collective institutions will be strongest and individualism will be at its weakest.

The fourness of the Turnings also pattern match onto other things. Strauss and Howe associate each social generation with the Prophet, Nomad, Hero, and Artist archetypes, in that order. (The Millenials are Heroes, Generation X are Nomads, etc.) I also wonder if this fourness also maps onto Simulacra Levels, after reading Zvi's The Four Children of the Seder as the Simulacra Levels, which also has a generational feel.

Given that framework, strong institutions develop in response to a crisis, when culture tilts toward collectivism. When that collective culture unravels into individualism, the institutions decay.

[Epistemic status: stab-in-the-dark pattern matching.]

I studied a lot of stuff linked to this, what follows is my attempt to organise my knowledge. Here are the main processes I think are draining the kind of cooperation you described from society.

Inequality

Cooperation in society is suffering immensely from the rising economical inequalities. The evidence and theories linking the rise of economical inequality to the rise of a wide series of social problems, including a strong decrease of mutual trust, cooperation, altruism and a decline in social bonds between people. These are all factors that would be needed for the kind of cooperation described in the question.

Essentially, the two minute version of the theory it's that in a society where wealth is unfairly distributed, with a very small percentage of people owning most of it and a lot of steps in the social ladder all the way down, everyone is stressing over social status all the time, seeing everyone else as a potential rival, and so everyone is worse off, even the rich.

I highly suggest The Spirit Level by Wilkinson and Pickett, they do an excellent work at explaining the theory and the evidence, and all the rest of the scientific evidence coming in is saying the same things.

Deregulation

Most of the progress obtained by western societies was made when the worse (domestic) follies of capitalism were kept in check by good rules that were there for extremely good reasons.

You can go see the results of deregulation with extreme clarity in the banking sector, from the exact moment a bunch of rules went out of the window with the various deregulation sprees we got stuck in a repetitive cycle of banking crisis.

With these deregulation sprees there were also massive cuts on a lot of forms and institutions of social welfare, which studies are showing now to have been things society extremely needed to function properly. I think most of the research I had read on this was in The Body Economic: Why Austerity Kills by Stuckler and Basu, which were able to present really strong evidence of the harmful effects of these welfare cuts and deregulation sprees on society. These policies of welfare were essential to keep inequalities more or less in check, with the effects mentioned above. So deregulation interacts with inequality by being one of its main causes,  the richest part of the population has basically turned into a black hole that keeps getting richer every year why the rest of the population isn't, and it's wealth is not so slowly getting eroded. The less rules there are in an economy, the more the strong can grind down the weak. A lot of rules were also related to how much company should pay employees, how easy is to fire people and so on, so more inequality for everyone. 

Also, deregulation walked hand in hand with privatisation, and it turns out (I think it's in The Body Economic as well) that public company are not, in fact, less efficient than private ones, and that they do actually tend to have incentive systems that are more aligned with providing good services than to just make money.

I remember reading deregulation and privatisation being linked in attempts to cut public information medias in the UK by Thatcher, since you explicitly mentioned newspapers.

General thoughts on this

So it seems that if you try hard to optimise only for a single factor, say profit, and take off all the rules that try to limit the allowed tactics to those that don't actually harm society, thus letting a large number of psychopaths get real creative at making money while also being able to learn from each other tactics, and if you keep accidentally selecting for said psychopaths in your managerial levels as well, what you get it's not more wealth for everyone, but all kind of horrible consequences, and an extremely messed up society. 

And why should you, if you systematically try to make your genies less aligned to human and society values before you make your wish.

Managers who don't care about long term consequences but just rake up short term profits are systematically advantaged over the ones who would care for the company in the long term. The less rules there are, the more their advantage grow. The occasional billionaires that own the whole company and have on top of that a vision for the future are just too rare when the number of psychopaths between managers is six times the common ratio in the population. 

Oh, and there is also a lot of research showing that employees mostly acted in the interest of the company if the company was perceived as being loyal to them, wages and other incentives didn't even got close to get that kind of results. Research also showed that just thinking about moneys made people less moral and more likely to cheat, so there's also that problem in trying to use money as the only goal someone should have in a system.

Fairness is an extremely powerful incentive for optimisation, because if someone is being fair toward you, you try to be fair in return, you don't try to game the incentive system to get more fairness. Cialdini studies on reciprocity come to mind. 

But this kind of relationship between company and employee was being enforced by rules that have been long thrown out.

Then, if we add on top of all that an ideology that explicitly praise only personal economical success and egoistical individualism and we market said ideology to the masses, and we add in the common knowledge that your society would just let you alone to die if something goes wrong with your health or wealth... 

If a hunter-gatherer brain found itself in a pack where everyone was just in a competition to get more resources, was explicitly told the rule in this pack is to care only for its own success, didn't know most of its members at all and knew for a fact the pack would abandon you to die if something went wrong... well, it seems pretty hard this brain would care at all about trying the kind of cooperation you describe. It would mainly care about getting to the top, or staying out of troubles and looking out for himself if he wasn't confident in its skill.

As an European, I think USA culture is just poisoned with this kind of out of control individualism and "free market" ideology. Reasonably free market are good for society, excessively free market... aren't.

I'd advise everyone from USA to try and look at those concepts with fresh eyes, and to check how other societies are seeing the issue. I was really surprised nobody here seemed to have noticed this.

I don't know if I can really do any justice to everything I've studied on this without trying to write a sequence, I tried my best to explain but there's too much to cover...

If anything I tried to explain was unclear or anyone would be interested in more links and evidence please ask me so. I only mentioned studies and facts on which my memory was certain.

Edit 06/11/20: If it's not a bother, I'd like to ask people to give me their hypothesis on why this answer is so "controversial", karma wise. I don't mean this as a complaint at all, I simply plan to write more about this subjects here on Lesswrong and feedback would be useful. 

I'd like to talk about the broader topic too, but for now I just want to try out some Disputes.

Governmental institutions: There seems to be some degree of institutional failure (mild-ish, so far) in a number of American and especially Californian institutions: California's electricity is less reliable than it used to be, due basically to bad governance. 

There are definitely a number of institutional failures in recent years (another is Flint's government choice to ignore health and safety recommendations to save money resulting in contaminated water) but do we know that these sorts of events are actually getting worse than the type/frequency found in previous 20-year-chunks of American history?

San Francisco, especially, is seeing rising crime, due more or less to decriminalizing a lot of crime. 

Is there a source for this? Preliminary investigation indicates otherwise. Some crimes in particular are up, but it's not clear that they're because things like "arson" or "car thefts" were decriminalized? Has that actually happened?

Many aspects of the covid-19 response also cast our institutions in a worse light than I'd previously anticipated, though it is plausible (given my ignorance) that my anticipations were the silly thing here and that we would not in fact have done better in previous eras. (I'm thinking here of: America being slower than I'd anticipated re: acquiring testing and PPE; putting very little money in the extensive stimulus bill to reducing covid via testing/research/etc.; America staying in semi-lockdown for an extended time instead of trying harder either to head toward actual zero (via border control, testing + tracing, etc.) so that we could relax again, or toward something more like herd immunity (while metering it out; but it seems to me that as a country we probably lost more to the costs many parts of America seeming not to lock down for extended periods of time without a plan to use that time to do anything constructive, and without (I think?) adequate accounting for what that would cost in terms of social stability and mental health.)

It feels important for me to ask why this is related to "forming stable, cooperative institutions?" The evidence I've seen points to it being fairly easily explained by an excessively bad administration and lack of leadership, with many well documented unforced errors in preparedness and coordination, as well as simple failures to pick low-hanging-fruit. 

I'm not saying we should discount "The US government" as an institution, obviously it is a massively important one, but that importance comes from the way it effects others, and thus makes it hard to judge overall competence of institutions downstream of it. The Executive Branch in particular has massive amounts of control and effect on all sorts of areas beyond what people would generally think of as "their job." Personally I trust the CDC about as much as I did before, I'm just more aware of how much a bad Executive can hinder them.

Non-governmental parts of our national sense-making apparatus: Most brand names, e.g. the NYT, Harvard, Science and Nature magazines, the Democrats, the Republicans, the police, the CDC, etc. seem less well-regarded than they used to be. I can't think of many brands of any sort that are instead better-regarded (Amazon, SpaceX and bitcoin, probably).

This seems true, and largely due to the overall democratization of news/opinions/science/all manner of gatekeepers in our culture. In terms of clear-seeing, we're more aware of the mistakes they make, and in terms of exaggerated criticism, we're more influenced by antagonistically-produced memes.

Subcultures: David Chapman claims that subcultures are much harder to form now / more or less don’t exist anymore. I have also tried to look myself, and this matches my own experience: rationality and EA seem among the few things that are sort-of here, and even we are only sort-of here, I think. ("The rationalist diaspora," not "the happening applied rationality scene.") (I can think of some others, e.g. the authentic relating / circling communities; some other parts of the Thiel-o-sphere; maybe the group at the Stoa; surely some others. But... fewer than I would have expected, and I think fewer than I would have found in past decades?)

This seems genuinely surprising to me. There actually seems to be far more subcultures being formed than there ever were before? Even discounting subcultures formed around specific media, there's certainly a lot of political subcultures that have formed in the past ~20 years. What's the standard for what qualifies as a "culture" in this space?

A lot of the answer to this question is in Charles Murray's Coming Apart (2010). In it he makes extensive use of Government statistics from surveys and economic analysis to trace the fortunes of working class and upper income community types from 1960 to 2010. There are four key founding virtues: industriousness, honesty, religiosity, and marriage. America had these in abundance from its founding up through 1960. After 1960 upper classes retained most of them, but the working classes experienced major declines. These were societal in extent; no blame assigned, it is simply what happened. The two classes have diverged strongly, and while the upper income class will be fine, without these virtues, working class communities that have come apart probably cannot be put together again.

Though he does pick the "virtues" that he chooses to study, the book is jammed with graphs and explanations of the data sources. To dismiss it you'd need to dismiss the validity of data that is from disparate sources that pre-date the book.  

One big challenge is that the double income, university-educated, urban elites in WDC, SF, LA, and NYC have little knowledge of the experiences of those in "flyover country." This leads to political polarization that is fed from different lived experiences, but I think the point of the book is that even if you removed politics completely from your view of this divergence, the four virtues are what are needed to get things back together. Government can't do that on its own. These are values that people pick for themselves. Government policies have actually nudged people towards discarding these values, but there is no political will for the types of programs (or absence thereof) that would undo that nudging.

One possible hypothesis for the question about the lack of new institutions may be related to the ability to assuage oneself with comparatively meaningless activities. For example, I can now write an angry message on a city's social media page and make myself feel like I've done something about the lack of fire protection. I will receive just as much social affirmation (aka likes) that way compared to cold calling all my neighbors and asking them if they'd like to form a committee to raise our taxes and staff an additional fire station.

Individuals' ethics is the cornerstone

To answer the first question: I think it's mostly individuals' ethics. I think the shared ethics lead to a culture based on those ethics, which in turn reinforces the individuals' ethics. This accounts for the changes created by a large group of people as well as the changes created by individuals.

I think personal ethics have been slowly degrading as evidenced by the reception of Jordan Peterson's (and others') message along the same lines. The message is: pay close attention to your individual ethics and how they are impacting the people around you; if your life is a mess, start fixing it locally.

Speaking of JP, one of his points is that we lost respect for things that we have, like the government, industries, and policies). We have a lot of people trying to disrupt the system for the greater good without realizing they are breaking down the Chesterton's fence. I think this explains, for example, people fighting for the the reduction of police force, while it pretty conclusively leads to increased crime.

Overall, I think I mostly disagree with your sentiment. I agree that things are changing, but I think a lot of the things you mentioned still exist and work fine. They might appear worse than they are due to modern media's alarmist reporting and short term (~20y) trends.

Internet is responsible for degradation of real-world communities

Internet leads to online communities and cooperation, but directly drains time from local and national communities. (Not counting the insane time drain from all the distractions it creates.) Plus the social media rewards social bubbles + polarized thinking, leading to intense tribalism within your bubble and overall divisiveness.

If you look at how many online communities there are, it's kind of insane. How many movements start online. Previously a lot of that effort would have gone into the real-world communities and movements.

The best are getting better

Given increased mobility, people go for the best. Case in point: all the rationalist chapters around the world went into a decline, but it's because the bay area attracted most of the people who were really into rationality. I think the best of the best (companies, communities, etc...) are growing faster than ever before. 

Government and decline of nationalism

I think short, acute wars (WW2 + Cold War) lead to an increase in nationalism. Plus they shift people towards Level 1. ("There's a sniper across the river" has to be interpreted on Level 1.) People operating on Level 1 likely have better individual ethic, leading to my first point. Increased nationalism leads to higher interest and prestige in working for the government, and during the war itself there's an increased urgency as well.

There are some important relevant insights in Henrich's book The WEIRDest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous. I'll say more after I finish reading the book.

I think there are a few puzzles about a good institutional culture (IC) and their tendency to fade/thrive here.

1. If an institution has a good IC, does it always thrive?
2. Can an institution thrive even without a good IC?
3. How can we distinguish carefully between a good IC and the condition of fading/thriving? For example, if a good IC brings a good reputation, but we say that the NY Times is fading because its reputation is declining, then saying that lacking a good IC caused the NY Times to fade is tautological.
4. A good IC may neither be necessary nor sufficient for an institution to thrive. But is the creation of a good IC necessary (if not sufficient) to create an institution?
5. If a good IC is of overriding importance to the existence, success, or failure of institutions, then what are the necessary and sufficient conditions to create a good IC from scratch? To turn a bad IC into a good one?

My dad has a story from when he took over his first nonprofit clinic as a CEO. It was in severe financial distress when he started. Two of the doctors, he discovered, liked to play a game. Rather than taking on patients (they were salaried), they would stand in one of the rooms and try to toss pennies out the window and get them to land on the window ledge of the building across the alley. One of the things my dad did to turn it around was to tell them that they'd be fired if they didn't change their ways.

In this case, I think the order of events was something like:

The clinic was fading. Decision-makers responsible for it brought in a new leader to investigate the IC. He discovered that there was a disconnect in the self-reinforcing cycle: two of the producers were neither being rewarded for working nor punished for not working. By threatening to punish them for not working, the leader took a step towards restarting a good IC.

This suggests a new, important facet to your question. How does the IC of an institution get repaired and maintained when it breaks down? If an institution fails entirely, when and how do its parts get recycled into healthier institutions? When a fading institution breaks down, is this generally bad (because it's gone from bad to nonexistent) or good (because now its parts can be reincorporated into institutions that are thriving).

It also suggests to me that a hierarchy of responsible leadership is important. Having supervisors who can add, remove, or replace workers, or leaders, at various levels of the hierarchy has two good effects. It is both incentivizing as a reward or punishment, and it allows a more detailed investigation of the inner workings of a particular institution.

This loosely suggests that one failure mode for an institution is when the highest leadership is corrupt. In a democracy, the highest leader is the citizenry who elect officials. If the citizens are corrupt, then the democracy will suffer. In a small business, the highest leader is the owner. If the owner is corrupt, then the business will suffer. In a publicly-traded company, it is the shareholders and the government (and, by proxy, the citizens) who are the highest leaders; if one or both is corrupt, then the corporation will suffer.

Great Founder Theory looks like a detailed (and lengthy) attempt to answer something close to this.

What drives social change throughout history and the present? What are the origins of institutional health or sclerosis? My answer is that a small number of functional institutions founded by exceptional individuals form the core of society. These institutions are imperfectly imitated by the rest of society, multiplying their effect. The original versions outperform their imitators, and are responsible for the creation and renewal of society and all the good things that come with it—whether we think of technology, wealth, or the preservation of a society’s values. Over time, functional institutions decay. As the landscape of founders and institutions changes, so does the landscape of society.

This answer forms the basis of the lens through which I analyze current and historical events, affairs, and figures. But though it may be intuitively compelling, fully substantiating such a framework is no small task. This manuscript, titled Great Founder Theory, is my substantiation. It explains all of the models that are key to understanding how great founders shape society through the generations, covering such topics as strategy, power, knowledge, social technology, and more.

The answer is pretty simple Imo.

In pre-modernity, communities were small. everyone knows everyone else. Contentiousness and agreeableness were desirable. If you were selfish or mean you were ostracized. cooperativeness is rewarded by the system.

In modernity, communities are big and no one knows each other. If you are selfish, you can put on a friendly persona and hire a PR team even. Narcissists and psychopaths who are capable of draining the system of as much resource as possible have the most reward in the system. If you’re cooperative and trusting in a city, you’re more likely to be taken advantage of.

As we urbanized and industrialized, society as a whole slowly shed pre-modern values for post-modern values. Pre-modern values build Civilization and post-modern values tear it down. Humans (or any life form) become whatever their environment rewards.

The rate of change in society is also high and accelerating, which speeds up the decay of existing institutions.

The reason it may seem our societal ability to create well-working institutions is declining could also have to do with the apparent fact that the whole idea of duty and the honor that this used to confer is not as much in vogue anymore as it used to be. Also, Equality and Diversity aside, being "ideological" is not really a thing anymore... the heydays of being an idealist and brazenly standing for something are seemingly over.

The general public seem to be more interested in rights and not responsibilities, somehow unable to understand that they can only meaningfully exist together. I was having a conversation the other day about whether it would be a good idea to introduce compulsory voting in the US, as this would render moot a significant number of dirty tricks used to de-facto disenfranchise certain groups... almost all objections came from the "I"-side; I have a right to this, I am entitled to that... the whole idea that, gee, you know, you might be obliged to spend 1-3 hours every 2 or 4 years to participate in society is already too much of a bloody hassle. Well yeah... with that kind of mindset, it's no wonder the institutions that require an actual commitment to maintaining robust societal functions is hard to find...

Love it!

To mirror what I got:

Institutions are structured groups of agent pulling in the same direction to gain redistributable value.

They work by aligning the incentives (especially the long term ones) of the agents with the institution through the technology of an institutional culture to provide guidance and help police detection.

An additional point I've been thinking about since I read Sapiens:

• This cultural process recruits map/territory machinery to help people make sense of it. "Journalistic Ethics" is presented as an objective value like "Honour" or "Privilege"

• From the inside "I am a valued member of a cohesive and effective institution" can feel more motivating than "I am working to provide this institution long term value that it will redistribute to me"