[Crossposted from my Medium blog]
Only a few institutions fulfill their intended purpose. Such a functional institution stands out as remarkable. It is the exception rather than the rule, always tracing its beginnings to a founder. Such institutions at first always have a skilled pilot — he can alter and direct the institution in a way that preserves or improves its functionality. If he weren’t able to do so, he would not have been able to create a functional institution.
However, the founder cannot remain the pilot forever. Another pilot, a successor, has to step in and take the reins for the institution to remain piloted. Furthermore, in order for the institution to remain functional and a live player, this new pilot must also be skilled. Such a person extends the life of the institution, allowing it to achieve more than it otherwise would. Ensuring the institution acquires this new, skilled pilot is the succession problem.
Institutional longevity is a matter of skill and power
As we see, the succession problem has two components: power succession (handing off the reins of the institution, keeping it piloted) and skill succession (transferring the skill needed to pilot the institution well, keeping it a live player).
If the founder handles both parts of the succession problem, successfully handing off the institution to a person who can skillfully alter it as necessary, then the institution remains piloted and a live player. If neither part of the succession problem is handled, then the institution becomes unpiloted and a dead player.
If power succession is successful but skill succession is not, then the institution remains piloted, but not a live player. Someone is at the controls, but they don’t really know how to use them.
There are multiple possible outcomes to such a scenario. At worst, the pilot aggressively mismanages the institution. This situation can be catastrophic; the pilot might crash the plane. At best, the unskilled pilot remains at the controls but intervenes minimally, allowing the institution to function while also defending his ability to alter and direct it. Of course, if the pilot is also not skilled enough to maintain his power then the institution will eventually become unpiloted unless a skilled pilot steps in.
If skill succession is successful but power succession is not, then the institution becomes unpiloted and a dead player unless and until the skilled person gains the necessary institutional power to pilot it.
As an example, the founder of a company might retire, giving way to an outside CEO appointed by a rather conservative board. Even if there is a junior engineer or designer that has the passion and expertise needed for a whole new kind of product that the team could deliver, he will not be positioned to realize this vision.
To successfully change the company, the engineer would have to at best persuade, at worst bypass, the new management. Ideally the employee eventually maneuvers themselves to the position of CEO. If this happens at all, it can take years. Instead, he might do better to fundraise and recruit for a new start-up.
If you imagine a chaotic and disorganized parent company, with shards of responsibility and bureaucratic entrenchment, in the above example the problem of succession gets harder and not easier.
Over the lifespan of a bureaucracy, power lent out to various delegates becomes owned, allowing individuals to use organizational resources to pursue agendas at odds with the purpose of the organization. They form an entrenched opposition, that makes rendering the institution functional a notable challenge. Gaining control of institutions that have become unpiloted is often more difficult than founding one’s own institution, even for a skilled actor.
Even after becoming unpiloted, a functional institution can remain effective for a while, but it will decay and eventually cease to be functional or even cease to exist unless someone captures it and starts piloting it.
Creative destruction is not a necessity for innovation
Silicon Valley enthuses over disruption because we have become so used to the succession problem remaining unsolved. To disrupt an organization, industry, or culture can only be good if it isn’t possible to cooperatively transform it. Under such conditions each generation of innovators must start anew or waste their efforts with a sclerotic structure.
Successful skill succession coupled with failed power succession can lead to destructive strife. People of exceptional ability and ambition do not necessarily seek out conflict, but will generally be willing to endure it. Depending on their options it might be the best course of action to attempt to dismantle or destroy the old organization which they couldn’t work with.
Functioning firms are repositories of many kinds of capital that cannot be liquidated, and when they die, it is destroyed. Such capital includes position, team synchronization, good organization and tacit technical know-how. These are casualties of economic competition.
We have no problem identifying this phenomenon as problematic in politics. We view the destruction of an old political order by means such as civil war or political strife as regrettable necessity at best, not something to celebrate. This stands in stark contrast with our view of the phenomenon in the economy, likely because we overlook the destructive side.
Few mature technological companies today use their position to support effective innovation. Many companies spend significant resources on research, but few manage to aggressively implement and deploy. Amazon provides a contemporary proof of possibility, with its constant pursuit of technical innovation in service of ever larger economies of scale and logistical efficiency.
That Amazon is one of the exceptions reflects the poor health of the current batch of institutions, rather than the nature of mature companies or even underlying market incentives. Disruption should be the backup rather than the first choice for innovation. That it isn’t is the result of poor institutional health.
An overabundance of talent in the absence of sufficient opportunity and power succession can render society quite chaotic. If ambition is outlawed, only outlaws are ambitious. On the other side of the spectrum, buying stability through the absence of talent is futile in the long run. Institutions ultimately decay without renovation either from within or without.
Sclerotic institutions eventually break rather than bend, which is the source of catastrophic instability. When thinking of a company, this might result in a desolated company town, when thinking of a civilization, the result is societal collapse.
Organizations and societies that solve the succession problems will have a less harsh trade-off between stability and innovation. When institutions of the previous generation are actively handed off to the next, they retain needed flexibility to pursue restructuring.
While variation between individual organizations is notable, most rely on social technology that is widely distributed and implemented around their society. A society is best thought of as a dense ecosystem of institutions always borrowing from each other, outsourcing services and sometimes clashing over resources. It can be very difficult to implement a unique solution. If none of the institutions in a society solve a particular problem, the fragility of those institutions will be reflected in the fragility of society as a whole.
The foundation of a flourishing civilization is an abundance of functional institutions. These originate with founders who bring new social designs into being. In the natural course of events their institutional legacy decays, becoming less and less suited to achieving the desired positive effects.
The succession problem is the problem of ensuring founders can hand off institutions they have built to other founders. The key problems here are the creation and identification of sufficient skill, together with ensuring the next founder has inherited a position of sufficient power to remake the institution. Of course, even if the succession problem is handled once, it always returns.
If the succession problem is unsolved, the only process of institutional reform is the destruction of abandoned institutions by new ones, the process sometimes described as ‘creative destruction’.. That our society valorizes rather than bemoans such outcomes, unfortunately demonstrates that we have become accustomed to failed succession and notable dysfunction.
We should temper our enthusiasm for intense political and economic competition and instead develop a greater appreciation for the importance of successful succession. This change would go far in remedying contemporary institutional sclerosis and stagnation.