I’m really enjoying this series of posts. Perhaps this will be addressed in 4-6, but I’m wondering about prescriptions which might follow from c-risks.
I have a sense that humanity has built a tower of knowledge and capital and capacities, and just above the tower is a rope upwards labeled “AGI”. We don’t know exactly where it leads, aside from upwards.
But the tower we’ve built is also dissolving along multiple dimensions. Civilizations require many sorts of capital and investment, and in some ways we’ve been “eating the seed corn.” Decline is a real possibility, and if we leap for the AGI rope and miss, we might fall fairly hard and have to rebuild for a while.
There might be silver linings to a fall, as long as it’s not too hard and we get a second chance at things. Maybe the second attempt at the AGI rope could be more ‘sane’ in certain ways. My models aren’t good enough here to know what scenario to root for.
At any rate, today it seems like we’re in something like Robin Hanson’s “Dreamtime”, an era of ridiculous surplus, inefficiency, and delusion. Dreamtimes are finite; they end. I think either AGI or civilizational collapse will end our Dreamtime.
What’s worth doing in this Dreamtime, before both surpluses and illusions vanish? My sense is:
Investing in solving the dysfunctions of our time and preventing a hard collapse also seems hugely worthwhile, if it’s tractable!
Looking forward to your conclusions.
Thanks for the comment! I look into some of the philosophy required in part 4, x-risk relationships in part 5 and todos in part 6. Understanding consciousness is an important sub-component and might be needed sooner than we think. I think an important piece is understanding what modifications to consciousness are harmful or beneficial. This would have a sub-problem of what chemicals or organisms alter it and in what ways as well as what ideas and experiences seem to have a lasting effect on people. It's possible this is more doable than understanding consciousness as a whole, but it certainly touches a lot of the core problem.
TL; DR (for all six parts)
Empires have collapsed previously due to decreases in social cohesion. Narrow AIs, such as large language models combined with overly aggressive social media A/B testing aimed at behavioral modification, can cause decreases in social cohesion. This anti-economy (economy of externalities) of behavioral modification centralizes benefits to a few tech controllers, distributes costs, and is similar to physical pollution. Virtual signal pollution of vital personal signs of trust, such as AI-enabled scams and spam, is especially worrisome in a low social cohesion environment. Other narrow AIs aimed at molecular assembly and runaway tech risks such as bio-lab leaks compound already existing problems. Civilizational risk (c-risk), while bad enough by itself, can lower civilizational competence. Civilization needs this competence to deal with x-risks, which may also arise along the same causal pathways. However, caring decision-makers can take steps to reduce the impact of AI's improper modification of groups and individuals.
Difference from Other Models of Civilization
In this essay, I discuss narrow current AI and general future AI as a civilizational risk. Civilizational risk (c-risk) is separate from existential risk (x-risk) and involves the collapse of nations similar to prior historical collapses. C-risk can be made worse by un-aligned narrow and general AI. My probabilities of c-risk: P(c-risk by 2033 | lack of narrow alignment) = 31% AND P(c-risk by 2070 | lack of narrow or general alignment) = 92%, where c-risk is the collapse of Western Civilization. I also argue that P(misalignment x-risk|AGI) > 35% by 2070 in part 5, after establishing the likely backdrop of civilizational risk.
While I agree with Eliezer and Bostrom's view that an extremely powerful AGI could destroy humanity or drastically limit its potential, I highly disagree with the likely pathway civilization would take towards that point, conditioned on safety research not being done.
One of Eliezer's models is that "it is going to be increasingly cool right up until … "an existential failure point of fast unaligned FOOM. To be more precise, in Soares, Tallinn, and Yudkowsky discuss AGI cognition - LessWrong, Eliezer considers multiple possible phases, such as "Plotting," "Hiding and Biding Phase," and "Doomed." However, they generally look like "everything is going mostly well" from the operators' perspective. The "good looking phase" followed by "doom" is similar to Bostrom's idea of the treacherous turn, discussed here: Treacherous turns in the wild. In those models, civilization goes steadily upwards and then at some point hits a treacherous turn with AI "fooming." Once FOOM happens, AI achieves a decisive strategic advantage, and the world is doomed.
I expect a lack of alignment to seriously affect narrow AIs, especially ones involved in modifying human behavior. In my model, large portions of human civilizations are already on a self-destructive path using narrow AI. This path could lead to having entire nations or alliances collapse due to internal societal pressures. Other people's models do not necessarily forbid this type of outcome. However, I want to argue for reasonably high probabilities of it, point to signals that we are or will be on this path, and how this affects the strategy of dealing with AGI.
My model of declining social cohesion is similar to previous historiographic accounts of empires collapsing. I also look at current life expectancy trends and technological health effects. Thus, I believe my model is better calibrated based on both the "outside view" priors of history as well as due to an "inside-view" gears-level explanations of incentives of the players involved. While many people have proposed historically-informed general mechanisms by which civilizations could collapse, they generally need to connect narrow or general AI to these trends.
Historical Priors on Collapses and Inflation
Civilizational collapse is still severe but does not include the extinction of the entire humanity. Civilizations have collapsed previously. Nearly all civilizations have gone through periods described as a "collapse." Empires are especially prone to drastic changes compared to their height. Many historians, as early as the 14th century, have considered that empires have a particular lifetime and follow their destiny to end in particular ways.
The cyclical theory of empires proposed by a famous Muslim historiographer, Ibn Khaldun, advances the idea that empires have a definitive end time of approximately 250 years, give or take about 20 years. These numbers have entered the public discourse given that American Empire is coming up on 250 years in 2026. However, the essential part of the theory is not the actual number but the core variable of social cohesion (assabiyyah).
He claims that social cohesion is high in the earliest years of empires, such as within nomadic warrior people. They need socially cohesive kin networks to survive. Then they use those capacities as a well-coordinated military to conquer more numerous but potentially less socially cohesive people. Then social cohesion slowly degrades through time and ends in a societal collapse.
Other thinkers modified this theory to include specific stages:
Age of Explorers
Age of Warriors
Age of Commerce
Age of Affluence
Age of Science / Intellect
Age of Celebrities/Decadence
Age of Decline / Collapse.
The United States does have signals of being in the celebrity stage of empire. The specific transition points that would correspond to that theory are: The end of the Civil War is the transition away from "The Age of Warriors" to "The Age of Business / Commerce." The Great Depression transitioned from "The Age of Business / Commerce" to "The Age of Science / Intellect." The transition away from "The Age of Science / Intellect" was between 1971 and 1973, which culminated in the 1973 oil crisis. While this may seem like a more minor "crisis" than the last two, many economic metrics showed this period as a core transitionary period. There is a whole website dedicated to looking at various trends around that time. Another point toward this being an age of celebrity is the relative rise of celebrities in political power, starting with Ronald Reagan.
A key metric not mentioned in "WTF happened in 1971" is life expectancy. Using surplus value to extend human life may partially explain the issue of increasing productivity but not increasing wages from 1973 to 2014. Life expectancy was 71.36 in 1793, 77.6 in 2005, and 78.8 in 2014. However, life expectancy was 76.1 in 2021, so this explanation does not apply to the period of 2014 - 2022.
There are other cyclical history theories separate from the "empire cycle" theory. The Fourth Turning Thesis, proposed in a 1997 book, has predicted unrest in the US and a "crisis period" in the early 2020s.
COVID has also been compared to Chernobyl in that it produced similar levels of shock among the population toward the state's capacity to solve problems. Notably, the Soviet Union collapsed only five years after Chernobyl.
There are many pointers that the US and the West are on a path toward collapse. However, it is worth expanding on what the likely modalities are. When I talk about civilizational collapse, I view it as one of three primary adverse outcomes. The first and most likely one is economic collapse through hyperinflation. Two is a Civil War, and three is a massive drop in life expectancy (below 65 average life expectancy). They are not mutually exclusive. I estimate the probability of civilizational risk is high even in the short term, such as a 31% risk of collapse of the United States or large portions of Western Civilization by 2033. I view such collapse as 92% by 2070, even if humanity does not develop AGI, but narrow AIs keep gaining function in an unaligned manner.
Different modalities of decreasing social cohesion interact with different modalities of collapse. Civil war happens when social cohesion is low in the nation, but two or more clusters each have high internal cohesion. Hyperinflation happens when the politically connected elite loses cohesion with the middle-class cash holders and decides to print money for themselves. Life expectancy drops can happen after a mass death due to either war, resource scarcity, or lack of individual will to continue living. These are likely when the powers outside the collapsing nation do not have cohesion with it and thus are willing to take advantage of the situation rather than aid it.
It is worth understanding the historical priors of high inflation and hyperinflation specifically. It happens a lot around the collapse of empires. The world's first recorded hyperinflation came during the French Revolution (1790-1795), where monthly inflation peaked at 143 percent. After the collapse of the Russian Empire, during communism, the early Soviet hyperinflationary period was marked by three successive redenominations of its currency, in which "new rubles" replaced old at the rates of 10,000-to-1 (January 1, 1922), 100-to-1 (January 1, 1923), and 50,000-to-1 (March 7, 1924), respectively. After the collapse of the British Empire, from 1950-1982, the British pound lost 90% of its value.
In ancient times similar things happened.
While this is not high inflation by modern standards, this was undoubtedly high inflation by standards of the time and accompanied the fall of the roman empire as well.
Of course, non-empires are also prone to suffering high inflation periods. Germany: A loaf of bread in Berlin that cost around 160 Marks at the end of 1922 cost 200,000,000,000 Marks by late 1923. Hungary: In 1944, the highest banknote value was 1,000 P. By the end of 1945, it was 10,000,000 P, and the highest value in mid-1946 was 100,000,000,000,000,000,000 P (10^20 pengő), Yugoslavia had multiple periods: One spanned 22 months, from March 1992 to January 1994. Inflation peaked at a monthly rate of 313 million percent in January 1994. Poland also had periods in both 1923 and 1989. Furthermore, all the above examples are only from Europe; there are plenty of examples elsewhere. Argentina has ~80% CPI growth this year. Hyperinflation, while likely underemphasized in popular historical narratives, is extremely common around the collapse of empires and other transitionary periods. Simply on priors, one ought to expect this to be a definite possibility in America.
Given a collapse, I expect the conditional probability of high inflation to be above 95%. Given the historical priors, do we have any US-specific data that would update the probability of hyperinflation in one direction or another? Evidence points to hyperinflation or other forms of "economic collapse" is more likely and more severe compared to historical priors rather than less.
A specific point of having national debt be 90% of GDP is a core point when the economy of a country is expected to slow down significantly. The US is likely to reach this point in 2026.
Other nations holding reserve currency are likely to begin divesting from it as mutual hostility increases. Mutual hostility between nations is likely to increase even further due to social media and other pressures.
Whether or not you believe in the cyclical theories of history, it is worth knowing that civilizational collapse is a common occurrence in history. Furthermore, the core variable that underlines the probability and severity of the collapse and the possibility of a comeback is social cohesion.
Social cohesion is not truly a formalized notion. One way to think through its formalization analogy to AI is that social cohesion between two agents is high when they both act as if they include the other's utility as part of their concern. I model social cohesion of groups as to what extent each of the group's members acts with total group utility maximization as a goal. Social cohesion is high in small kin networks where people readily root for each other. Social cohesion can also be high in societies where people have a strong sense of "playing by the rules" and readily enter into mutually beneficial trade agreements, even if the people are not related.
Things that have historically increased social cohesion and decreased war chances are: forming elite kin networks and intermarriage. Another social cohesion-increasing idea is just simply successfully doing hard things together.
Things that decrease social cohesion are "Wedge issues," or issues specifically created to divide one portion of the population from another. Poor mental health has a negative feedback cycle with social cohesion. Violent Rhetoric, especially coming from "official sources," is both a sign and a potential cause of low social cohesion.
There are many signals "blinking red" that social cohesion in the US is low and falling. Americans have fewer friends, less sex, and use mental health treatment more than before.
However, the core problem discussed in future parts is that narrow AIs and social media optimization algorithms automate boosting of "wedge issues" and destroying social cohesion both within nations and outside of them. If left unchecked, this inevitably brings about the collapse of civilization either through great power contests or internal implosions.
All parts
P3: Anti-economy and Signal Pollution