I loved markets. I got a Ph.D. in economics from the University of Chicago, wrote a free market microeconomics textbook, and recognized that markets have made most people in the developed world richer than medieval kings. Yet, I'm increasingly convinced that these very markets are likely steering us towards the creation of an uncontrollable artificial superintelligence that probably kills everyone. In this essay, I discuss how the challenges of AI existential risks distinctively leverage market weaknesses. It’s as if a vengeful demon, channeling Karl Marx, created a scenario highlighting their flaws. The thesis of this essay is that if you believe that a smarter-than-human AI would create a significant existential risk, then even if you normally are a staunch advocate of free markets, you should be concerned that these markets are potentially leading us to a disastrous outcome.

In my microeconomics classes I present a hypothetical scenario to students, drawing inspiration from an example by the great 18^th century economist Adam Smith. Imagine, while cooking and watching TV, you see a news report about a devastating earthquake in an unfamiliar country, claiming over 100,000 lives, including children. The images of the catastrophe deeply unsettle you. In your distraction, you accidentally cut off the tip of your pinky finger. Upon calling their mother in distress and being asked what’s wrong, my students admit they would mention the finger injury over the distant tragedy, revealing an innate tendency to value personal afflictions over the suffering of strangers, no matter the scale.

In an ideal world, the economy would operate with each individual dedicated to the greater good of humanity. However, as E.O. Wilson wryly noted about communism, “Wonderful theory, wrong species.” While we may deeply care for our immediate circle—friends and family—the same level of concern rarely extends to distant others. So how have we managed to build an extensive, interconnected global economy? The insight of Adam Smith is key here: "It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner, but from their regard to their own interest." Our economic system leverages this self-interest; it's structured so that serving others often coincides with personal gain. For instance, a baker who makes tastier or more affordable bread not only boosts his profits but also serves the community. He is guided as if by an “invisible hand” to understand and cater to his customers' desires, sometimes even before they do. 

But markets don't magically align our self-interest with the broader good. There are challenges to market efficiency that create a wedge between profit maximization and the betterment of society. A critical wedge is externalities – unintended consequences of an activity that affect individuals not directly involved. Take, for example, a bakery whose operations generate noise pollution harmful to nearby residents.

Although consumers might generally prefer a market where all businesses mitigate externalities such as noise pollution, their individual purchasing decisions often don't reflect this preference. This is because businesses that don't invest in reducing externalities can offer lower prices. For an individual consumer, the negligible impact of a single purchase on the overall noise pollution levels often leads to a preference for cheaper options. Consequently, there's a paradox where everyone might agree that a market with less noise pollution is preferable, but individual incentives lead to a collective outcome where businesses that ignore such externalities thrive. Yet, this situation is far from hopeless.

Direct negotiations might address this local externality, given the small number of stakeholders and their probable existing relationships. The bakery could offer compensation or complimentary bread to the affected residents, or alternatively, the residents might impose a cost on the bakery as an incentive to resolve the issue. Alternatively, residents bothered by the noise might relocate, making way for others less sensitive to it. While this scenario highlights a market challenge in promoting societal benefit, it often lends itself to resolution through community-level interventions.

When the bakery's operations result in city-wide externalities like air pollution, the challenge for market solutions intensifies. The problem's vast scale renders direct negotiations between the bakery and each impacted resident impractical. Moreover, the idea of residents relocating to escape pollution is impractical, considering the large number affected and the extreme measure of leaving the city.

Unchecked pollution externalities can severely degrade urban environments, challenging the effectiveness of markets. However, even imperfect governments can play a successful role in managing these externalities, as demonstrated by the clean air and water in almost all the cities of wealthier nations. Furthermore, concrete air quality measurements greatly assist in developing and assessing relevant policies. Governments, already accustomed to managing disputes among city residents, are frequently well-equipped to address these issues.  While pollution within a legal jurisdiction presents a significant challenge to market-based solutions, I believe it doesn't negate the overall advantages of market systems.

To add complexity to the challenge to market efficiency, envision a scenario where a company's actions result in global pollution, an externality affecting everyone. Individual negotiations or domestic solutions are ineffective here; global cooperation is essential. However, international decision-making is notoriously less developed and more complex than national governance, compounded by conflicts between countries. This complexity echoes the challenges in tackling climate change, a major global concern, despite its widely acknowledged severity and proposed solutions such as the carbon tax, which is popular among economists.

Further compounding the challenge to market effectiveness, consider pollution that yields immediate benefits but harbors long-term negative consequences. Politicians, driven by self-interest, often focus on short-term gains, reducing their motivation to tackle such externalities. This is evident in democratic systems where the complexity of issues and the perceived insignificance of individual votes lead to a lack of accountability for long-term outcomes. This phenomenon can be seen in the ongoing crisis of U.S. entitlement programs, where delaying resolution for short-term comfort exacerbates future costs. This contrasts with a homeowner who, knowing their roof will fail in ten years, understands the economic benefit of early repair to enhance the home's resale value even if he plans on selling his house this year. Therefore, anticipating that politicians will steer markets towards long-term sustainability is unrealistic, akin to expecting children to persuade their parents to opt for healthy family diets over the immediate gratification of sweets.

Adding a layer of confusion only intensifies our externality dilemma. Suppose there is a division of opinion about whether the pollution is beneficial, uncertainty about when its effects will manifest, and a general consensus that the mechanisms at play are beyond the grasp of the political elite. This is the case with AI existential risk.

In a free market economy, profit serves as the primary motivator for innovation and development. Companies and investors are constantly seeking new technologies and advancements that promise significant returns. AI represents one of these frontier technologies where the potential for profit is enormous. From automating routine tasks to revolutionizing industries like healthcare, finance, and transportation, AI promises to unlock new levels of efficiency and capability. AI offers literally trillions of dollars of profit and enormous consumer satisfaction. It is incentivized by markets like nothing else in history. 

Competition propels us towards artificial superintelligence, as any AI firm slowing its pace risks being overtaken by others, and workers understand that refusing to engage in capacity research merely leads to their replacement. Consequently, even if you personally think AI will probably kill everyone, you can morally justify working on capacity research because you know the marginal benefit to AI safety of your quitting just determined by how much better you are at your job than the person who would be hired to replace you. We have a true market failure when even if all the relevant participants recognized they were pushing humanity towards extinction and were all willing to accept a significant reduction in wealth for a non-trivial reduction in risk, the logic of markets and competition could keep them working towards quickly creating an artificial superintelligence.

While the pursuit of profit motivates firms to develop increasingly advanced AIs, the potential catastrophic consequences represent an externality that current market structures struggle to address. The existential risk of AI, being global, elusive, and long-term, sharply contrasts with its immediate, tangible benefits. This mismatch challenges governments to intervene effectively.

Confronting the AI existential risk may require revisiting a concept highlighted by Adam Smith: the propensity for collusion in any trade to reduce competition. He wrote "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices." This inherent tendency towards collusion among market players could be leveraged in regulating AI. Permitting collusion among existing AI firms could decelerate the race to create a superintelligent computer. While this would likely involve restricting competition, leading to higher profits and reduced innovation for these firms, it could be a lesser evil compared to the alternative risks of unregulated AI development. However, admittedly, this approach carries its own dangers, such as the possibility of these firms becoming too powerful or even using their position to take over the world.

Written with the assistance of GPT-4.