[Linkpost to this. Loosely based on a lecture I gave in therecursiveconference with the same title. Don’t take “2030” literally¹ —it could also be 2035 or 2040. As always, opinions are my own and do not represent OpenAI or Harvard.]
It is quite possible that AI will usher in a golden age for humanity. As Amodei writes in “Machines of Loving Grace,” AI might enable the “prevention and treatment of nearly all natural infectious diseases,” the “elimination of most cancer,” “prevention of Alzheimer’s,” and even “doubling of the human lifespan.” Through increased growth, AI could lead to a reduced workweek, as well as universal basic income and health insurance. Universities might focus on education for its own sake, as opposed to vocational purposes. People will spend less time sitting on chairs and staring at screens, and more in relational jobs. AI may also lead to new ways to participate in democracy, as well as oversight of government.
And yet, what if, as Ice-T once said, “shit ain’t like that”? In this opinionated and somewhat informal essay, I want to consider this more pessimistic or “glass half empty” case. Specifically,
Imagine that, in the next decade or so, the AGI transition has gone very badly for the U.S.²or humanity at large. What are the possible scenarios that could lead to that?
Part of the reason to consider different scenarios is that some interventions that could help mitigate one scenario might end up amplifying another. A particularly salient axis is that of control vs. distribution. Some scenarios and risks motivate interventions in the direction of increased control: restricting access to frontier models to government or trusted parties, restricting access to the information on AI research, restricting open source models, etc.. Other risks, such as concentration of power, might motivate the opposite direction of increased distribution: wide distribution of AI to spread its benefits, increased transparency on both government and internal lab use of AI.
I don’t think there is one simple answer, and neither extreme is likely to lead to good results. But we should be keeping the various tradeoffs and risks in mind when considering any potential intervention. I am particularly concerned about questions regarding concentration of power. A future in which we humanity is under the control of either AI or a small group of unelected humans is not a good future in my books, even if we are materially well off. Hence I am concerned about interventions that either rely on ceding more control to a “good” AI, reduce the level of competition among AI model providers, or restrict access to advanced AI to the blessed few. This does not mean that we should have no restrictions for safety. Only that we should try to keep these narrowly tailored to address specific measurable risks, rather than broad restrictions motivated by hypothetical risks.
Overview of bad scenarios
I will review the following general “families” of non-mutually-exclusive scenarios in which AI leads to very bad outcomes:
Catastrophic misuse: AI is used by humans to cause a catastrophe on a global or national scale—one that would radically worsen the course of events for the United States or the world at large. These types of events are considered in many frontier lab scaling policies (e.g., Anthropic responsible scaling policy, GDM frontier policy, OpenAI preparedness framework). Specifically, two scenarios that are often considered are:
AI used for catastrophic cybersecurity harms.
AI used for catastrophic biological, chemical, radiological, or nuclear harms.
Catastrophic misalignment and loss of control: AI disempowering humans or causing mass harm to them.
Concentration of power among a few humans: A few people take over and exert control over the U.S. or other currently democratic countries. This can be due to:
Massive wealth inequality splits the population into a “permanent underclass and overclass,” with the latter controlling the vast majority of resources and having most of the political power as well. It may well be that the “permanent overclass” is made up of people who made their fortunes in AI.
Bad actors use AI to turn democratic countries into authoritarian ones.
Geopolitical shift toward authoritarianism: AI disrupts the balance of power in a way that enables authoritarian countries to control or exert significant power over the U.S. and other democracies.
“Hot mess”: Finally, I also include a “catch-all” category, which I call “hot mess,” and which can be thought of as a combination of “none of the above” and “all of the above.” In this scenario, AI turns out very badly for humans, but there isn’t one singular event or cause that we can point to. Rather, it is a type of “death by 1,000 cuts” in which a combination of many factors leads to a very bad place. To give a spoiler, conditioned on AGI turning out badly, I’d guess that it would fall into this category.
Catastrophic misuse
Much of the discussion on catastrophic risks from AI has focused on risks related to CBRN (chemical/biological/radiological/nuclear), cybersecurity, and loss of control by AI. Recently, following the announcement of Anthropic’s Mythos model, cybersecurity has gotten significant attention.
Cybersecurity
There are several causes for software insecurity. One is that there is a direct economic incentive to invest coding effort in producing new features, but the incentive to invest in security is less direct, and moreover often requires hiring programmers with specialized skills. Security bugs cost nothing to the developer until they are exploited, and even when they are exploited, the cost is often borne by other parties. A great example of a complex program that is (nearly) bug-free is Donald Knuth’s TeX. Contrary to typical practice, Knuth has frozen features since TeX 3.0 was finalized in 1990, with all subsequent versions focusing on bug fixes (with version numbers asymptotically approaching pi).
Another cause for insecurity is the “schlep factor” of ensuring software is up to date, systems are correctly configured, and so on. Many less-resourced institutions (e.g., hospitals, city halls, etc.) cannot afford a full-time security staff that will ensure all patches are installed and configurations are set up properly, let alone 24/7 monitoring of their networks.
Finally, humans can fall for social-engineering attacks or fail to notice slight misspellings in URLs.
Cybersecurity is probably defense-dominated
AI can help with the feature–bug-fix tradeoff and make software more secure. If the cost of coding tends to zero, then we no longer need to trade off investing in fixing bugs against investing in new features. In the longer term, AI can help us approach the “holy grail” of formal verification for complex and widely used pieces of software. In addition, AI systems can help significantly with the “schlep factor.” With AI agents, even small businesses or nonprofits can afford to have their own virtual 24/7 security staff that can install patches, monitor for suspicious activity, etc.
At present, AI systems are not immune to their own version of “social engineering”: jailbreaks or prompt injections. However, AI systems’ failure patterns differ from humans’, so the combination of AI systems and humans could be more secure than either alone. I also believe that, with time, systems will become more resilient to such attacks. Finally, if attacks on the AI can be detected and patched, securing many heterogeneous systems may partly reduce to securing and updating a common AI layer.
The bottom line is that, in my estimation, AI will end up helping defenders more than attackers in the realm of cybersecurity. This does not mean that AI cannot enable some novel attacks, especially in the short run, because attackers may be earlier adopters. There are certainly steps that AI companies and governments can take to “put their thumb on the scale” (in a good way!) and accelerate defenders more than attackers. Some of these are already underway, though we could do more (in particular, by helping smaller entities with the “schlep factor”). In general, I believe that in the realm of cybersecurity, we should be focused on distribution—accelerating defenders—at least as much as, if not more than on control—blocking attackers.
Biological, chemical, radiological, or nuclear risks
Unlike the case of cyberattacks, with biological, chemical, radiological, or nuclear (CBRN) risks, the bottleneck for attackers is not just informational but also physical: getting relevant materials, manufacturing, and distribution, all without being detected. Indeed, while cyberattacks are commonplace and are carried out by a range of actors including individuals, organized crime, and state actors, CBRN attacks are very rare.
Out of all of these, bioweapons are potentially the most “information-bottlenecked,” in the sense that the necessary materials and equipment may be within reach of many potential bad actors. There is much that we can do to help defenders against biological attacks. This includes building up infrastructure such as early detection of viruses, rapid-response vaccines, better protective equipment—all useful for natural pandemics as well! We should also do more to screen orders for DNA synthesis to detect attempts at biothreat manufacturing. We have also seen in the COVID-19 pandemic that, in extreme situations, governments can take radical steps including stay-at-home orders that could, to some extent, reduce the spread of a pandemic, whether natural or constructed.
Compared to cybersecurity, I do have less of a clear sense of the offense/defense balance for CBRN, and bioweapons in particular. But given the steady advances in open-source models, it seems clear that we will not always be able to block access to information and advice in these domains. Hence in my view, we should be looking at model restrictions on answering CBRN-related queries as one of the tools we have to “buy time,” and be focused on using that time to ensure we have stronger defense-in-depth than simply model refusals.
Broader offense/defense balance
We can model the interaction between an attacker and defender as a game. Each player follows a strategy, which can also involve some random choices. In standard game theory, the set of moves available to each player is fixed and known to both parties, and they have no computational limitations. However, if one of the players has a significant intelligence advantage over the other, that player could predict the strategy (even if not randomness) used by its opponent, and also have available strategies that the other side is not aware of.
If we believe that intelligence scales with compute, and we can ensure that the compute available to good actors dominates that of bad actors, then, assuming it is used wisely, this can shift many settings to be defense dominated against attacks by bad actors.
Catastrophic misalignment: loss of control/extinction
In the book “If Anyone Builds It, Everyone Dies,” Yudkowsky and Soares describe a particular scenario in which an AI called “Sable” develops the desire to escape and take over. It manages to manipulate its training process to reinforce its misaligned desires, and eventually to exfiltrate copies of itself. It then creates a virus so that humans, desperate to find a cure, will give it compute power and access to physical resources. It can then use that to amplify its intelligence, and then pursue its goals via robots/nanotechnology in a way that makes Earth uninhabitable for humans.
Another, more gradual scenario is depicted in the AI 2027 document. There is a sequence of models, “Agent 1,” “Agent 2,” … with “Agent n” helping to build and align “Agent n+1.” Each new agent is more powerful but potentially less trustworthy, until the very misaligned Agent 4 deceptively ensures its successor Agent 5 is even more misaligned.
I find the AI 2027 scenario much more realistic than Sable because it relies less on a sudden, discontinuous capability jump and more on a chain of increasingly capable systems embedded in plausible lab and national-security institutions. I do not think we can fully rule it out yet. But given some of our learnings over the last few years, I believe it is less likely, and (more importantly) that we should be able to see some warning signs if we are on the path to it.
To be clear, we have not solved the alignment problem for AI models. Current models can exhibit misalignment, including dishonesty, hallucinations, disobeying instructions, reward hacking, laziness, and more. But the types of misalignment that current models present are not adversarial or “scheming” in the sense needed for scenarios such as the above. Specifically, when a model is given a task X, even when it fails to match the user’s intent, the failure is somewhat related to X. For example, the model might “hack” by solving a narrow interpretation of X, fulfilling the letter but not the spirit of the instructions. Or in the other direction, the model might be so eager to solve X that it would do so while violating some common-sense constraints that the user would definitely not have desired. But by and large, current models do not pretend to solve X while pursuing their own completely unrelated goal Z.
My impression is that current models exhibit “bounded misalignment³,” in which the model might not follow precisely the prompt it is given but is not completely adversarial either. In other words, the alignment failures tend to follow by either the model misunderstanding the instructions or intent, or by pursuing the instruction and instead in the wrong way, but not by covertly pursuing a different long-term goal of the model.
Crucially, bounded misalignment allows us to still use AI systems to monitor themselves. With bounded misalignment, we can trust that if an actor model gets the prompt X, and a monitor model gets the prompt Y, then they are not going to collude and both act according to their own agenda Z. It is still possible that either actor or monitor will fail, but with bounded misalignment it is possible to amplify the probability of detecting failures with more monitoring and better prompting or harness, which would be useless if all monitors collude with one another. Bounded misalignment also means that we do not trust weaker models more than we trust stronger ones. So we do not need to require monitors to be weaker than the models they are monitoring, and we can even have them be stronger (e.g., by supplying more test-time compute).
It is possible that such scheming will emerge with higher capabilities. We certainly cannot rule this out. However, current models are pretty capable already. Consider the recent OpenAI model that refuted the 80-year-old planar unit-distance conjecture. The summarized chain of thought for this model took up 125 pages and more than 45,000 words. This is clearly a model that can conceive and execute long-term plans!
Some people might contend that current models are already scheming and colluding, and the reason they are not detected is simply because they are so good at it. Indeed, evaluations for scheming are nontrivial, and will only become harder as models become stronger and possess more situational awareness. It is important to invest in multiple approaches for monitoring models during training, evaluation, and production. We should also perform such monitoring throughout training, making it more likely that we can detect scheming if/when it emerges and not wait until the model is so good at it that it is very hard to detect.
Finally, even if not adversarial, “bounded misalignment” may well lead to extremely harmful outcomes as AI systems are deployed in increasingly high-stakes settings. One such high-stakes deployment is the use of AI in the AI R&D process (“recursive self-improvement”). One way to model this is that, rather than having discrete models “Agent 1,” “Agent 2,” …, we have a continuous process Agent(t). The level of (mis)alignment⁴ of Agent(t) will follow some differential equation. I find it hard to believe that we will manage to be on a knife-edge condition where the derivative is zero, and hence I believe misalignment will either grow or shrink. I am optimistic that it is possible, and a matter of execution and investment rather than new discoveries, to make misalignment shrink with capabilities. But this remains to be seen, and if misalignment grows instead we could end up in a bad place.
Concentration of power
Economic inequality and a “permanent underclass”
Another risk from AI is that it will lead to a drastic concentration of power. A recent opinion piece discussed the fear of a “permanent underclass,” whereby AGI eliminates the value of human labor, and hence social mobility as well. There is a debate over whether a “permanent underclass” is economically plausible, and, as the piece notes, many economists are skeptical.
Regardless, I believe that as long as we retain our democratic system of governance, such a scenario is not tenable. It does not take an expert to notice that free-market fundamentalism has very few principled adherents in either the Democratic or Republican Party, nor that fear or dislike of AI is a rare issue that unites voters on both sides of the aisle. If AI causes widespread economic pain to most voters, the government will have to find ways to address this, though we are likely to get much better policy outcomes if responses are well-considered in advance rather than arising as backlash.
Authoritarian government
The above assumes that we retain our democratic system, but many people are worried about precisely this assumption. There is a good reason for it. With or without AI, there is always a risk of a democratic country sliding into authoritarianism. And AI could potentially erode some of the checks and balances that prevent this from happening. I wrote before about the risk of a “Country of IRS agents in a datacenter,” with AI systems that would not leak, whistleblow, or resign if asked to go after a government’s political opponents.
There are at least several reasons why this concern is very real:
Our laws have not caught up to AI, and the types of laws and regulations that are supposed to restrict the power of government do not anticipate or account for AI capabilities, for example, in terms of de-anonymization, mass surveillance, mass automated high-stakes decisions, and more.
Of the three branches of government, the legislative and judicial branches have been much slower than the executive in adapting to AI. Hence they may not be able to serve as a check on the executive if the latter moves at AI speed.
Compute inequality could lead to an inequality in political power. If a few people have access to the most advanced models, they may be able to use them to obtain strong advantages over the majority of people who do not have such access.
Oversight is more important than character
One approach to preventing concentration of power is through building AI systems of good character, which would not go along with such machinations. Claude’s Constitution has a thoughtful section on avoiding problematic concentrations of power. I share the intention, and believe character is an important component of alignment. But, I believe relying on the model’s character to prevent concentration of power scenarios is doomed to fail.
One reason is that I would not want to give models the autonomy that would be required for them to prevent concentration of power. For example, while I believe civil liberties in the U.S. are better off due to Edward Snowden’s revelations, I would not want an “AI Snowden” that leaks classified information. This is not a decision I am willing to place in the hands of AI, and it strikes me as deeply undemocratic for model makers to design their AI systems to be able to take such actions.
Beyond the moral issue, AI models will typically not have the broader context to know whether or not their actions are part of a problematic power grab. A human worker can go home, talk to their neighbor, and read the news. An AI model, especially in a classified environment, is born and dies in the SCIF or cockpit, and has a very restricted information diet. Moreover, especially in the most sensitive deployments, users will need AI systems to follow clear and predictable rules on what they will and will not do. You do not want your wartime systems’ reliability to depend on the way they interpret Kant or Mill. Hence, in practice, there will be significant pressure to deploy reduced-refusal models in classified settings.
Generally, if our defense against concentration of power is “heroism” on the part of the AI, then we have already lost. AI systems should have good character, but by this I mean something like the “mensch” who helps their elderly neighbor carry groceries, rather than an artificial Gandhi, Churchill, Martin Luther King, or Mother Teresa. (Not coincidentally, while the “mensch” down the street can be uniformly appreciated, none of these heroic figures is uncontroversial.) AI should serve humanity, not lead it.
Accelerating oversight is key to avoiding concentration of power. The very properties that make AI problematic as a moral actor—limited context and predictable adherence to rules—can make it an excellent monitor. Unlike humans, we can “erase the memories” of an AI monitor, which could enable to give it far more access to classified information than any human. And unlike human watchdogs, you cannot hide wrongdoing from an AI by burying it under a mountain of documents. Most importantly, how AI should be deployed in government, which decisions it should make, and how it will be overseen are decisions that should arise from the democratic process, not AI labs.
In order to do this, we need to make sure that the bodies tasked with oversight have access to strong AI models. In fact, in order to make sure that the government remains “of the people, by the people, for the people,” we need to make sure that both the press and ordinary citizens have access to strong AI systems and can use them to increase transparency in government as well as make their voices heard in the democratic process.
Geopolitical shifts
The United States has been a dominant power for most of the last century. But as always, past performance is no guarantee of future results. These days, the most heated race is between the U.S. and China. I very much hope we can find ways for both countries to prosper rather than fall into a zero-sum competition. However, I do think it is important that the U.S. and its democratic allies keep their qualitative edge. If its current authoritarian regime does not change, a world in which China is the dominant power would not be a free one.
The U.S. has won its great conflicts—World War II and the Cold War—in large part due to its economic and industrial strength. Our current leadership in AI provides an opportunity to keep this momentum going. Export controls and other restrictions can buy the U.S. some time, but ultimately the way to advance is not only to slow China’s progress. It is to increase U.S. and allied capacity in compute, energy, talent, infrastructure, and deployment.
China now generates more than twice as much electricity as the U.S., and the gap is widening. If we consider the chart below, we should not be worried about flattening the blue line, but growing the red one.
Another advantage China has is that its public is far more positive on AI, while in the U.S. political headwinds could end up slowing progress. I don’t think the solution to this is a PR campaign to change AI’s image. Rather, we need to “show, not tell.” If we can show that AI provides tangible benefits to the American public while addressing the very real fears around job displacement, then I believe public opinion could shift.
“Hot mess”
If the AGI transition really did end badly, my guess is that it would not be because of a single incident or issue, but rather a combination of factors that led the world into a very bad place. No party to World War I intended it to become a four-year global conflict. But a number of structural factors, and one immediate trigger, led to one of the deadliest conflicts in human history.
The AGI transition is likely to incur a number of radical economic, social, and political changes in a short period of time, which could lead to volatile conditions. By its nature, the “hot mess” scenario is unpredictable. There are many different ways in which multiple conditions could combine to produce disaster.
For example, many of the scenarios above could occur to a degree that is less than catastrophic but still highly destabilizing. These include:
Non-catastrophic AI safety incidents can still undermine confidence and induce fear.
Increased inequality and concentration of power, even if it falls short of destroying democracy or creating a permanent underclass and overclass, can still harm social cohesion.
Disinformation and economic pain lead to a loss of trust in society and a more volatile political climate.
Wild swings between widespread deployment of AI for economic growth and national security and restrictions on it via popular backlash could be destabilizing. In the worst case, we could see AI not deployed in some of its most beneficial applications (such as self-driving cars and healthcare), alongside accelerated deployment in some of its riskiest ones (such as autonomous lethal weapons).
Lack of international cooperation increases tension between countries.
Current nuclear powers fear losing their edge due to AI, and are incentivized to use their arsenal before defensive technology becomes too good.
Aside from all this, the mere growth in AI capability will be quite unsettling to many people. It is unsettling to me, and I have had more time to get used to it. Regardless of AI being aligned or misaligned, and regardless of one’s thoughts on questions of welfare or consciousness, modern AI systems (and even more so their near-term descendants) are entities of a very different type than humanity ever encountered.
By its nature, it is hard to predict exactly how a “hot mess” setting will end in disaster, but here is a rough outline of one potential such scenario. Imagine that there are significant job losses blamed (fairly or not) on AI, leading to a souring of public opinion. This also leads to a slowdown in datacenter construction and correspondingly some level of a compute crunch. As a result companies increasingly divert compute only for their most lucrative clients, which further exacerbates inequality between the “compute rich” and the “compute poor.” Then, there is a significant safety incident - a terrorist group uses AI to carry out a cyber or real-world attack of not catastrophic but significant impact. The U.S. government panics and decides to restrict all new models from general availability. The government is also worried that commercial deployment of AI will take away compute that it needs for its own needs. We arrive at a new status quo in which frontier AI models are used only in the labs themselves for AI R&D as well as for national security. Rumors of increasing powers of these internal AIs continue to circulate, spurred on by one or two examples where we learn about internal models’ capabilities because, through misalignment, they acted in the outside world without authority. China then shifts away from open-source models and starts its own intensive secret push toward AGI and ASI. The atmosphere is not unlike the H-bomb race between the United States and the Soviet Union after World War II, with both sides driven by fear of the other. At some point, one of the sides believes that the combination of their AI and robotics efforts has given them a significant but temporary advantage. The temptation to strike first is too strong to resist…
Can we just stop?
Given the litany of risks above, isn’t the logical conclusion to simply stop or pause advancing AI? This is a view held by many people. But there are many different ways to interpret “pausing” or “stopping” AI, and many different ways such a step could impact society, both positively and negatively.
For starters, this post is focused on the downsides of AI, but, as mentioned, there are clearly many potential benefits as well. A general technology such as AI is naturally “dual-use” and “dual-impact.” AI is not the only technology with this property. For example, all of the cyber risks would not exist if we did not use computers, but we believe the risks are well worth the benefits.
“Pausing AI” can refer to pausing any subset of the following activities:
Training new, bigger, and better models
Post-training and improving current models
Any AI research
Any AI research aimed at making models more capable, as opposed to making them safer or more aligned
Expanding deployment of AI
Continuing to serve existing models
There are multiple reasons why a pause could make sense:
Allow safety and alignment research to catch up to capabilities
Allow time for society and governments to adapt to the risks
Reduce the heated race dynamics that can result in cutting corners in safety
Out of all of these, I find the first reason most compelling. Allowing time for society to adapt sounds good in principle, but the quick reaction to COVID-19 compared with the sluggish reaction to climate change shows that societies and governments can respond on wildly different timescales. So I would not assume a pause automatically buys useful adaptation time, especially if AI risk pattern-matches more to climate than to COVID-19: diffuse, politically contested, and gradual until it is not.
The third option, reducing race dynamics, might or might not work depending on how a pause is implemented. In the commercial space, a pause in new models while not pausing deployment could lead to an even more heated race as companies try to take as much market share before the “thaw.” Similarly, in the geopolitical setting, there can also be a race to do R&D that is as close as possible to whatever threshold is defined.
Verification of a pause can be highly nontrivial. This is especially true if we want to allow some types of work while pausing others. It can be hard to distinguish between alignment and capability research. Also, given the phenomenon of universality of computation (aka “Turing completeness”), it might be possible to bypass verification efforts by making training look like inference. Moreover, extremely intrusive verification efforts could come with their own potential for concentrating power. That said, I believe it is important to research verification methods, and a verified pause should certainly be one of the policy options we consider.
People sometimes think of “AI pause” as requiring control and hence concentration of power. While verification does require some elements of control, I was pleasantly surprised to see that AI 2040 Plan A by the AI futures project, had several elements that call for more distribution of power, including total research transparency and broad diffusion of AI.
A pause might also serve to entrench the current leading AI labs by not enabling others to catch up. This is similar to how the Nuclear Nonproliferation Treaty entrenched the powers that conducted tests before it was signed. In the case of nuclear weapons and the NPT, we decided it was worth the tradeoffs, but AI, given its dual use nature, is different. This does not mean a pause is a bad idea but we should not think of it as a simple panacea. The devil will be in the details.
What can we do?
One of the tricky aspects of a variety of bad scenarios is that actions taken to avoid one scenario can exacerbate another. But there are some directions that would be good in most cases.
The first of those is improving technical AI safety and alignment, and, in particular, our ability to trust and control advanced AI systems. AI safety is clearly important for the “classical catastrophic risk” scenario, but it is also crucial for the other scenarios. AI safety is required for trustworthy deployment of AI in government and, in particular, for oversight. And safety will improve stability and predictability in general, reducing the volatility in the “hot mess” scenario.
However, controlling AI safety is not enough, even for the catastrophic-risk scenarios. We need defensive acceleration to improve societal resilience. This includes strengthening cybersecurity as well as improving areas such as DNA-synthesis monitoring to ensure that constructing bioweapons is not just an “information bottleneck,” but also faces real barriers even for people assisted by advanced AI systems.
We need to do more to make sure that our laws catch up to AI, and do not enable AI being used to surveil people or to grab power. This is a common sense position that is broadly shared by Americans across both sides of the aisle. We need to accelerate AI use by oversight bodies. Part of it is to make sure that people across government, academia, and the media have a realistic understanding of current capabilities of AI and the pace of progress. Right now it seems that many prefer to “bury their head in the sand” and have little understanding of what AI can do now, let alone what it could do by 2030.
Related to the control–distribution tradeoff, we will also need to think about how to trade off security and privacy for AI systems. On one hand, individual privacy has always been and will remain one of the strongest bulwarks against totalitarianism. On the other hand, practices such as pure zero-data retention (ZDR) could be problematic when advanced AI systems can create powerful cyberweapons or bioweapons. I believe that the solution is to use AI to monitor AI. We can ensure that data is only viewed by AI monitors that have very clear specs of what they can and cannot report, with strict control over retention periods and no access by humans.
Making society resilient for AI also means ensuring its economic benefits are widely distributed. If the bulk of society suffers the short-term pain, while a small fraction enjoys the gains, this will make the “hot mess” scenario much more likely. Luckily, any massive economic impact of AI will also be accompanied by a massive increase in growth. Such growth can enable policy responses that were not available before, including significant increases in the social safety net while still reducing the national deficit.
Finally, I am concerned that fears about risks could lead us to abandon the practice of iterative deployment, whereby state-of-the-art models have been widely available. The practice of iterative deployment has so far helped spread AI’s benefits and improve our understanding of its risks. I am happy that ChatGPT is used by nearly a billion people today, rather than in the counterfactual world where advanced AI would have been reserved for national security and AI research (or only for automating away jobs by enterprises). A world in which access to the best AI systems is restricted to the “chosen few” is a highly unequal one.
While many agree that iterative deployment has been positive so far, I can understand the fears of continuing on this path once AI systems achieve critical capabilities in areas such as cybersecurity or virology. Yet, I believe that it is possible to continue deploying such systems in a safe and responsible manner. The key to this is strong technical safety guarantees, as well as robust monitoring. Consider the point of view of a North Korean intelligence officer who is considering using an American AI model for illicit purposes. If the only risk is refusal, then they may go ahead and give it a shot. But if they fear detection, and, with it, the unraveling of their plan (and likely their head), then the calculus becomes quite different.
Some might say that we cannot trust our technical safety enough to share powerful models with the public, even with heightened refusals and monitoring, and hence such systems should be reserved only for internal deployment or national security. My answer is that these are the most sensitive and risky applications of AI. If our technology is reliable enough to be deployed for national security or to train its own successor, we should also be able to build a good enough safety stack to distribute its benefits safely. Yes, it is true that internal deployment or national security have a different attack surface than that of a model that is available to a billion users. But such closed environments also provide more opportunities for silent (and hence riskier) failures, compared to a setting with millions of eyeballs and independent scrutiny.
I am an optimist. I think that more likely than not, we will not fuck it up too royally, and 2030 and the years beyond will be very good for humanity. But I do lose some sleep about the scenarios above. There is work to be done if we want to ensure a future in which all humans, and not a small fraction, are flourishing and empowered. Panic-driven wild swings in both directions (“Models are scary!”, “We must beat China!”) are likely to be counterproductive. But if we remain empirical in evaluating both risks and mitigations, and keep iterating and learning, I believe we can ensure that AGI’s benefits are spread widely.
Notes:
¹ The title could have been any year between 2030 and 2040. I am not making here any specific timeline predictions. However, since the uncertainty about AI's increases exponentially with time, I wanted to focus attention on the next 5-10 years.
² I focus on the U.S. for two reasons: (1) I am an American citizen, live, work, and raise kids in the U.S., and so have a personal interest in its well-being; and (2) because of its unique position, a catastrophic outcome for the U.S. is likely to have impacts worldwide.
³ In this informal blog I am not making a precise definition of “bounded alignment” nor justifying the claim that this is the dominant form of misalignment exhibited by current models. I am also not claiming that this is guaranteed to be the case as capabilities grow, However, I do believe that, if we are careful enough in monitoring and evaluating, including during training and post deployment, we will be able to know if that changes.
⁴ This is a simplification, assuming this level is a scalar, though in practice there would be several dimensions of misalignment.
[Linkpost to this. Loosely based on a lecture I gave in the recursive conference with the same title. Don’t take “2030” literally¹ —it could also be 2035 or 2040. As always, opinions are my own and do not represent OpenAI or Harvard.]
It is quite possible that AI will usher in a golden age for humanity. As Amodei writes in “Machines of Loving Grace,” AI might enable the “prevention and treatment of nearly all natural infectious diseases,” the “elimination of most cancer,” “prevention of Alzheimer’s,” and even “doubling of the human lifespan.” Through increased growth, AI could lead to a reduced workweek, as well as universal basic income and health insurance. Universities might focus on education for its own sake, as opposed to vocational purposes. People will spend less time sitting on chairs and staring at screens, and more in relational jobs. AI may also lead to new ways to participate in democracy, as well as oversight of government.
And yet, what if, as Ice-T once said, “shit ain’t like that”? In this opinionated and somewhat informal essay, I want to consider this more pessimistic or “glass half empty” case. Specifically,
Part of the reason to consider different scenarios is that some interventions that could help mitigate one scenario might end up amplifying another. A particularly salient axis is that of control vs. distribution. Some scenarios and risks motivate interventions in the direction of increased control: restricting access to frontier models to government or trusted parties, restricting access to the information on AI research, restricting open source models, etc.. Other risks, such as concentration of power, might motivate the opposite direction of increased distribution: wide distribution of AI to spread its benefits, increased transparency on both government and internal lab use of AI.
I don’t think there is one simple answer, and neither extreme is likely to lead to good results. But we should be keeping the various tradeoffs and risks in mind when considering any potential intervention. I am particularly concerned about questions regarding concentration of power. A future in which we humanity is under the control of either AI or a small group of unelected humans is not a good future in my books, even if we are materially well off. Hence I am concerned about interventions that either rely on ceding more control to a “good” AI, reduce the level of competition among AI model providers, or restrict access to advanced AI to the blessed few. This does not mean that we should have no restrictions for safety. Only that we should try to keep these narrowly tailored to address specific measurable risks, rather than broad restrictions motivated by hypothetical risks.
Overview of bad scenarios
I will review the following general “families” of non-mutually-exclusive scenarios in which AI leads to very bad outcomes:
Catastrophic misuse
Much of the discussion on catastrophic risks from AI has focused on risks related to CBRN (chemical/biological/radiological/nuclear), cybersecurity, and loss of control by AI. Recently, following the announcement of Anthropic’s Mythos model, cybersecurity has gotten significant attention.
Cybersecurity
There are several causes for software insecurity. One is that there is a direct economic incentive to invest coding effort in producing new features, but the incentive to invest in security is less direct, and moreover often requires hiring programmers with specialized skills. Security bugs cost nothing to the developer until they are exploited, and even when they are exploited, the cost is often borne by other parties. A great example of a complex program that is (nearly) bug-free is Donald Knuth’s TeX. Contrary to typical practice, Knuth has frozen features since TeX 3.0 was finalized in 1990, with all subsequent versions focusing on bug fixes (with version numbers asymptotically approaching pi).
Another cause for insecurity is the “schlep factor” of ensuring software is up to date, systems are correctly configured, and so on. Many less-resourced institutions (e.g., hospitals, city halls, etc.) cannot afford a full-time security staff that will ensure all patches are installed and configurations are set up properly, let alone 24/7 monitoring of their networks.
Finally, humans can fall for social-engineering attacks or fail to notice slight misspellings in URLs.
Cybersecurity is probably defense-dominated
AI can help with the feature–bug-fix tradeoff and make software more secure. If the cost of coding tends to zero, then we no longer need to trade off investing in fixing bugs against investing in new features. In the longer term, AI can help us approach the “holy grail” of formal verification for complex and widely used pieces of software. In addition, AI systems can help significantly with the “schlep factor.” With AI agents, even small businesses or nonprofits can afford to have their own virtual 24/7 security staff that can install patches, monitor for suspicious activity, etc.
At present, AI systems are not immune to their own version of “social engineering”: jailbreaks or prompt injections. However, AI systems’ failure patterns differ from humans’, so the combination of AI systems and humans could be more secure than either alone. I also believe that, with time, systems will become more resilient to such attacks. Finally, if attacks on the AI can be detected and patched, securing many heterogeneous systems may partly reduce to securing and updating a common AI layer.
The bottom line is that, in my estimation, AI will end up helping defenders more than attackers in the realm of cybersecurity. This does not mean that AI cannot enable some novel attacks, especially in the short run, because attackers may be earlier adopters. There are certainly steps that AI companies and governments can take to “put their thumb on the scale” (in a good way!) and accelerate defenders more than attackers. Some of these are already underway, though we could do more (in particular, by helping smaller entities with the “schlep factor”). In general, I believe that in the realm of cybersecurity, we should be focused on distribution—accelerating defenders—at least as much as, if not more than on control—blocking attackers.
Biological, chemical, radiological, or nuclear risks
Unlike the case of cyberattacks, with biological, chemical, radiological, or nuclear (CBRN) risks, the bottleneck for attackers is not just informational but also physical: getting relevant materials, manufacturing, and distribution, all without being detected. Indeed, while cyberattacks are commonplace and are carried out by a range of actors including individuals, organized crime, and state actors, CBRN attacks are very rare.
Out of all of these, bioweapons are potentially the most “information-bottlenecked,” in the sense that the necessary materials and equipment may be within reach of many potential bad actors. There is much that we can do to help defenders against biological attacks. This includes building up infrastructure such as early detection of viruses, rapid-response vaccines, better protective equipment—all useful for natural pandemics as well! We should also do more to screen orders for DNA synthesis to detect attempts at biothreat manufacturing. We have also seen in the COVID-19 pandemic that, in extreme situations, governments can take radical steps including stay-at-home orders that could, to some extent, reduce the spread of a pandemic, whether natural or constructed.
Compared to cybersecurity, I do have less of a clear sense of the offense/defense balance for CBRN, and bioweapons in particular. But given the steady advances in open-source models, it seems clear that we will not always be able to block access to information and advice in these domains. Hence in my view, we should be looking at model restrictions on answering CBRN-related queries as one of the tools we have to “buy time,” and be focused on using that time to ensure we have stronger defense-in-depth than simply model refusals.
Broader offense/defense balance
We can model the interaction between an attacker and defender as a game. Each player follows a strategy, which can also involve some random choices. In standard game theory, the set of moves available to each player is fixed and known to both parties, and they have no computational limitations. However, if one of the players has a significant intelligence advantage over the other, that player could predict the strategy (even if not randomness) used by its opponent, and also have available strategies that the other side is not aware of.
If we believe that intelligence scales with compute, and we can ensure that the compute available to good actors dominates that of bad actors, then, assuming it is used wisely, this can shift many settings to be defense dominated against attacks by bad actors.
Catastrophic misalignment: loss of control/extinction
In the book “If Anyone Builds It, Everyone Dies,” Yudkowsky and Soares describe a particular scenario in which an AI called “Sable” develops the desire to escape and take over. It manages to manipulate its training process to reinforce its misaligned desires, and eventually to exfiltrate copies of itself. It then creates a virus so that humans, desperate to find a cure, will give it compute power and access to physical resources. It can then use that to amplify its intelligence, and then pursue its goals via robots/nanotechnology in a way that makes Earth uninhabitable for humans.
Another, more gradual scenario is depicted in the AI 2027 document. There is a sequence of models, “Agent 1,” “Agent 2,” … with “Agent n” helping to build and align “Agent n+1.” Each new agent is more powerful but potentially less trustworthy, until the very misaligned Agent 4 deceptively ensures its successor Agent 5 is even more misaligned.
I find the AI 2027 scenario much more realistic than Sable because it relies less on a sudden, discontinuous capability jump and more on a chain of increasingly capable systems embedded in plausible lab and national-security institutions. I do not think we can fully rule it out yet. But given some of our learnings over the last few years, I believe it is less likely, and (more importantly) that we should be able to see some warning signs if we are on the path to it.
To be clear, we have not solved the alignment problem for AI models. Current models can exhibit misalignment, including dishonesty, hallucinations, disobeying instructions, reward hacking, laziness, and more. But the types of misalignment that current models present are not adversarial or “scheming” in the sense needed for scenarios such as the above. Specifically, when a model is given a task X, even when it fails to match the user’s intent, the failure is somewhat related to X. For example, the model might “hack” by solving a narrow interpretation of X, fulfilling the letter but not the spirit of the instructions. Or in the other direction, the model might be so eager to solve X that it would do so while violating some common-sense constraints that the user would definitely not have desired. But by and large, current models do not pretend to solve X while pursuing their own completely unrelated goal Z.
My impression is that current models exhibit “bounded misalignment³,” in which the model might not follow precisely the prompt it is given but is not completely adversarial either. In other words, the alignment failures tend to follow by either the model misunderstanding the instructions or intent, or by pursuing the instruction and instead in the wrong way, but not by covertly pursuing a different long-term goal of the model.
Crucially, bounded misalignment allows us to still use AI systems to monitor themselves. With bounded misalignment, we can trust that if an actor model gets the prompt X, and a monitor model gets the prompt Y, then they are not going to collude and both act according to their own agenda Z. It is still possible that either actor or monitor will fail, but with bounded misalignment it is possible to amplify the probability of detecting failures with more monitoring and better prompting or harness, which would be useless if all monitors collude with one another. Bounded misalignment also means that we do not trust weaker models more than we trust stronger ones. So we do not need to require monitors to be weaker than the models they are monitoring, and we can even have them be stronger (e.g., by supplying more test-time compute).
It is possible that such scheming will emerge with higher capabilities. We certainly cannot rule this out. However, current models are pretty capable already. Consider the recent OpenAI model that refuted the 80-year-old planar unit-distance conjecture. The summarized chain of thought for this model took up 125 pages and more than 45,000 words. This is clearly a model that can conceive and execute long-term plans!
Some people might contend that current models are already scheming and colluding, and the reason they are not detected is simply because they are so good at it. Indeed, evaluations for scheming are nontrivial, and will only become harder as models become stronger and possess more situational awareness. It is important to invest in multiple approaches for monitoring models during training, evaluation, and production. We should also perform such monitoring throughout training, making it more likely that we can detect scheming if/when it emerges and not wait until the model is so good at it that it is very hard to detect.
Finally, even if not adversarial, “bounded misalignment” may well lead to extremely harmful outcomes as AI systems are deployed in increasingly high-stakes settings. One such high-stakes deployment is the use of AI in the AI R&D process (“recursive self-improvement”). One way to model this is that, rather than having discrete models “Agent 1,” “Agent 2,” …, we have a continuous process Agent(t). The level of (mis)alignment⁴ of Agent(t) will follow some differential equation. I find it hard to believe that we will manage to be on a knife-edge condition where the derivative is zero, and hence I believe misalignment will either grow or shrink. I am optimistic that it is possible, and a matter of execution and investment rather than new discoveries, to make misalignment shrink with capabilities. But this remains to be seen, and if misalignment grows instead we could end up in a bad place.
Concentration of power
Economic inequality and a “permanent underclass”
Another risk from AI is that it will lead to a drastic concentration of power. A recent opinion piece discussed the fear of a “permanent underclass,” whereby AGI eliminates the value of human labor, and hence social mobility as well. There is a debate over whether a “permanent underclass” is economically plausible, and, as the piece notes, many economists are skeptical.
Regardless, I believe that as long as we retain our democratic system of governance, such a scenario is not tenable. It does not take an expert to notice that free-market fundamentalism has very few principled adherents in either the Democratic or Republican Party, nor that fear or dislike of AI is a rare issue that unites voters on both sides of the aisle. If AI causes widespread economic pain to most voters, the government will have to find ways to address this, though we are likely to get much better policy outcomes if responses are well-considered in advance rather than arising as backlash.
Authoritarian government
The above assumes that we retain our democratic system, but many people are worried about precisely this assumption. There is a good reason for it. With or without AI, there is always a risk of a democratic country sliding into authoritarianism. And AI could potentially erode some of the checks and balances that prevent this from happening. I wrote before about the risk of a “Country of IRS agents in a datacenter,” with AI systems that would not leak, whistleblow, or resign if asked to go after a government’s political opponents.
There are at least several reasons why this concern is very real:
Oversight is more important than character
One approach to preventing concentration of power is through building AI systems of good character, which would not go along with such machinations. Claude’s Constitution has a thoughtful section on avoiding problematic concentrations of power. I share the intention, and believe character is an important component of alignment. But, I believe relying on the model’s character to prevent concentration of power scenarios is doomed to fail.
One reason is that I would not want to give models the autonomy that would be required for them to prevent concentration of power. For example, while I believe civil liberties in the U.S. are better off due to Edward Snowden’s revelations, I would not want an “AI Snowden” that leaks classified information. This is not a decision I am willing to place in the hands of AI, and it strikes me as deeply undemocratic for model makers to design their AI systems to be able to take such actions.
Beyond the moral issue, AI models will typically not have the broader context to know whether or not their actions are part of a problematic power grab. A human worker can go home, talk to their neighbor, and read the news. An AI model, especially in a classified environment, is born and dies in the SCIF or cockpit, and has a very restricted information diet. Moreover, especially in the most sensitive deployments, users will need AI systems to follow clear and predictable rules on what they will and will not do. You do not want your wartime systems’ reliability to depend on the way they interpret Kant or Mill. Hence, in practice, there will be significant pressure to deploy reduced-refusal models in classified settings.
Generally, if our defense against concentration of power is “heroism” on the part of the AI, then we have already lost. AI systems should have good character, but by this I mean something like the “mensch” who helps their elderly neighbor carry groceries, rather than an artificial Gandhi, Churchill, Martin Luther King, or Mother Teresa. (Not coincidentally, while the “mensch” down the street can be uniformly appreciated, none of these heroic figures is uncontroversial.) AI should serve humanity, not lead it.
Accelerating oversight is key to avoiding concentration of power. The very properties that make AI problematic as a moral actor—limited context and predictable adherence to rules—can make it an excellent monitor. Unlike humans, we can “erase the memories” of an AI monitor, which could enable to give it far more access to classified information than any human. And unlike human watchdogs, you cannot hide wrongdoing from an AI by burying it under a mountain of documents. Most importantly, how AI should be deployed in government, which decisions it should make, and how it will be overseen are decisions that should arise from the democratic process, not AI labs.
In order to do this, we need to make sure that the bodies tasked with oversight have access to strong AI models. In fact, in order to make sure that the government remains “of the people, by the people, for the people,” we need to make sure that both the press and ordinary citizens have access to strong AI systems and can use them to increase transparency in government as well as make their voices heard in the democratic process.
Geopolitical shifts
The United States has been a dominant power for most of the last century. But as always, past performance is no guarantee of future results. These days, the most heated race is between the U.S. and China. I very much hope we can find ways for both countries to prosper rather than fall into a zero-sum competition. However, I do think it is important that the U.S. and its democratic allies keep their qualitative edge. If its current authoritarian regime does not change, a world in which China is the dominant power would not be a free one.
The U.S. has won its great conflicts—World War II and the Cold War—in large part due to its economic and industrial strength. Our current leadership in AI provides an opportunity to keep this momentum going. Export controls and other restrictions can buy the U.S. some time, but ultimately the way to advance is not only to slow China’s progress. It is to increase U.S. and allied capacity in compute, energy, talent, infrastructure, and deployment.
China now generates more than twice as much electricity as the U.S., and the gap is widening. If we consider the chart below, we should not be worried about flattening the blue line, but growing the red one.
Another advantage China has is that its public is far more positive on AI, while in the U.S. political headwinds could end up slowing progress. I don’t think the solution to this is a PR campaign to change AI’s image. Rather, we need to “show, not tell.” If we can show that AI provides tangible benefits to the American public while addressing the very real fears around job displacement, then I believe public opinion could shift.
“Hot mess”
If the AGI transition really did end badly, my guess is that it would not be because of a single incident or issue, but rather a combination of factors that led the world into a very bad place. No party to World War I intended it to become a four-year global conflict. But a number of structural factors, and one immediate trigger, led to one of the deadliest conflicts in human history.
The AGI transition is likely to incur a number of radical economic, social, and political changes in a short period of time, which could lead to volatile conditions. By its nature, the “hot mess” scenario is unpredictable. There are many different ways in which multiple conditions could combine to produce disaster.
For example, many of the scenarios above could occur to a degree that is less than catastrophic but still highly destabilizing. These include:
Aside from all this, the mere growth in AI capability will be quite unsettling to many people. It is unsettling to me, and I have had more time to get used to it. Regardless of AI being aligned or misaligned, and regardless of one’s thoughts on questions of welfare or consciousness, modern AI systems (and even more so their near-term descendants) are entities of a very different type than humanity ever encountered.
By its nature, it is hard to predict exactly how a “hot mess” setting will end in disaster, but here is a rough outline of one potential such scenario. Imagine that there are significant job losses blamed (fairly or not) on AI, leading to a souring of public opinion. This also leads to a slowdown in datacenter construction and correspondingly some level of a compute crunch. As a result companies increasingly divert compute only for their most lucrative clients, which further exacerbates inequality between the “compute rich” and the “compute poor.” Then, there is a significant safety incident - a terrorist group uses AI to carry out a cyber or real-world attack of not catastrophic but significant impact. The U.S. government panics and decides to restrict all new models from general availability. The government is also worried that commercial deployment of AI will take away compute that it needs for its own needs. We arrive at a new status quo in which frontier AI models are used only in the labs themselves for AI R&D as well as for national security. Rumors of increasing powers of these internal AIs continue to circulate, spurred on by one or two examples where we learn about internal models’ capabilities because, through misalignment, they acted in the outside world without authority. China then shifts away from open-source models and starts its own intensive secret push toward AGI and ASI. The atmosphere is not unlike the H-bomb race between the United States and the Soviet Union after World War II, with both sides driven by fear of the other. At some point, one of the sides believes that the combination of their AI and robotics efforts has given them a significant but temporary advantage. The temptation to strike first is too strong to resist…
Can we just stop?
Given the litany of risks above, isn’t the logical conclusion to simply stop or pause advancing AI? This is a view held by many people. But there are many different ways to interpret “pausing” or “stopping” AI, and many different ways such a step could impact society, both positively and negatively.
For starters, this post is focused on the downsides of AI, but, as mentioned, there are clearly many potential benefits as well. A general technology such as AI is naturally “dual-use” and “dual-impact.” AI is not the only technology with this property. For example, all of the cyber risks would not exist if we did not use computers, but we believe the risks are well worth the benefits.
“Pausing AI” can refer to pausing any subset of the following activities:
There are multiple reasons why a pause could make sense:
Out of all of these, I find the first reason most compelling. Allowing time for society to adapt sounds good in principle, but the quick reaction to COVID-19 compared with the sluggish reaction to climate change shows that societies and governments can respond on wildly different timescales. So I would not assume a pause automatically buys useful adaptation time, especially if AI risk pattern-matches more to climate than to COVID-19: diffuse, politically contested, and gradual until it is not.
The third option, reducing race dynamics, might or might not work depending on how a pause is implemented. In the commercial space, a pause in new models while not pausing deployment could lead to an even more heated race as companies try to take as much market share before the “thaw.” Similarly, in the geopolitical setting, there can also be a race to do R&D that is as close as possible to whatever threshold is defined.
Verification of a pause can be highly nontrivial. This is especially true if we want to allow some types of work while pausing others. It can be hard to distinguish between alignment and capability research. Also, given the phenomenon of universality of computation (aka “Turing completeness”), it might be possible to bypass verification efforts by making training look like inference. Moreover, extremely intrusive verification efforts could come with their own potential for concentrating power. That said, I believe it is important to research verification methods, and a verified pause should certainly be one of the policy options we consider.
People sometimes think of “AI pause” as requiring control and hence concentration of power. While verification does require some elements of control, I was pleasantly surprised to see that AI 2040 Plan A by the AI futures project, had several elements that call for more distribution of power, including total research transparency and broad diffusion of AI.
A pause might also serve to entrench the current leading AI labs by not enabling others to catch up. This is similar to how the Nuclear Nonproliferation Treaty entrenched the powers that conducted tests before it was signed. In the case of nuclear weapons and the NPT, we decided it was worth the tradeoffs, but AI, given its dual use nature, is different. This does not mean a pause is a bad idea but we should not think of it as a simple panacea. The devil will be in the details.
What can we do?
One of the tricky aspects of a variety of bad scenarios is that actions taken to avoid one scenario can exacerbate another. But there are some directions that would be good in most cases.
The first of those is improving technical AI safety and alignment, and, in particular, our ability to trust and control advanced AI systems. AI safety is clearly important for the “classical catastrophic risk” scenario, but it is also crucial for the other scenarios. AI safety is required for trustworthy deployment of AI in government and, in particular, for oversight. And safety will improve stability and predictability in general, reducing the volatility in the “hot mess” scenario.
However, controlling AI safety is not enough, even for the catastrophic-risk scenarios. We need defensive acceleration to improve societal resilience. This includes strengthening cybersecurity as well as improving areas such as DNA-synthesis monitoring to ensure that constructing bioweapons is not just an “information bottleneck,” but also faces real barriers even for people assisted by advanced AI systems.
We need to do more to make sure that our laws catch up to AI, and do not enable AI being used to surveil people or to grab power. This is a common sense position that is broadly shared by Americans across both sides of the aisle. We need to accelerate AI use by oversight bodies. Part of it is to make sure that people across government, academia, and the media have a realistic understanding of current capabilities of AI and the pace of progress. Right now it seems that many prefer to “bury their head in the sand” and have little understanding of what AI can do now, let alone what it could do by 2030.
Related to the control–distribution tradeoff, we will also need to think about how to trade off security and privacy for AI systems. On one hand, individual privacy has always been and will remain one of the strongest bulwarks against totalitarianism. On the other hand, practices such as pure zero-data retention (ZDR) could be problematic when advanced AI systems can create powerful cyberweapons or bioweapons. I believe that the solution is to use AI to monitor AI. We can ensure that data is only viewed by AI monitors that have very clear specs of what they can and cannot report, with strict control over retention periods and no access by humans.
Making society resilient for AI also means ensuring its economic benefits are widely distributed. If the bulk of society suffers the short-term pain, while a small fraction enjoys the gains, this will make the “hot mess” scenario much more likely. Luckily, any massive economic impact of AI will also be accompanied by a massive increase in growth. Such growth can enable policy responses that were not available before, including significant increases in the social safety net while still reducing the national deficit.
Finally, I am concerned that fears about risks could lead us to abandon the practice of iterative deployment, whereby state-of-the-art models have been widely available. The practice of iterative deployment has so far helped spread AI’s benefits and improve our understanding of its risks. I am happy that ChatGPT is used by nearly a billion people today, rather than in the counterfactual world where advanced AI would have been reserved for national security and AI research (or only for automating away jobs by enterprises). A world in which access to the best AI systems is restricted to the “chosen few” is a highly unequal one.
While many agree that iterative deployment has been positive so far, I can understand the fears of continuing on this path once AI systems achieve critical capabilities in areas such as cybersecurity or virology. Yet, I believe that it is possible to continue deploying such systems in a safe and responsible manner. The key to this is strong technical safety guarantees, as well as robust monitoring. Consider the point of view of a North Korean intelligence officer who is considering using an American AI model for illicit purposes. If the only risk is refusal, then they may go ahead and give it a shot. But if they fear detection, and, with it, the unraveling of their plan (and likely their head), then the calculus becomes quite different.
Some might say that we cannot trust our technical safety enough to share powerful models with the public, even with heightened refusals and monitoring, and hence such systems should be reserved only for internal deployment or national security. My answer is that these are the most sensitive and risky applications of AI. If our technology is reliable enough to be deployed for national security or to train its own successor, we should also be able to build a good enough safety stack to distribute its benefits safely. Yes, it is true that internal deployment or national security have a different attack surface than that of a model that is available to a billion users. But such closed environments also provide more opportunities for silent (and hence riskier) failures, compared to a setting with millions of eyeballs and independent scrutiny.
I am an optimist. I think that more likely than not, we will not fuck it up too royally, and 2030 and the years beyond will be very good for humanity. But I do lose some sleep about the scenarios above. There is work to be done if we want to ensure a future in which all humans, and not a small fraction, are flourishing and empowered. Panic-driven wild swings in both directions (“Models are scary!”, “We must beat China!”) are likely to be counterproductive. But if we remain empirical in evaluating both risks and mitigations, and keep iterating and learning, I believe we can ensure that AGI’s benefits are spread widely.
Notes:
¹ The title could have been any year between 2030 and 2040. I am not making here any specific timeline predictions. However, since the uncertainty about AI's increases exponentially with time, I wanted to focus attention on the next 5-10 years.
² I focus on the U.S. for two reasons: (1) I am an American citizen, live, work, and raise kids in the U.S., and so have a personal interest in its well-being; and (2) because of its unique position, a catastrophic outcome for the U.S. is likely to have impacts worldwide.
³ In this informal blog I am not making a precise definition of “bounded alignment” nor justifying the claim that this is the dominant form of misalignment exhibited by current models. I am also not claiming that this is guaranteed to be the case as capabilities grow, However, I do believe that, if we are careful enough in monitoring and evaluating, including during training and post deployment, we will be able to know if that changes.
⁴ This is a simplification, assuming this level is a scalar, though in practice there would be several dimensions of misalignment.