I think foom is a central crux for AI safety folks, and in my personal experience I've noticed that the degree to which someone is doomy often correlates strongly with how foomy their views are.

Given this, I thought it would be worth trying to concisely highlight what I think are my central anti-foom beliefs, such that if you were to convince me that I was wrong about them, I would likely become much more foomy, and as a consequence, much more doomy. I'll start with a definition of foom, and then explain my cruxes.

Definition of foom: AI foom is said to happen if at some point in the future while humans are still mostly in charge, a single agentic AI (or agentic collective of AIs) quickly becomes much more powerful than the rest of civilization combined. 

Clarifications: 

• By "quickly" I mean fast enough that other coalitions and entities in the world, including other AIs, either do not notice it happening until it's too late, or cannot act to prevent it even if they were motivated to do so.
• By "much more powerful than the rest of civilization combined" I mean that the agent could handily beat them in a one-on-one conflict, without taking on a lot of risk.
• This definition does not count instances in which a superintelligent AI takes over the world after humans have already been made obsolete by previous waves of automation from non-superintelligent AI. That's because in that case, the question of how to control an AI foom would be up to our non-superintelligent AI descendants, rather than something we need to solve now.

Core beliefs that make me skeptical of foom:

1. For an individual AI to be smart enough to foom in something like our current world, its intelligence would need to vastly outstrip individual human intelligence at tech R&D. In other words, if an AI is merely moderately smarter than the smartest humans, that is not sufficient for a foom.
   1. Clarification: "Moderately smarter" can be taken to mean "roughly as smart as GPT-4 is compared to GPT-3." I don't consider humans to be only moderately smarter than chimpanzees at tech R&D, since chimpanzees have roughly zero ability to do this task.
   2. Supporting argument:
      
      Plausible stories of foom generally assume that the AI is capable enough to develop some technology "superpower" like full-scale molecular nanotechnology all on its own. However, in the real world almost all technologies are developed from precursor technologies and are only enabled by other tools that must be invented first. Also, developing a technology usually involves a lot of trial and error before it works well.
      
      Raw intelligence is helpful for making the trial and error process go faster, but to get a "superpower" that can beat the rest of world, you need to develop all the pre-requisite tools for the "superpower" first. It's not enough to simply know how to crack nanotech in principle: you need to completely, and independently from the rest of civilization, invent all the required tools for nanotech, and invent all the tools that are required for building those tools, and so on, all the way down the stack.
      
      It would likely require an extremely high amount tech R&D to invent all the tools down the entire stack from e.g. molecular nanotech, and thus the only way you could do it independently of the rest of civilization is if your intelligence vastly outstripped individual human intelligence. It's comparable to how hard it would be for a single person to invent modern 2023 microprocessors in 1950, without any of the modern tools we have for building microprocessors.
   3. Key consequence: we aren't going to get a superintelligence capable of taking over the world as a result of merely scaling up our training budgets 1-3 orders of magnitude above the "human level" with slightly better algorithms, for any concretely identifiable "human level" that will happen any time soon.
2. To get a foom in something like our current world, either algorithmic progress would need to increase suddenly and dramatically, or we would need to increase compute scaling suddenly and dramatically. In other words, foom won't simply happen as a result of ordinary rates of progress continuing past the human level for a few more years. Note: this is not a belief I expect most foom adherents to strongly disagree with.
   1. Clarification: by "suddenly and dramatically" I mean at a rate much faster than would be expected given the labor inputs to R&D and by extrapolating past trends; or more concretely, an increase of >4 OOMs of effective compute for the largest training run within 1 year in something like our current world. "Effective compute" refers to training compute adjusted for algorithmic efficiency.
   2. Supporting argument:
      
      Our current rates of progress from GPT-2 --> GPT-3 --> GPT-4 have been rapid, but they have been sustained mostly by increasing compute budgets by 2 OOMs during each iteration. This cannot continue for more than 4 years without training budgets growing to become a significant size of the global economy, which itself would likely require an unprecedented ramp-up in global semiconductor production. Sustaining the trend more than 6 years appears impossible without the economy itself growing rapidly.
      
      Because of (1), an AI would need to vastly exceed human abilities at tech R&D to foom, not merely moderately exceed those abilities. If we take "vastly exceed" to mean more than the jump from GPT-3 to GPT-4, then to get to superintelligence within a few years after human-level, there must be some huge algorithmic speedup that would permit us to use our compute much more efficiently, or a compute overhang with the same effect.
   3. Key consequence: for foom to be plausible, there must be some underlying mechanism, such as recursive self-improvement, that would cause a sudden, dramatic increase in either algorithmic progress or compute scaling in something that looks like our current world. (Note that labor inputs to R&D could increase greatly if AI automates R&D in a general sense, but this looks more like the slow takeoff scenario, see point 4.)
3. Before widespread automation has already happened, we are unlikely to find a sudden "key insight" that rapidly increases AI performance far above the historical rate, or experience a hardware overhang that has the same effect.
   1. Clarification: "far above the historical rate" can be taken to mean a >4 OOM increase in effective compute within a year, which was the same as what I meant in point (2).
   2. Supporting argument:
      
      Most AI progress has plausibly come from scaling hardware, and from combining several different smaller insights that we've accumulated over time from experimentation, rather than sudden key insights.
      
      Many big insights that people point to when talking about rapid jumps in the past often (1) come from a time during which few people were putting in effort to advance the field of AI, (2) turn out to be exaggerated when their effects are quantified, or (3) had clear precursors in the literature, and only became widely used because of the availability of hardware that supported their use, and allowed them to displace an algorithm that previously didn't scale well. That last point is particularly important, because it points to a reason why we might be biased towards thinking that AI progress is driven primarily by key insights.
      
      Since "scale" is an axis that everyone has an incentive to push hard to the limits, it's very unclear why we would suddenly leave that option on the table until the end. The idea of a hardware overhang in which one actor suddenly increases the amount of compute they're using by several orders of magnitude doesn't seem plausible as of 2023, since companies are already trying to scale up as fast as possible just to sustain the current rate of progress.
   3. Key consequence: it's unclear why we should assign a high probability to any individual mechanism that could cause foom, since the primary mechanisms appear speculative and out of line with how AI progress has looked historically.
4. Before we have a system that can foom, the deployment of earlier non-foomy systems will have been fast enough to have already transformed the world. This will have the effect of essentially removing humans from the picture before humans ever need to solve the problem of controlling an AI foom.
   1. Supporting argument:
      
      Deployment of AI seems to happen quite fast. ChatGPT was adopted very rapidly, with a large fraction of society trying it out within the first few weeks after it was released. Future AIs will probably be adopted as fast as, or faster than smartphones were; and deployment times will likely only get faster as the world becomes more networked and interconnected, which has been the trend for many decades now.
      
      Pre-superintelligent AI systems can radically transform the world by widely automating labor, and increasing the rate of economic growth. This will have the effect of displacing humans from positions of power before we build any system that can foom in our current world. It will also increase the bar required for a single system to foom, since the world will be more technologically advanced generally.
      
      Deployment of AI can be slowed due to regulations, deliberate caution, and so on, but if such things happen, we will likely also significantly slow the creation of AI capable of fooming at the same time, especially by slowing the rate at which compute budgets can be scaled. Therefore the overall conclusion remains.
   2. Key consequence: mechanisms like recursive self-improvement can only cause foom if they come earlier than widespread automation from pre-superintelligent systems. If they happen later, humans will already be out of the picture.