Preamble

Honest question.

• Some clarification on terms:
  SSI: Strongly Superhuman Intelligence
• Misaligned: for the purposes of this question an AI  is misaligned if it is indifferent to human welfare and wellbeing.
  • An SSI that was actively malicious would also be unfriendly, but would in all likelihood have been deliberately aligned in that direction

Introduction

For the purposes of this question, I'd like to posit a concrete example of a misaligned SSI. For the sake of convenience, I'll consider a unipolar scenario with a single misaligned SSI (because it's the simplest and [more importantly] most dangerous scenario). Let's suppose an agent: Alpha. Alpha wants to maximise the number of digits of pi that have been computed. Alpha is clearly misaligned.

I guess my questions are:

1. Will Alpha kill us all?
2. Under what circumstances is Alpha likely to kill us all?

Why I Don't Think Alpha Will Kill Us All

When I try to imagine what the outcome of letting Alpha loose on the world is, human omnicide/depopulation seems like a very unlikely outcome? Alpha seems like it would be dependent on the continued functioning of the human economy to realise its goals:

• Stable and uninterrupted electricity supply for its computing clusters
  • Other maintenance services for said clusters
• Production of more computing chips and other hardware for its clusters
• Construction/refurbishment/upgrades of said clusters
• Stable and high bandwidth internet connectivity
• The global distributed supply chains for all of the above
• Etc.

Furthermore, Alpha will almost certainly not be the most capable actor at providing all the economic activity it finds valuable. It will have neither a comparative nor absolute advantage at many tasks it finds valuable. Given the same compute budget, a narrow optimiser would probably considerably outperform a general optimiser (because the general optimiser's performance is constrained by the pareto frontier of the multitude of domains it operates in while the narrow optimiser is free to attain the global optimum in its particular domain). 

This dynamic applies even to general agents specialising on narrow tasks, and explains why the human peak for some cognitive activity may be orders of magnitude beyond beginners at said activity. 

Thus, for many goods and services that Alpha would seek to obtain, the most profitable way to do it is to pay civilisation to do it. It's absurd to postulate that Alpha is just better at literally every economic task of value. That's just not how intelligence/optimisation manifests in our universe. 

As such, I expect that Alpha will find it very profitable to trade with civilisation. And this may well be the case indefinitely? There may never be a time where Alpha has a comparative advantage vs civilisation in the production of literally every good and service (it's likely that civilisation's available technology will continue to rise and may even keep pace with Alpha [perhaps due to said aforementioned trade]). 

Human depopulation (even just the kill one billion humans) seems like it would just make Alpha strictly (much) poorer. The death of a billion humans seems like it would impoverish civilisation considerably. The collapse of human civilisation/the global economy does not seem like it would be profitable for Alpha.

That is, I don't think omnicide/depopulation is positive expected utility from the perspective of Alpha's utility function. It seems an irrational course of action for Alpha to pursue.

Reasons Alpha Might Destroy the World

I'll present the plausible arguments I've seen for why Alpha might nonetheless destroy the world, and what I think about them.

1. Humanity are adversaries to Alpha
2. Alpha doesn't need humanity

I've seen some other arguments, but I found them very stupid, so I don't plan to address them (I don't think insulting bad arguments/the source(s) of said bad arguments is productive).

Humans are Adversaries to Alpha

The idea is that humans are adversaries to Alpha. We don't share its utility function and will try to stop it from attaining utility.

I don't think this is necessarily the case:

1. A human group with the goal of maximising the number of digits of pi computed can function very adequately as economic actors within human civilisation
   1. Human actors have in fact calculated pi to strictly ridiculous lengths (100 trillion digits on Google Cloud)
2. An amoral human group with arbitrary goals can also function very adequately as economic actors within human civilisation
   1. The traditional conception of a homo economicus corporation as strictly profit maximising assumes/presupposes amorality.

Human civilisation is only actively hostile towards malicious utility functions (we couldn't coexist with an AI who instead wanted to maximise suffering), not apathetic ones.

Alpha having values at odd with civilisation's collective values, but not actively hostile to it is not an issue for coexistence. We already meditate such value divergences through the economy and market mechanisms.

Humans are not necessarily adversaries to Alpha.

Alpha Doesn't Need Humanity

This is an argument that if true, I think would be cruxy. Humans aren't necessarily adversaries to Alpha, but we are potential adversaries. And if Alpha can just get rid of us without impoverishing itself — if its attainment of value is not dependent on the continued functioning of human civilisation — then the potential risk of humans getting in its way may outweigh the potential benefits of pursuing its objectives within the framework of human civilisation.

I do believe that there's a level of technological superiority, at which Alpha can just discard human civilisation. I just doubt that:

• Said level can be attained quickly
• That independently pursuing such technological superiority is cheaper than amplification/progress of existing capabilities within civilisation
• Issues with specialisation/comparative advantage and the aforementioned trade may prevent said gap in technological superiority from ever manifesting.

This question may perhaps be reframed as: "at what point will it be profitable for Alpha to become an autarchy"? 

The above said, I do not think it is actually feasible for a strongly superhuman intelligence to do any of the following within a few years:

• Replace the entire human economy with robots/artificial agents that Alpha controls or are otherwise robustly aligned with it
  • Consider that many automatable jobs today are still done manually because it's cheaper or infrastructure doesn't exist to facilitate automation, etc.
  • Full automation of the entire economy will not happen until it's economically compelling, and I can't really alieve that near term
  • Even if it's potentially doable said aforementioned issues with specialisation and comparative advantage make me doubt it would necessarily be an optimal allocation of resources. 
• Develop a parallel economy whose output exceeds the gross world product of human civilisation (sans the AI)
  • An alternative measure of economic output would be energy consumption

The level of cognitive capabilities required to exceed civilisation within a few years seem very "magical" to me. I think you need an insanely high level of "strongly superhuman" for that to be feasible (to a first approximation, civilisation consumes eleven orders of magnitude more energy than an individual adult human operating by themselves could consume).

I don't think the AI making many copies of itself would be as effective as many seem to think:

• Groups of agents are economically superior to individual agents due to specialisation and division of labour
  • This implies that the copies would likely need to retrain and specialise on their particular tasks
• The aggregate "intelligence" of a group seems to scale strongly sublinearly with additional members
  • Garry Kasparov famously beat a world team (50,000 people supervised by Grandmasters) at chess
• Groups of actors are plagued by the general problems of bureaucracies

Nor do I think a speed superintelligence would be easy to attain:

• Faster thought seems like it requires faster serial processing and I'm under the impression that we're running up against fundamental limits there
  • As far as I'm aware, more powerful processors are mostly driven by better parallel processing
  • It's not clear that "Jon Von Neumann but running one million times faster" is actually physically possible
    • I suspect even a 1,000x speedup may be implausible for reasonably sized computers giving thermodynamic constraints
• Can we actually speed up the "thinking" of fully trained ML models by K times during inference if we run it on processors that are K times faster?
  • How does thinking/inference speed scale with compute?

Speed superintelligence also has some limitations:

• Time spent waiting for I/O operations
• Time spent waiting for stuff to happen in the real world

Beyond a very high bar for "doing away with humanity", I expect (potentially sharply) diminishing returns to cognitive investment. I.e:

• Superlinear increases in investment of computational, cognitive (and other economic) resources for linear increases in cognitive capabilities
• Superlinear increases in cognitive capabilities for linear increases in real world capabilities

Diminishing marginal returns are basically a fundamental constraint of resource utilisation; I don't really understand why people think intelligence would be so different.

(P.S: I'm not really considering scenarios where Alpha quickly bootstraps to advanced self replicators that reproduce on a timeframe orders of magnitude quicker than similarly sized biological systems. I'm not:

• Confident that such powerful self replicators are physically possible
  • There seems to be a robust relationship between organism size and replication efficiency in the natural world. This relationship may actually be fundamental.
  • Non biological self replicators may also be bound by similar constraints
  • People who are knowledgeable about/work in nanotech that I've spoken to seem incredibly sceptical that it is physically possible
• I do not expect Alpha to be able to develop them quickly enough that it is profitable to disregard the human economy.
  • Alpha will need to trade with humanity before it can amass the resources to embark on such a project
  • Alpha will still need to work through the basic product life cycle when developing high technology:
    • Basic science
    • Research and development
    • Engineering
    • Commercialisation/mass production

Conclusions

There are situations under which human depopulation would be profitable for an AI, but I just don't think such scenarios are realistic. An argument for incredulity isn't worth much, but I don't think I'm just being incredulous here. I have a vague world model behind this, and the conclusion doesn't compute under said world model?

I think that under the most likely scenarios, the emergence of a strongly superhuman intelligence that was indifferent to human wellbeing would not lead to human depopulation of any significant degree in the near term. Longer term, homo sapiens might go extinct due to outcompetion, but I expect homo sapiens to be quickly outcompeted by posthumans even if we did develop friendly AI.

I think human disempowerment and dystopian like scenarios (Alpha may find it profitable to create a singleton) are more plausible outcomes of the emergence of misaligned strongly superhuman intelligence. Human disempowerment is still an existential catastrophe, but it's not extinction/mass depopulation.