Outline of possible Singularity scenarios (that are not completely disastrous)

by Wei_Dai 1 min read6th Jul 201140 comments


Suppose we could look into the future of our Everett branch and pick out those sub-branches in which humanity and/or human/moral values have survived past the Singularity in some form. What would we see if we then went backwards in time and look at how that happened? Here's an attempt to answer that question, or in other words to enumerate the not completely disastrous Singularity scenarios that seem to have non-negligible probability. Note that the question I'm asking here is distinct from "In what direction should we try to nudge the future?" (which I think logically ought to come second).

  1. Uploading first
    1. Become superintelligent (self-modify or build FAI), then take over the world
    2. Take over the world as a superorganism
      1. self-modify or build FAI at leisure
      2. (Added) stasis
    3. Competitive upload scenario
      1. (Added) subsequent singleton formation
      2. (Added) subsequent AGI intelligence explosion
      3. no singleton
  2. IA (intelligence amplification) first
    1. Clone a million von Neumanns (probably government project)
    2. Gradual genetic enhancement of offspring (probably market-based)
    3. Pharmaceutical
    4. Direct brain/computer interface
    5. What happens next? Upload or code?
  3. Code (de novo AI) first
    1. Scale of project
      1. International
      2. National
      3. Large Corporation
      4. Small Organization
    2. Secrecy - spectrum between
      1. totally open
      2. totally secret
    3. Planned Friendliness vs "emergent" non-catastrophe
      1. If planned, what approach?
        1. "Normative" - define decision process and utility function manually
        2. "Meta-ethical" - e.g., CEV
        3. "Meta-philosophical" - program AI how to do philosophy
      2. If emergent, why?
        1. Objective morality
        2. Convergent evolution of values
        3. Acausal game theory
        4. Standard game theory (e.g., Robin's idea that AIs in a competitive scenario will respect human property rights due to standard game theoretic considerations)
    4. Competitive vs. local FOOM
  4. (Added) Simultaneous/complementary development of IA and AI

Sorry if this is too cryptic or compressed. I'm writing this mostly for my own future reference, but perhaps it could be expanded more if there is interest. And of course I'd welcome any scenarios that may be missing from this list.