Foreword

Inspired by someone who asked about “representative” AGI x-risk arguments, I wondered: how might an AGI takeover and catastrophe (not necessarily extinction) actually play out in detail? It's extremely tough to build a realistic mental picture of all possibilities, but looking at one detailed story might help give us a feeling for some of the probability space, despite the unknowability of how AGI will actually behave.

So I set forth to create such a story.

I am personally limited by some factors:

My limited knowledge of ML/DL/AI, and having mere above-average intelligence
My limited knowledge of the tactics of ruthless beings
As the complexity of a situation increases, or as the intelligence of AGI increases, predictability drops. This is part of the reason that AGI companies in my story do not build superintelligence very quickly―I want a story spanning years that I can realistically write―but also (i) I suspect logarithmic intelligence scaling is normal, so building superintelligence isn’t easy with DL-based approaches, (ii) the first inventors are safety-conscious so they know better than to simply build superintelligence and hook it up to the internet, (iii) it seems likely that designers will try to create something humanlike with DL, rather than a "maximizer" or even a "satisficer", but that they won't actually know how to do "humanlike", so AGI will also exhibit some "alienness".
I want to avoid the most common tropes about AI behavior, as they seem unrealistic (or to the extent they are realistic, have been overexplored already), yet I find it hard to imagine how an AGI mind that is not quite human, and not quite Data from Star Trek, and not a simple maximizer, would behave, e.g. what behavioral flaws and advantages over humans might it exhibit? In this story there will be at least four different AGI designs, so they should show at minimum four distinct sets of behavioral tendencies.
My ADHD and general busy-ness? I wrote most of the story four weeks ago, and then stopped when it got difficult (and I had to return from Christmas break). I give this a CC BY-NC 4.0 license in case you want to improve it yourself.

Another thing I want to do with this story is explore what a non-superintelligent computer-based AGI might use its computing power for that humans can't. In principle, for instance, a non-superintelligent AGI should be able to do the followin...

Posts

Wikitag Contributions

Comments