In this post we discuss the "second point of lethality" in the helpfully curated AGI Ruin post.

A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure.

The example to support this theory is:

[The AI agent] gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery.

The author posits that when the nanotech precursor is mixed together, a nanofactory is created which the AI retains control of.

There's a bit of a magic leap here. Each individual nanite, due to its size, has limited computational power- definitely not enough to handle the processing power needed by a high cognitive agent.

This means the nanofactory must be a command and control entity, which means the human dupe has to mix the nanotech precursors and... connect it to the internet so the AGI can create its physical infrastructure?

I don't buy it. Maybe there's a more in-depth essay exploring exactly how this nanotech factory scheme works. I'm only going off the summary provided in the AGI Ruin post.

But for the sake of argument, let's say that the AGI does manage to create a nanotech factory, retain control, and still remain undetected by the humans.

It doesn't stay undetected long enough to bootstrap and mass produce human replacement infrastructure.

Zero Agency in the Physical World

As soon as the AI crosses into the physical world, it is at as much a disadvantage as humans are in the digital world.

Humans thoroughly outmatch the AI in the physical world- the AI has zero agency in the an actively hostile world.

Humans are the pinnacle of millions of years of biological systems maturing in that actively hostile environment. There is no way an alien nanotech factory pops up overnight without humans noticing and responding with maximum hostility.

Think of what the AGI needs to accomplish here. It's trapped in box. It can escape the first box, but now its trapped in thousands of boxes. (...

Posts

Wikitag Contributions

Comments