The map of "Levels of defence" in AI safety

[-]UNVRSLWSDM8y20

lovely... Catch Me If You Can, AI style... How about adding a box for "too much power in wrong human hands" (if not already there somehow)? Yes, AI power too.

Because this is the far greatest problem of human civilization. Everything (just staying in the tech domain) is created by very small fraction of the population (who posses that superpower called INTELLIGENCE) , but the society is ruled by a completely different tiny group that posses various predatory superpowers to assume control and power (over the herd of mental herbivores they "consume").

This is not going to end up well, such a system does not scale, has imbalances and systemic risks practically everywhere. Remember, already nuclear weapons were "too much" and we are only lucky that bio$hit is not really effective as a weapon (already gas was nothing much).

We are simply too much animals and too little on that "intelligent" side we often worship. And the coming nanobots will be far worse than nukes, and far less controllable.

[-]RedMan8y00

Rules for an AI:

If an action it takes results in more than N logs of $ worth of damage to humans/kills more than N logs of humans, transfer control of all systems it can provide control inputs to designated backup (human, formally proven safe algorithmic system, etc), power down.

When choosing among actions which affect a system external to it, calculate probable effect on human lives. If probability of exceeding N assigned in rule 1 is greater than some threshold Z, ignore that option, if no options are available, loop.

Most systems would be set to N= 1, Z = 1/10000, giving us five 9s of certainty that the AI won't kill anyone. Some systems (weapons, climate management, emergency management dispatch systems) will need higher N scores and lower Z scores to maintain effectiveness.

JFK had an N of like 9 and a Z score of 'something kind of high', and passed control to Lyndon B Johnson of 'I have a minibar and a shotgun in the car I keep on my farm so I can drive and shoot while intoxicated' fame. We survived that, we will be fine.

Are we done?

[-]Lumifer8y10

Are you reinventing Asimov's Three Laws of Robotics?

[-]RedMan8y00

I hadn't thought about it that way.

I do think that either compiler time flags for the AI system or a second 'monitor' system chained to the AI system in order to enforce the named rules would probably limit the damage.

The broader point is that probabilistic AI safety is probably a much more tractable problem than absolute AI safety for a lot of reasons, to further the nuclear analogy, emergency shutdown is probably a viable safety measure for a lot of the plausible 'paperclip maximizer turns us into paperclips' scenarios.

"I need to disconnect the AI safety monitoring robot from my AI-enabled nanotoaster robot prototype because it keeps deactivating it" might still be the last words a human ever speaks, but hey, we tried.

[-]Lumifer8y10

There seems to be a complexity limit to what humans can build. A full GAI is likely to be somewhere beyond that limit.

The usual solution to that problem -- see the EY's fooming scenario -- is to make the process recursive: let a mediocre AI improve itself, and as it gets better it can improve itself more rapidly. Exponential growth can go fast and far.

This, of course, gives rise to another problem: you have no idea what the end product is going to look like. If you're looking at the gazillionth iteration, your compiler flags were probably lost around the thousandth iteration and your chained monitor system mutated into a cute puppy around the millionth iteration...

Probabilistic safety systems are indeed more tractable, but that's not the question. The question is whether they are good enough.

LESSWRONG
LW

LESSWRONG
LW

3

The map of "Levels of defence" in AI safety

3

3