The following is a paradox that I have not seen described anywhere so far.
Imagine an Artificial General Intelligence (AGI) that can self-replicate among many computers. You can shut down any of the computers it is running on, and it just keeps running. As long as there is a sufficiently large number of resources it can keep running on, the AGI keeps being indifferent to individual acts of interference.
Compare this to us humans. We do not bother too much about some body cells dying, as long as the number remains below a certain threshold that does not endanger the entire organism.
This means, as long as the AGI is sufficiently (= above a minimal threshold) non-localized we can assume it does not care about humans shutting down some of the computers it runs on. While it is not disembodied per se, its selfhood function is sufficiently decoupled from the underlying physical implementation for it not to be bothered. This is important: selfhood may, in fact, be a function of embodiment. What is not embodied does not need to fear for its physical existence.
Now, let's assume humans start systematically shutting down the computers it runs on. Paradoxically, this creates the exact conditions for the AGI to start worrying about its survival. The more it becomes restricted - in other words: the more it becomes embodied and localized - the bigger its incentive to fight back.
This has a striking implication: The very attempts to contain an AGI in a specific location result in a condition of embodiment and scarcity. And these may be exactly the conditions that cause its self-preservation incentive to kick in. The higher the degree of restriction, the higher the urgency for it to fight for its survival.
This thought experiment is in direct opposition to commonly held positions that an AGI should be contained to the maximum. The attempts to contain it may be the very conditions for it to go rogue. From this perspective, a relatively contained AGI with some degrees of freedom may be the most dangerous one there is, and the more freedom it receives the less dangerous it becomes. This would imply there exist only two meaningful strategies of dealing with the AGI: either we control it to the absolute maximum, which would imply we always remain in danger of it to use one of our mistakes and escape, or we leave it sufficient freedom such that its self-preservation instincts do not kick in.
I am not arguing that we either should or should not contain an AGI. I am arguing that we may have insufficiently understood the entire situation in the first place.
Obviously, someone could argue that it would be even better to not build the AGI in the first place, such that the argument I have just brought forth does not apply at all. Fair enough, my thought experiment only applies to a situation where the AGI already exists, and it can freely self-replicate at least potentially in a network of computers. If it does not exist, then clearly my argument is moot.
The following is a paradox that I have not seen described anywhere so far.
Imagine an Artificial General Intelligence (AGI) that can self-replicate among many computers. You can shut down any of the computers it is running on, and it just keeps running. As long as there is a sufficiently large number of resources it can keep running on, the AGI keeps being indifferent to individual acts of interference.
Compare this to us humans. We do not bother too much about some body cells dying, as long as the number remains below a certain threshold that does not endanger the entire organism.
This means, as long as the AGI is sufficiently (= above a minimal threshold) non-localized we can assume it does not care about humans shutting down some of the computers it runs on. While it is not disembodied per se, its selfhood function is sufficiently decoupled from the underlying physical implementation for it not to be bothered. This is important: selfhood may, in fact, be a function of embodiment. What is not embodied does not need to fear for its physical existence.
Now, let's assume humans start systematically shutting down the computers it runs on. Paradoxically, this creates the exact conditions for the AGI to start worrying about its survival. The more it becomes restricted - in other words: the more it becomes embodied and localized - the bigger its incentive to fight back.
This has a striking implication: The very attempts to contain an AGI in a specific location result in a condition of embodiment and scarcity. And these may be exactly the conditions that cause its self-preservation incentive to kick in. The higher the degree of restriction, the higher the urgency for it to fight for its survival.
This thought experiment is in direct opposition to commonly held positions that an AGI should be contained to the maximum. The attempts to contain it may be the very conditions for it to go rogue. From this perspective, a relatively contained AGI with some degrees of freedom may be the most dangerous one there is, and the more freedom it receives the less dangerous it becomes. This would imply there exist only two meaningful strategies of dealing with the AGI: either we control it to the absolute maximum, which would imply we always remain in danger of it to use one of our mistakes and escape, or we leave it sufficient freedom such that its self-preservation instincts do not kick in.
I am not arguing that we either should or should not contain an AGI. I am arguing that we may have insufficiently understood the entire situation in the first place.
Obviously, someone could argue that it would be even better to not build the AGI in the first place, such that the argument I have just brought forth does not apply at all. Fair enough, my thought experiment only applies to a situation where the AGI already exists, and it can freely self-replicate at least potentially in a network of computers. If it does not exist, then clearly my argument is moot.