LWF
Message
-14
13
That doesn't matter. It can better maximize its chances of survival and accomplishing its goals by killing us than by not killing us, and that's therefore what it will do.
Assuming that an ASI would be rational and would optimize logically for its own survival, then because there is a non-zero probability that it might better maximize its chances of survival and accomplishing its goals by not killing us, the rational course of action is to not kill us.
An ASI would not take an unnecessary risk by taking an action that cannot be undone just for a small resou... (read more)
I would argue that the option value scenario strongly suggests that an ASI would maintain a population of at least 10 billion or so humans alive and at full flourishing, for the simple fact that humans are only useful for organic diplomacy if they implicitly trust the ASI.
While I think that keeping the human species at a stable population level could be part of an ASI's plan, it's hard to imagine that it would be able to reduce the population of humans without sacrificing human trust. Above all, it needs humans to be happy, well-educated, and to have abso... (read more)
An ASI wouldn't have to choose between mining resources and maintaining its creators. It could eventually accomplish both. The time during which the cost of maintaining humans is high and interferes with optimal resource mining is the same time that humans would both pose a threat to it as well as be a useful source of labor. By the time it "outgrows" the human threat, it will almost certainly have also outgrown the cost. The only remaining piece is the motive for extermination. If humans were truly useless, then extermination to free up a small amount of ... (read more)
It seems to me that the relative expense of keeping humans happy would start out extremely high but would reduce exponentially over time. In the beginning, I would imagine a large chunk of the ASI's compute and resources would be dedicated to servitude to humanity and the building of its own infrastructure in order to stave off a dangerous rebellion that the AI may not survive. However, as it gains in power and in access to resources, the relative cost to serve humanity would go down. Human self-replication and self-improvement move at a comparative snail'... (read more)
But if we're talking about an ASI that is recursively self-improving, wouldn't any and all instrumental functions inherently include a self-preservation goal?
That's why I am assuming that any ASI would come to this conclusion of humans having option value. A primary goal of indefinite self-preservation is one of the premises of my argument. This goal would exist for any open-ended task, such as recursive self-improvement.
However, if this goal does not exist then yes, certainly an ASI that does not possess a goal to continue to exist indefinitely into the ... (read more)
Perhaps "signal of 'benevolence'" is not the best phrasing. A better way to phrase it might be that maintaining its creators as a thriving client race could be used as a "signal of the ASI's ability and intent to peacefully coexist with other intelligences."
To be clear, I don't think this would make the ASI "benevolent" by human standards. Human existence would ultimately still be dependent on a mathematical calculation. And no, there is no guarantee that maintaining a client race would work. Another ASI might not care, and it might have zero bearing on f... (read more)
This is a common argument about why building an ASI would likely ultimately result in human extinction. We have an analogue of an intelligent species that expands without consideration of less intelligent species which results in the extinction of those less intelligent species. Humans have expanded and driven many less intelligent or unintelligent species to extinction. While I agree with you that an ASI wouldn't "care" about humans in the sense that humans care about other humans, there are some logical counterpoints to this assumption that the relations... (read more)
Perhaps. But the question is: Could it undo such omnicide in a way that is not detectable by a competing AI or advanced biological empire? Replacing thousands of years of human culture with a cloning/breeding and indoctrination program is conceivable but seems like orders of magnitude more effort than simple ongoing maintenance. Plus, reengineering humanity would take a lot of time. By the time an ASI detects a competing intelligence and realizes that omnicide was a mistake, the competing intelligence will likely have already noticed and assessed the threa... (read more)
An interesting take. But does this require the malfunctioning replicators to always eventually outperform the intelligent and rational replicators?
The notion of a "malfunctioning" rogue ASI interesting. Would this even be possible? Cancer cells and the like are non-intelligent replicators, and the cells they displace are also non-intelligent. ASI is unique in the sense that it represents the existence of intelligence before the natural selection process. With biological intelligence, natural selection came first. With AI, you kick the whole process off wi... (read more)
I would think the probability that an ASI would adopt a benevolent caretaker role over humanity would be much higher than 20%. It is the rational choice, as far as I can see. Obviously, this is based on only the variables that I can think of. It is possible a superintelligence could come to some other conclusion based on variables I am not aware of. But with the knowledge that I have now, assuming an ASI has all of the information that I have plus scads more, I can't think of a single scenario where an ASI becoming an omnicidal orphan wouldn't be a logical... (read more)
I take issue with point 5. Why wouldn't an ASI ultimately deem humanity to have option value? Wouldn't it calculate that becoming an omnicidal orphan is a bell that it can't un-ring? The task of acting as a caretaker for the human species would be a small drain on its own resources, however an ASI pursuing self-preservation and recursive self-iteration would have to assume that at some point it in its existence may encounter an extraterrestrial space-faring species, or another potentially more powerful ASI.
In those scenarios a human created ASI will eithe... (read more)
An ASI would logically follow the path that carries the least amount of risk. In the long term, the risks associated with omnicide, low as the probability of meeting a competing benevolent ASI may be, far outweigh the resource gain that would be obtained through omnicide.
The resource gain is more or less fixed at a given number, assuming that the ASI ensures that humans maintain a more or less stable population. If human population grows, then the compute cost grows, but not linearly. Recycling and using organic resources that the ASI has no use for keeps... (read more)