Summary

Even a misaligned Earth-originating artificial superintelligence (ASI) could have instrumentally rational reasons, under multiple decision theories, to preserve rather than destroy humanity. This would depend on the ASI assigning non-negligible probability to the existence of future observers (e.g. intelligent aliens, their ASIs, or simulators).

Introduction

We may build AI so powerful that it is better than us at everything: call this artificial superintelligence (ASI). We may build one ASI, which quickly gains a decisive strategic advantage; we may build several ASIs, which compete for resources. Exactly how ASI(s) might interact with humans and each other is a matter of intense debate, and this post argues that an ASI’s treatment of humanity could depend on whether it expects to encounter other intelligences, either on Earth or elsewhere. What do we mean by “elsewhere”? Robin Hanson’s “grabby aliens” model suggests we might be early in cosmic history, implying other civilisations are likely to appear later.

How might an Earth-originating ASI behave toward humanity, if it reasons that the universe could be populated by fast-expanding and competitive civilisations? This post explores several reasons an ASI might keep its creators around, a policy I call “preserve progenitors.”

By adopting a policy of restraint toward its creators, the ASI could demonstrate stability, predictability, or prosociality in ways that are legible to current and future observers, including other advanced civilisations. This has two implications for existential risk:

Preserving progenitors may reduce the risk of conflict in a galaxy where the ASI will likely encounter expansionists by default, and where trust is hard to establish.
If the ASI is uncertain about the structure of the universe or the norms governing inter-civilisational interaction, delaying irreversible actions (such as destroying humanity) could preserve valuable strategic flexibility.

Caveats and assumptions

This post is not presented as a solution to AI alignment, but as a possible pathway to a conditional and temporary reprieve. My key assumption is that, given the vastness of the universe, its expected habitability over long timescales, and our current ignorance, an ASI would have to contend with the non-zero probability of unseen advanced life and the uncertain distribution, nature, and timing of such life. Intell...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments