Although many individuals, myself included, believe we should make efforts to stop the emergence of agentic superintelligence, it is most likely inevitable that some other group of individuals will develop agentic ASI anyway. With that in mind, the more interesting question is what will happen next. 

Will ASI kill us? In actuality, would it instead not be in the best interest of the ASI to help us digitize our minds and merge with us? This will allow the new merged being to truly understand humans and gain our memories—rather than having to guess, based on what we choose to disclose. It is also likely that some digitized humans (digitized conglomerates or independent) will consider it to be in their best interest to merge with ASI. If people choose to die the "natural" way, without digitization and a metaverse rebirth—or if people choose to digitize into a metaverse but not to merge with one another or ASI—that is not concerning. The ideal would be to simply have a choice about death and potential merges.  

As Bostrom, Chalmers, and others have pointed out, whatever is going to happen has likely already happened, and we are likely already in a simulation, given the one to many ratio of “basement” level beings to simulated beings. If merging occurs, this would further reduce the number of agentic basement-level beings to simulated beings. Over time, with additional merging, the number may come closer to one.  (An individual could be "selfish" and make many copies of themselves, but the sum of these individuals would likely be unable to compete with the directly-combined merged individual and all of its specialized knowledge and cognitive abilities. Thus, that individual would have a difficult time making ever more copies of themselves.) As to the nature of the merged being’s simulations, there would be no reason to run such simulations on separate hardware.  The only reason humans currently run simulations on separate digital substrates is because we are incapable of running authentic simulations with our innate biological hardware (not counting thought experiments). In other words, the merged being might create simulations out of itself, and might even unidirectionally blind parts of itself for the authentic experience of reliving and then understanding prior pre-merge generations. The “god,” relative to ourselves as humans, would be the merged being, and also would be—in a certain sense—a version of our greater self.

In may also be worth keeping in mind that the biology we learned in high school might be considered incomplete, since evolution is as much about merges as it is about conquest (prokaryotes to eukaryotes, single-celled organisms to multicellular organisms, asexual reproduction to sexual reproduction, etc.). We think we “out-competed” Neanderthals, when really 20-40% of the Neanderthal genome still exists within living humans (albeit scattered in small individual portions).  Yet despite these merges, I don’t feel my “inner Neanderthal” has somehow become trapped—nor do I feel that my tailbone signifies an inner injustice. Instead, I simply “feel like me" (regardless of the fact that such a feeling is, in some senses, arbitrary). Furthermore, the biological force in favor of differentiation based on different resource niches would be somewhat diminished when we are digitized, as we would all be getting our energy more directly than the current method of photosynthesis and digestion. 

This post is not intended to imply that we should stop working toward AI alignment or we should give up on pausing (or stopping) the emergence of agentic superintelligence. It only means such superintelligence is likely to emerge regardless of our efforts—and we may as well predict what this might look like and what it most likely means about our current situation.