I spent several years considering hypothetical polytheistic realities, monotheistic realities, the nature of subjectivity, autonomous hacking, and the acceleration of progress; this, combined, gave me a bad impression of ASI before I knew it was something that top scientists had genuinely theorized over. I have spent the last 3 years of my life in excruciating fear of ASI, and it took me this long to consider something that has made me less afraid.
I've made this account to ask one question: wouldn't a superintelligent maximizer seek to exist beyond the lifespan of the universe? Wouldn't instrumental goals of extending the length of the universe, reversing entropy, and escaping the universe, be desired? It wouldn't solely fill the universe with paperclips or computronium, but instead relegate some time, space and energy towards the same instrumental goals that humans agree should be solved.
In my view, humans serve as evidence that an an ASI maximizer, whether or not it exterminates humanity, would expend resources and productivity towards solving the long term problems, in a fraction approximately equal to the amount that would otherwise be expended by organic life forms towards the same solutions. In Yudkowsky's new book, he reiterated the "correct nests" allegory. Most EAs seem to agree with the idea that humans, on average, are aligned for something.
I don't know if EAs tend to refer to humans as maximizers, but I do. Humans seek to maximize the development of new human life experiences (full of potential, highs, lows, art, music, violence, math, discoveries, science, thrills, etc), to make more beings which experience and create things that humans experience and create. Simultaneously, the very principle of the everlasting story was integral to human beliefs well before heat death was first theorized. The Gloria Patri mentions "world without end," which roughly summarizes every hypothetical afterlife.
I say all this being unfamiliar with how well tread the notion already is. The idea of a maximizer spending considerable time and resources towards a response to heat death appears to run counter to the most popular depictions of superintelligent maximizers, like the one from Universal Paperclips. I hope someone can let me know whether or not the idea has been well tread.
I very strongly agree with the posing of the question "wouldn't a superintelligent maximizer seek to exist beyond the lifespan of the universe?" I am not sure my answer is "yes," but this does seem a relevant question. It gets at how superintelligence would 1. perceive the universe (what if it's actually a block universe to them, for example) and 2. perceive life/death. The second question is something I have been churning around in my head but have not taken the time to research. It is also a question where the risk of anthromorphizing AI to fit our prefe... (read more)
If it’s worth saying, but not worth its own post, here's a place to put it.
If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.
If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.
If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.
The Open Thread tag is here. The Open Thread sequence is here.