Matthew McRedmond

Epistemic Status: I think the dilemma as outlined in Section 1 follows from well-established ideas about Astronomical Waste. However, given that I have not seen it anywhere before I might have made some oversight I am unaware of. You don't know what you don't know but maybe someone on LessWrong does. UPDATE: I have found a reference to this scenario as a footnote in Kaj Sotala's Chapter in "Artificial Intelligence Safety and Security" on Disjunctive Scenarios of Catastrophic AI Risk on page 318: > "An AI which was simply seeking to maximize human happiness in general might be willing to sacrifice all currently living humans, if it thought that this allowed it to create more happy humans later on." Introduction The potential dangers posed by a misaligned superintelligence have been extensively explored on this forum. However, in this short post, I will outline a moral dilemma that introduces the possibility that beings who are living on Earth when a superintelligence emerges would be at risk even if the superintelligence is value-aligned[1]. In one sentence; once it can be done safely, the immense benefit of colonising the accessible universe quickly may vastly outweigh the welfare of beings living on Earth at the time colonisation becomes possible, potentially justifying extreme disregard for their welfare. Now, this is not the life-or-death question facing humanity, but it is potentially a life-or-death question that a generation of humans might have to consider. If nothing else, it is a scenario that may challenge readers to consider some of the critical assumptions that they use to justify longtermism and/or their motivations for working on the alignment problem. In this post, I will simply outline the dilemma. If the dilemma turns out to be of interest to the community I will write a follow-up post with a more extensive examination of the scenario. The AI loves you, but you are made of atoms which it can use to help create an inconceivably large number