This is a draft written by FSH, Postdoc at Hong Kong University, as part of the Center for AI Safety Philosophy Fellowship. This draft is meant to solicit feedback.

Abstract:

If a future AI system can enjoy far more well-being than a human per resource, what would be the best way to allocate resources between these future AI and our future descendants? It is obvious that on total utilitarianism, one should give everything to the AI. However, it turns out that every Welfarist axiology on the market also gives this same recommendation. Without resorting to deontological normative theories that suggest that we ought not always to create the world with the most value, or non-welfarist theories that tell us that the world with the most value may not be the world with the most welfare, we propose a new theory that justifies the survival of humanity in the face of overwhelming AI wellbeing. We call this new theory, ``Group Prioritarianism".

Introduction:

In 1974, Robert Nozick famously introduced the idea of the ``utility monster", a being that is so efficient at converting resources into utility, that the best way to maximize utility in the world is to give all resources to this one monster at the cost of all of humanity. Nozick goes on to say the mere possibility of the utility monster refutes ``Utilitarian theories", which imply that one ought to sacrifice all humanity into the maws of this utility monster. Such a consequence is taken to be so outlandish that any theory that suggests that humanity ought to be sacrificed to the monster is, in Nozick's words, ``embarassed" [1974, p.41].

In this paper, we will take as a fixed point that Nozick is right in this case: it really is wrong to sacrifice all humanity to the monster, even if doing so increases total well-being^[1]. But even if we take this as a fixed-point, many important questions are left unanswered. For example, should we create these utility monsters if we had the option? And if these monsters do come to exist, and if we ought not to give everything to the monsters, then how much should we give to these monsters? And if Utilitarianism cannot accommodate our intuitions about distributive welfare in this case, then what kind of normative theory can?

These questions have gained a new urgency as the utility monster no longer exists as a mere thought experiment...

Posts

Wikitag Contributions

Comments