According to Aumann’s Agreement theorem, such a concurrence provides a tiny amount of Bayesian evidence that you’re onto something.
What? That's... not AAT at all.
such pleasure-wizards, to put it bluntly, do not exist... But their opposites do.
What possible justification could he have for this? "No one is better at happiness than others, but some people are worse at happiness" is obviously impossible, and if the claim is that there's a plateau of "normal" people who are all roughly equivalent at converting resources into happiness and then people who are worse than that plateau, that sounds more like wishful thinking than a justified empirical claim.
On closer inspection it was not hard to see why Carter had developed theories so close to those of Eliezer and other members of Less Wrong and SIAI communities.
They really don't look that similar to me; they're looking at very different problems and have very different approaches.
The basic problem is that utilitarianism simply doesn't work.
Carter takes the common critique of total utilitarianism and the common critique of average utilitarianism, and says "well, both critiques go away if we try to maximize a combination of total and average." But those are just the common critiques, not the most potent ones. The basic problem with utilitarianism is that utility is difficult to measure and impossible to compare- and so both total and average utilitarianism are not things that can actually be calculated.
Eliezer is trying to tackle the problem of what utilities actually cash out as, so that you can build a machine that can perform preference calculations and not get them horribly wrong. Will Alice be happier with an unlimited supply of cookies, or if she has to strive for them? The options satisfy different desires in different amounts, and so fun theory and complexity of value deal with the tradeoffs between different desires. If you just built a machine that knew about our desire to feel happy and didn't know about our desire to impact the real world, you would get a population of wireheads- something that many of us think would be a bad outcome now, but cannot justify that judgment in terms of average or total 'happiness.'
...knew about our desire to feel happy and didn't know about our desire to believe we're impacting the real world
Do you really mean this, as opposed to "our desire to impact the real world"?
The basic problem is that utilitarianism simply doesn't work.
...
Eliezer is trying to tackle the problem of what utilities actually cash out as, so that you can build a machine that can perform preference calculations and not get them horribly wrong. Will Alice be happier with an unlimited supply of cookies, or if she has to strive for them? The options satisfy different desires in different amounts, and so fun theory and complexity of value deal with the tradeoffs between different desires.
Utilitarianism, defined as an ethical theory that only has values involving happiness and suffering, doesn't work. But values dealing with happiness and suffering are a subset of our values, which also include freedom, challenge, impact on the real world, friendship, novelty, and so on. Carter's point can be extended fairly easily to these: just as we don't only value total happiness but also average happiness, we don't only value happiness but also all those other things. Carter was aiming to solve a problem with utilitarianism, but he managed to find a solution that extends to consequentialism in general.
But values dealing with happiness and suffering are a subset of our values, which also include freedom, challenge, impact on the real world, friendship, novelty, and so on.
Are those values an ideal consequentialist should have? Consider "I want to help people" and "I want people to be helped."
I don't think there's any one "ideal consequentialist". Some values may be more important to me than others, and I may want to self-modify to care less about some of those things in order to maximize the others, but no terminal values are themselves better or worse. My utility function is what it is.
The issue is that "freedom" and "challenge" aren't really outcome preferences so much as they are action preferences or decision theory preferences. The consequentialist doesn't see a difference between me proposing that we trade two of my apples for three of your lemons and a dictator ordering that we trade two of my apples for three of your lemons- the outcome is how many of which fruit each of us ends up with, and if the dictator is better at negotiating and knowing our preferences than we are, the consequentialist suggests that we use the dictator and get over our preference for freedom (which was useful when dictators were bad but isn't useful when dictators are good).
You can smuggle other moral systems into consequentialism- by, say, including features of the decision tree as part of the outcome- but it's far cleaner to just discard consequentialism.
I have different subjective experiences when I am making my own decisions and when I am doing something I was ordered to do, even if it's the same decision and action both times. This suggests "my having freedom" is a real quality of a state of the world, and that therefore I can have consequentialist preferences about its presence vs. its absence. Anything that can distinguish two states of the world is a valid thing consequentialists can have values about.
In the trolley problem, an agent will have different subjective experiences in the case where they do nothing and in the case where they murder someone. Most consequentialist prescriptions count such preferences as insignificant in light of the other outcomes.
I do think that most consequentialists go further, claiming that only the final world state should matter and not how you got there, but I agree with you that consequentialist tools are powerful enough to adopt systems that are typically seen as competing- like deontology- and the reverse is true as well. Because there's that flexibility in tools, I find conversations are easier if one adopts strict system definitions. If someone uses expected utility theory to pick an action, but their utility is based on the rules they followed in choosing actions, I don't see the value in calling that consequentialism.
The consequentialist doesn't see a difference between me proposing that we trade two of my apples for three of your lemons and a dictator ordering that we trade two of my apples for three of your lemons- the outcome is how many of which fruit each of us ends up with, and if the dictator is better at negotiating and knowing our preferences than we are, the consequentialist suggests that we use the dictator and get over our preference for freedom
I think your quandary can be resolved by dividing your example into more than one consequence.
Example 1 has the consequences:
Example 2 has the consequences:
I'm making up numbers here, but imagine you assign +10 utility to the consequence "end up with +3 lemons and -2 apples," +1 utility to the consequence "think hard and make a decision." and -3 to the consequence "dictator tells me what to do." Then in Example 1 the two consequences have a combined utility of 7, whereas in Example 2 they have a combined utility of 11.
In the trolley problem, an agent will have different subjective experiences in the case where they do nothing and in the case where they murder someone. Most consequentialist prescriptions count such preferences as insignificant in light of the other outcomes.
I think one reason that subjective experiences don't matter in the trolly problem is that the stakes are so high. In the trolley problem your desire not to be involved in someone's death is nothing compared to the desire of six people to not die. If the stakes were much lower, however, your subjective experiences might matter.
For instance, imagine a toned down trolley problem where if you do nothing Alice and Bob will get papercuts on their thumbs, and if you pull a switch Clyde will get a papercut on his thumb. In that case the stakes are low enough that the unpleasant feeling you get from pulling the switch and injuring someone might merit some consideration.
This is actually similar to how the preference for freedom is treated in real life. When the stakes are low freedom is respected more often, even if it sometimes leads to some bad consequences, but when they are higher (during war, viral epidemics, etc) freedom is restricted because the stakes are much higher. (Of course, it goes without saying that in real life treating freedom like this tends to encourage corruption)
What? That's... not AAT at all.
My understanding was that if two equally rational people have the same information they will draw the same conclusions. So if someone draws the same conclusion as you, that's evidence in your favor, but only very mild evidence since you don't know for sure if they're as rational as you and if they had the exact same info as you. I do think that might be a weak opening and am thinking of changing it.
What possible justification could he have for this? "No one is better at happiness than others, but some people are worse at happiness" is obviously impossible, and if the claim is that there's a plateau of "normal" people who are all roughly equivalent at converting resources into happiness and then people who are worse than that plateau, that sounds more like wishful thinking than a justified empirical claim.
It's perfectly true. People with severe medical problems need huge amount of resources merely to satisfy their preferences of "not being dead" and "not being sick." A normal person require far less resources to satisfy that preference. So people with severe medical problems are "reverse utility monsters." This is true regardless of whether you are a preference or happiness utilitarian, since obviously you need to be alive to be happy at all and healthy to reach your full happiness potential (FYI I'm a preference utilitarian).
The basic problem is that utilitarianism simply doesn't work.
Normal_Anomaly made most of the points I was going to, so I won't elaborate on this.
What? That's... not AAT at all.
My understanding was that if two equally rational people have the same information they will draw the same conclusions.
AAT says that if two people (who may have observed different evidence) have mutual knowledge that they are both perfect epistemic rationalists honestly reporting their posterior probabilities, then they cannot remain in an equalibrium where they both are aware of the other's posterior probabilities but disagree on their posterior probabilities.
They need to have the same priors? Wouldn't that make AAT trivial and vacuous?
I thought the requirement was that priors just weren't pathologically tuned.
I'm pretty sure they do need to have the same priors.
My intuition is that AAT is basically saying that the perfect epistemic rationalists involved can essentially transfer all of the evidence that they have to the other, so that each one effectively has the same evidence and so should have the same posteriors...except that they'll still have different posteriors unless they began with the same priors.
If they found that they had different priors, I think that they could just communicate the evidence which led them to form those priors from previous priors and so forth, but I think that if they trace their priors as far back as possible and find that they have different ones, AAT doesn't work.
I'm not actually super-familiar with it, so update accordingly if I seem to have said something dumb.
Nope, I was wrong. It is the case that agents require equal priors for ATT to hold. AAT is like proving that mixing the same two colors of paint will always result in the same shade or that two equal numbers multiplied by another number will still be equal.
What a worthless theorem!
I guess when I read that AAT required "common priors" I assumed Aumann must mean known priors or knowledge of each others' priors, since equal priors would constitute both 1) an asinine assumption and, 2) a result not worth reporting. Hanson's assumption that humans should have a shared prior by virtue of being evolved together is interesting, but more creative than informative.
Good thing I don't rely on ATT for anything. It's obvious that disagreeing with most people is rational so updating on people's posteriors without evidence would be pretty unreasonable. I'm not surprised that ATT would turn out to be so meaningless.
Yes, and even have mutual knowledge that they have the same priors. Which I was thinking, but apparently failed to actually type.
AAT is very specific. Independent invention is evidence for the attractiveness of an idea (and thus, one hopes, its truthfulness) but it's unrelated to AAT.
People with severe medical problems need huge amount of resources merely to satisfy their preferences of "not being dead" and "not being sick."
Sure. But you're telling me that two healthy individuals are equally efficient at converting resources into happiness? What evidence is there that the Brahmin is not ten times as capable of happiness as the Untouchable?
I have edited my post to remove the reference to AAT per your and JGWeissman's advice.
Sure. But you're telling me that two healthy individuals are equally efficient at converting resources into happiness? What evidence is there that the Brahmin is not ten times as capable of happiness as the Untouchable?
In Carter's paper he was discussing a hypothetical pleasure wizard so good at converting resources into happiness that it was better to give it all the resources in the world and let everyone else just have enough for a life barely worth living. It seems unlikely that such an extreme pleasure wizard exists, although it's quite possible for there to be some variation among people considered "normal." The psychological unity of humankind provides some evidence against extreme pleasure wizards' existence, although it's far from conclusive.
Preference utilitarianism makes things more complicated since someone may be inefficient at producing happiness, but efficient at satisfying some of their other preferences. However, since being alive and healthy are nearly universal preferences, I think that it's still accurate to call someone with severe illness a reverse pleasure wizard, even if you value preference satisfaction rather than happiness.
Even if you're right and Carter was mistaken in stating that pleasure wizards don't exist, that doesn't alter his main point, which is that equality is valuable for its own sake, and therefore it is immoral to give all resources to pleasure wizards, no matter how efficient they are.
AAT is very specific: if you are absolutely certain that someone else is rational, then your posteriors and their posteriors must be equal.
You need to have common knowledge of each other's estimates, common knowledge of each other's rationality, and common priors.
I thought you didn't need common priors, and I was wrong. Editing. (I might have had in mind Hanson's result that if you agree on the method to generate priors, then that's enough.)
I actually think that utility monsters (of a limited form) do exist: they're called children.
People will often go out of their way to satisfy the preferences of children, even at some inconvenience to themselves. Fortunately, childrens' preferences are often pretty easy to satisfy, but in general it seems that we weight them higher than we do those of most adults, suggesting that they are at least minor utility monsters.
My understanding of 'Utility Monster' is someone who gets increasing utility per resource unit when greater resources are spent (greater than linear return). For example it would get utility of 1x when getting one cupcake, and utility of >2x when getting 2 cupcakes.
If such a monster existed, you would increase the average utility by giving ALL of everyone's resources to it.
A child may get more enjoyment from the same amount of resource than an adult in some situations, but you don't raise the average utility by giving the entire cupcake to just one of your 2 children.
I thought the point of a utility monster was simply that they were more efficient at turning resources into utility - I suspect both notions serve to make similar points! I agree that children aren't utility monsters in the sense that you describe, but I think they are at least agents that turn resources into utility more efficiently than adults. I think the claim that such beings could exist is a conclusion that some have claimed is a dodgy consequence to be avoided - which obviously isn't the case if such things already exist.
I actually think that utility monsters (of a limited form) do exist: they're called children.
I agree, kind of. Reading this article changed my intuition about "utility monsters" by describing their opposites, the "dysutility monsters". Then I realized that if we convert money to utilons, the difference is not only in a pleasure derived from the results (which could be maximized by a psychopath), but also in ability to convert the money into meaningful results. Also in real life the "utility monster" does not need the ability to convert money to utilons infinitely, only in the range we consider now.
Therefore, "utility monster" could simply be someone who is able to translate the resources into growth more efficiently than others. Such as a child. The child can efficiently use only a limited amount of resources, and then the "utility monster" effect turns off, but within this range, it makes utilitarian sense to make some sacrifices.
Another example could be a poor family with many children, one of them exceptionally gifted, others average. It would make sense to allocate more resources to the gifted child, especially under assumption that after the child successfully converts resources into status and income, they will support their siblings, so this arrangement is even in the other siblings' net benefit. Assuming that exceptional people are a net benefit for their society, it could be useful to give more resources to education of gifted people. (Though in real life it would be difficult to check that the resources are really efficiently used for education, and not e.g. burned in a signalling contest.)
But this is all based on the assumption that for a human, the "utility monster" effect is only temporary, and that the boundaries are reasonable. And also it assumes our ability to measure other people's utility. In real life, people sometimes spend of lot of money for their children, but children use the money for immediate pleasure instead of education -- so they may seem like "utility monsters" to their parents, even when they are not.
Although you try to correct this on the footnote, your post still gives the idea Alan Carter created this concept. Value pluralism has been going on for quite some time and is recognized as a defensible position in ethics to the extent of having a SEP entry to it: http://plato.stanford.edu/entries/value-pluralism/ . More importantly, if you are aiming at bring Eliezer's ideas closer to mainstream philosophy, then I don't think Alan should be your choice. Not because he is not part of mainstream philosophy but because there are much bigger names defending some sort of value pluralism: Isaiah Berlin, Bernard Williams, Thomas Nagel, Larry Temkin and so on. In fact, some people argue that Stuart Mill was not value-monist. There is also the position called moral particularism which claims morality does not consist solely of general principles or guidelines but it is extremely context-dependent (which seems to mean it would be hard to compress), a position the US Supreme Court seems to adopt.
I have a question about the mere addition paradox. Why is it so often quietly accepted as a background assumption that adding more happiness (in the form of adding more people leading "worthwhile lives") is inherently an improvement over having a smaller population where the individual experiences more happiness? This seems like it presupposes some kind of "Fill all the Earth with your offspring" clause in one's utility function; that we should take into account the happiness of people that don't even exist.
Certainly, there are people who live by a "Fill the world" imperative of this sort, but I've never seen it spelled out when the mere addition paradox is presented. So, am I missing something? Is the "Fill the world"-clause officially a built-in feature of the question that no one bothers to spell out, or is there some other undisclosed reason for taking nonexistent people into account?
Well, if I believe that ending existing worthwhile lives is a bad thing to do (perhaps because contemplating doing so feels icky), and I believe generally that what makes an act bad is that the state of the world after performing that act is worse than the state of the world after not performing it, it's easy (perhaps not justified, but certainly easy) for me to conclude therefrom than more existing worthwhile lives is better than fewer.
It's possible that deontologists are less subject to this... they might believe that ending existing worthwhile lives is a bad thing to do, but just embrace that as a rule to follow, without any implication that worthwhile lives are themselves valuable.
Hmm. I hadn't looked at it from the angle of the implications of reversing the "murder is bad" maxim. Thanks.
It doesn't feel very satisfying, though. The questions of whether to add new people or not and whether to subtract existing people or not seem like two entirely different things; trying to address both of them with one ethical recipe about the number of worthwhile lives being lead doesn't seem justified - after all, only one of the questions deals with existing entities. Frankly, I'm more inclined to think it flat-out isn't justified to conclude that more (worthwhile) lives should be produced just from the rule/preference/whatever that pre-existing worthwhile lives shouldn't be ended; that conclusion looks like it requires some additional moral framework.
While that may be a true point about deontologists being less subject to it, ad-hoc deontological injunctions aren't necessary for consistently disliking murder and disliking overpopulation (i.e. situation A). Simply not assigning equal value to existing and non-existing lives does the trick just fine.
...Which is my kind of my Square One here. I just don't see how the whole line of argument leading towards the Repugnant Conclusion even gets off the ground unless equal value is assumed, or why the assumption should be made.
Well, it does seem to me that there are two different questions one can be asking here, and it's potentially valuable to separate them.
One question is "Why do people assume that making hypothetical people actual increases the net value of the world?" That's a question about psychology, though it may also be a moral question, or it may not, I'm not really sure. I tend to believe that moral questions are a subset of psychological questions in the same sense that economic questions are, but I don't feel able to defend that belief against serious challenge.
Anyway, I find that my tendency to assume that moral value is conserved through multiple iterations of a hypothetical reversible operation is a strong one. That is, if killing someone removes utility, then bringing them back to life should add roughly the same utility... otherwise I could in principle kill and resurrect the same person a million times in an instant, ending up with the world in the same state it started in but with massive utility gains or losses coming from no net state change, which seems... bizarre.
So the line of argument that leads to the Repugnant Conclusion doesn't seem all that alien to me, although as with a lot of thought experiments along these lines, my takeaway from the RC is that value has more than one source, and while more people may mean more value all else being equal, when adding more people starts to require trading off other sources of value, all else is no longer equal.
The other question is "Is it justified to assume that making hypothetical people actual increases the net value of the world?" I don't really know how to even approach this question, except maybe by asking what the expected results of assuming it are, which mostly doesn't seem like what people who ask this question mean.
Yeah, that first question is the one I'm stuck at. From my point of view it just looks like the utility function that assigns value to hypothetical people has to have an error somewhere... but then again, that might be a byproduct of some problem in me rather than them. Possible, though it sure doesn't seem likely from in here. Still, I do wonder what psychological fact I'd have to acquire that'd make sense of that perspective.
Anyway, I find that my tendency to assume that moral value is conserved through multiple iterations of a hypothetical reversible operation is a strong one. That is, if killing someone removes utility, then bringing them back to life should add roughly the same utility... otherwise I could in principle kill and resurrect the same person a million times in an instant, ending up with the world in the same state it started in but with massive utility gains or losses coming from no net state change, which seems... bizarre.
Well, trivially and possibly in violation of the spirit of the presented scenario, the effecting of those changes (such as switching a simulation of a person on and off at one million Hz) would itself consume energy, and pouring perfectly usable energy into a status quo outcome is likely to be undesirable.
The other question is "Is it justified to assume that making hypothetical people actual increases the net value of the world?" I don't really know how to even approach this question, except maybe by asking what the expected results of assuming it are, which mostly doesn't seem like what people who ask this question mean.
The RC can be justified (at least up to some point) by appealing to probable real-world consequences, such as the added capacity of producing Fun (for an individual) that arises from a civilization of a sufficient size. Specialization, gains from trade, added Social Fun opportunities from having lots of people, etc. Things such as mounting interstellar rescue operations also seem easier if the same people who put the ship together don't need to design it. But all of this seems beside the point - assumptions added on top of the original thought experiment which, for whatever reason, treats the sum total of existing happiness as an intrinsic good as if the universe cares how many humans containing utilons it contains.
The only avenue of approach to the original thought experiment I was able to think of (that doesn't include added assumptions) is trying to place the first round of burden of proof on the side claiming A) that hypothetical people don't have value, and/or B) that the universe doesn't care. Even if this side accepts the burden of proof, it seems like these things should be possible to prove, however hard those proofs may be to formalize.
But suffice to say I think this burden lies on the other party first, and that such a proof, should it ever be formulated, would not be likely to turn out to make much sense, especially if actually applied. If we can be convinced to privilege hypothetical entities at the expense of currently existing ones, reality-played-straight ends up looking like a crack fic where resources go to those who're able to devise the most powerful mathematical notations for expressing the very large numbers of hypothetical people they've got stashed in their astral Pokéballs.
If we can be convinced to privilege hypothetical entities at the expense of currently existing ones, reality-played-straight ends up looking like a crack fic
Sure. OTOH, if we give hypothetical entities no weight at all, it seems to follow naturally that any project that won't see benefits within a century or so is not worth doing, since no actual people will benefit from it, merely hypothetical people who haven't yet been born.
Personally, I conclude that when planning for the future, I should plan based on the expected value of that future, which includes the value of entities I expect to exist in that future. Whether those entities exist right now or not -- that is, whether they are actual or hypothetical -- doesn't really matter.
I'm realizing I made some overly sweeping generalizations about "hypothetical people" there. Whoops.
Personally, I conclude that when planning for the future, I should plan based on the expected value of that future, which includes the value of entities I expect to exist in that future.
This, I don't disagree with. Optimizing for the people we expect to exist seems fine to me; it's the normative leap from that to "we should produce more people" that throws me off.
The distinction between those two things gets a little tricky for me to hold on to when one of the things that significantly contributes to my expectations about the existence of someone is precisely how much I value them existing... or, more precisely, how much I expect my future self to value them if and when the opportunity to create them presents itself. E.g., if I really don't want a child, my expectation of a child of mine existing in the future should be lower than if I really want one.
Conversely, if I expect an entity X to exist a year from now if things remain as they are now, and I judge that X would, if actual, make the world worse, it seems to follow that I should take steps to prevent X from becoming actual.
It seems moderately clear to me that, while I value more people rather than fewer all else being equal, that's not a particularly important value; there are lots of things that I'll trade it for.
I just don't see how the whole line of argument leading towards the Repugnant Conclusion even gets off the ground unless equal value is assumed, or why the assumption should be made.
There are many lines of attack on this, but consider the case where you are choosing between different futures where all currently-existing humans are dead. Then, refusing to assign equal value to existing and non-existing lives doesn't buy you anything.
Interesting... Thanks for weighing in!
I can sort of get behind that reasoning on a thought experiment level, but it's harder to put to practice. Avoiding the more artificial scenarios where there's a mass extinction followed by a deliberate repopulation of the world with new people, the transition to any future state would be a gradual process that always involves actual, living people as intermediaries - who, being living, should (IMO) be accordingly valued.
That, and in my mind the privilegedness of existing versus non-existing lives arises not only from the fact that existing lives have greater value, but from the fact that non-existing lives have zero value; and multiplying that with whatever will still leave zero. The value of those lives would only become realized when their existence did, at which point we'd still be left with the same problem of resource allocation: if we went for a more situation B-ish solution at some past point in time, the people of our chosen future would have less resources per person to enjoy their lives with. In this case, too, it seems correct to plan for situation A.
Gastogh says:
Hmm. I hadn't looked at it from the angle of the implications of reversing the "murder is bad" maxim. Thanks. It doesn't feel very satisfying, though. The questions of whether to add new people or not and whether to subtract existing people or not seem like two entirely different things; trying to address both of them with one ethical recipe about the number of worthwhile lives being lead doesn't seem justified
I agree. My intuition is that, when calculating average wellbeing, you include dead people, and you include whomever is going to end up existing in the future, but not potential people who will never exist unless you take action. So killing someone lowers average wellbeing, but failing to create someone does not. A person who manages to live as long as they possibly can with good quality of life dies with a big positive contribution to average wellbeing, a person who dies prematurely has a much lower contribution and is a permanent black mark on our collective moral records. A person who never exists, however, isn't factored into the equation.
Of course, taken by itself this might imply the Problem of the Ecstatic Psychopath. But, as I state in my main post, average wellbeing isn't the only value, though it is an important one. Total wellbeing (having lots of people who contribute fun and other positive values to the world) is important too, sometimes it may be worth risking someone lowering the average if they increase the total..
Avoiding the more artificial scenarios where there's a mass extinction followed by a deliberate repopulation of the world with new people, the transition to any future state would be a gradual process that always involves actual, living people as intermediaries - who, being living, should (IMO) be accordingly valued.
The quandary we have seems to be that it seems like we have a duty to make sure people who don't exist yet, but will in the future, will have satisfied preferences, but also think that we have no duty to satisfy the hypothetical preferences of hypothetical people by creating them.
I've concluded that the primary reason this seems like a quandary is that we are trying to apply the Person-Affecting Principle to situations where it doesn't work. The Person Affecting Principle states, in short that an event is only good or bad if it makes things better or worse for some specific person. This works fine in situations where the population doesn't grow. However, in instances where it does, it gives insane-seeming results.
The classic example is this: imagine a plan to store nuclear waste in one of two places. In one place it'll keep forever, in another it will leak and kill everyone in the area in 500 years. However, due to the Butterfly Effect, what plan you pick will result in different people meeting, mating, and having children, so the future generations in the alternate storage scenarios will be composed of different individuals. So the choices aren't better or worse for any specific person, because they change what people will end up existing. According to the PAP neither scenario is better or worse.
I've concluded that this can be resolved by replaced the Person Affecting Principle with the World Creating Principle. The World Creating principle states that an event is good or bad if it creates a world which has lower (average utility)+(total utility)+(equality utility)+(other relevant complex values) for the world it creates, whomever the inhabitants end up being. So storing the nuclear waste permanently is a good thing, all other things being equal.
The Person Affecting Principle is a special case of the World Creating Principle, in the same way Newtonian physics is a special case of General Relativity. It's the World Creating Principle applied in special cases where there is no potential for population growth.
I might expand this thinking into a post at some point.
I'm afraid I'm downvoting this, not because I generally disagree with it, but because this is obvious stuff you should derive yourself. Nobody should have to read philosophy to come up with the ideas of value monism versus pluralism, average utilitarianism versus assigning utility to low variance of the utility distribution, or utility and disutility monsters and their consequences. This is a step backwards from other posts on the site addressing the issue.
It’s always good news when someone else develops an idea independently from you. It's a sign you might be onto something. Which is why I was excited to discover that Alan Carter, Professor Emeritus of the University of Glasgow’s Department of Philosophy, has developed the concept of Complexity of Value independent of Less Wrong.
As far as I can tell Less Wrong does not know of Carter, the only references to his existence I could find on LW and OB were written by me. Whether Carter knows of LW or OB is harder to tell, but the only possible link I could find online was that he has criticized the views of Michael Huemer, who knows Bryan Caplan, who knows Robin Hanson. This makes it all the more interesting that Carter has developed views on value and morality very similar to ones commonly espoused on Less Wrong.
The Complexity of Value is one of the more important concepts in Less Wrong. It has been elaborated on its wiki page, as well as some classic posts by Eliezer. Carter has developed the same concept in numerous papers, although he usually refers to it as “a plurality of values” or “multidimensional axiology of value.” I will focus the discussion on working papers Carter has on the University of Glasgow’s website, as they can be linked to directly without having to deal with a pay wall. In particular I will focus on his paper "A Plurality of Values."
Carter begins the paper by arguing:
Wouldn’t it be nice if we were to discover that the physical universe was reducible to only one kind of fundamental entity? ... Wouldn’t it be nice, too, if we were to discover that the moral universe was reducible to only one kind of valuable entity—or one core value, for short? And wouldn’t it be nice if we discovered that all moral injunctions could be derived from one simple principle concerning the one core value, with the simplest and most natural thought being that we should maximize it? There would be an elegance, simplicity and tremendous justificatory power displayed by the normative theory that incorporated the one simple principle. The answers to all moral questions would, in theory at least, be both determinate and determinable. It is hardly surprising, therefore, that many moral philosophers should prefer to identify, and have thus sought, the one simple principle that would, hopefully, ground morality.
And it is hardly surprising that many moral philosophers, in seeking the one simple principle, should have presumed, explicitly or tacitly, that morality must ultimately be grounded upon the maximization of a solitary core value, such as quantity of happiness or equality, say. Now, the assumption—what I shall call the presumption of value-monism—that here is to be identified a single core axiological value that will ultimately ground all of our correct moral decisions has played a critical role in the development of ethical theory, for it clearly affects our responses to certain thought-experiments, and, in particular, our responses concerning how our normative theories should be revised or concerning which ones ought to be rejected.
Most members of this community will immediately recognize the similarities between these paragraphs and Eliezer’s essay “Fake Utility Functions.” The presumption of value monism sounds quite similar to Eliezer’s description of “someone who has discovered the One Great Moral Principle, of which all other values are a mere derivative consequence.” Carter's opinion of such people is quite similar to Eliezer's.
While Eliezer discovered the existence of the Complexity of Value by working on Friendly AI, Carter discovered it by studying some of the thornier problems in ethics, such as the Mere Addition Paradox and what Carter calls the Problem of the Ecstatic Psychopath. Many Less Wrong readers will be familiar with these problems; they have been discussed numerous times in the community.
For those who aren’t, in brief the Mere Addition Paradox states that if one sets maximizing total wellbeing as the standard of value then one is led to what is commonly called the Repugnant Conclusion, the belief that a huge population of people with lives barely worth living is better than a somewhat smaller population of people with extremely worthwhile lives. The Problem of the Ecstatic Psychopath is the inverse of this, it states that, if one takes average levels of well-being as the standard of value, that a population of one immortal ecstatic psychopath with a nonsentient machine to care for all their needs is better than a population of trillions of very happy and satisfied, but not ecstatic people.
Carter describes both of these problems in his paper and draws an insightful conclusion:
In short, surely the most plausible reason for the counter-intuitive nature of any mooted moral requirement to bring about, directly or indirectly, the world of the ecstatic psychopath is that either a large total quantity of happiness or a large number of worthwhile lives is of value; and surely the most plausible reason for the counter-intuitive nature of any mooted injunction to bring about, directly or indirectly, the world of the Repugnant Conclusion is that a high level of average happiness is also of value.
How is it that we fail to notice something so obvious? I submit: because we are inclined to dismiss summarily any value that fails to satisfy our desire for the one core value—in other words, because of the presumption of value-monism.
Once Carter has established the faults of value monism he introduces value pluralism to replace it.1 He introduces two values to start with, “number of worthwhile lives” and “the level of average happiness,” which both contribute to “overall value.” However, their contributions have diminishing returns,2 so a large population with low average happiness and a tiny population with extremely high average happiness are both worse than a moderately sized population with moderately high average happiness.
This is a fairly unique use of the idea of the complexity of value, as far as I know. I’ve read a great deal of Less Wrong’s discussion of the Mere Addition Paradox, and most attempts to resolve it have consisted of either trying to reformulate Average Utilitarianism so that it does not lead to the Problem of the Ecstatic Psychopath, or redefining what "a life barely worth living" means upwards so that it is much less horrible than one would initially think. The idea of agreeing that increasing total wellbeing is important, but not the be all and end all of morality, did not seem to come up, although if it did and I missed it I'd be very happy if someone posted a link to that thread.
Carter’s resolution of the Mere Addition Paradox makes a great deal of sense, as it manages to avoid every single repugnant and counterintuitive conclusion that Total and Average Utilitarianism draw by themselves while still being completely logically consistent. In fact, I think that most people who reject the Repugnant Conclusion will realize that this was their True Rejection all along. I am tempted to say that Carter has discovered Theory X, the hypothetical theory of population ethics Derek Parfit believed could accurately describe the ethics of creating more people without implying any horrifying conclusions.
Carter does not stop there, however, he then moves to the problem of what he calls “pleasure wizards” (many readers may be more familiar with the term “utility monster”). The pleasure wizard can convert resources into utility much more efficiently than a normal person, and hence it can be argued that it deserves more resources. Carter points out that:
…such pleasure-wizards, to put it bluntly, do not exist... But their opposites do. And the opposites of pleasure-wizards—namely, those who are unusually inefficient at converting resources into happiness—suffice to ruin the utilitarian’s egalitarian pretensions. Consider, for example, those who suffer from, what are currently, incurable diseases. … an increase in their happiness would require that a huge proportion of society’s resources be diverted towards finding a cure for their rare condition. Any attempt at a genuine equality of happiness would drag everyone down to the level of these unfortunates. Thus, the total amount of happiness is maximized by diverting resources away from those who are unusually inefficient at converting resources into happiness. In other words, if the goal is, solely, to maximize the total amount of happiness, then giving anything at all to such people and spending anything on cures for their illnesses is a waste of valuable resources. Hence, given the actual existence of such unfortunates, the maximization of happiness requires a considerable inequality in its distribution.
Carter argues that, while most people don’t think all of society’s resources should be diverted to help the very ill, the idea that they should not be helped at all also seems wrong. He also points out that to a true utilitarian the nonexistence of pleasure wizards should be a tragedy:
So, the consistent utilitarian should greatly regret the non-existence of pleasure-wizards; and the utilitarian should do so even when the existence of extreme pleasure-wizards would morally require everyone else to be no more than barely happy.
Yet, this is not how utilitarians behave, he argues, rather:
As I have yet to meet a utilitarian, and certainly not a monistic one, who admits to thinking that the world would be a better place if it contained an extreme pleasure-wizard living alongside a very large population all at that level of happiness where their lives were just barely worth living…But if they do not bemoan the lack of pleasure-wizards, then they must surely value equality directly, even if they hide that fact from themselves. And this suggests that the smile of contentment on the faces of utilitarians after they have deployed diminishing marginal utility in an attempt to show that their normative theory is not incompatible with egalitarianism has more to do with their valuing of equality than they are prepared to admit.
Carter resolves the problem of "pleasure wizard" by suggesting equality as an end in itself as a third contributing value towards overall value. Pleasure wizards should not get all the resources because equality is valuable for its own sake, not just because of diminishing marginal utility. As with average happiness and total worthwhile lives, equality is balanced against other values, rather than dominating them. It may often be ethical for a society to sacrifice some amount of equality to increase the total and average wellbeing.
Carter then briefly states that, though he only discusses three in this paper, there are many other dimensions of value that could be added. It might even be possible to add some form of deontological rules or virtue ethics to the complexity of value, although they would be traded off against consequentialist considerations. He concludes the paper by reiterating that:
Thus, in avoiding the Repugnant Conclusion, the Problem of the Ecstatic Psychopath and the problems posed by pleasure-wizards, as well as the problems posed by any unmitigated demand to level down, we appear to have identified an axiology that is far more consistent with our considered moral judgments than any entailing these counter-intuitive implications.
Carter has numerous other papers discussing the concept in more detail, but “A Plurality of Values” is the most thorough. Other good ones include “How to solve two addition paradoxes and avoid the Repugnant Conclusion,” which more directly engages the Mere Addition Paradox and some of its defenders like Michael Huemer; "Scrooge and the Pleasure Witch," which discusses pleasure wizards and equality in more detail; and “A pre-emptive response to some possible objections to a multidimensional axiology with variable contributory values,” which is exactly what it says on the tin.
On closer inspection it was not hard to see why Carter had developed theories so close to those of Eliezer and other members of Less Wrong and SIAI communities. In many ways their two tasks are similar. Eliezer and the SIAI are trying to devise a theory of general ethics that cannot be twisted into something horrible by a rules-lawyering Unfriendly AI, while Carter is trying to devise a theory of population ethics that cannot be twisted into something horrible by rules-lawyering humans. The worlds of the Repugnant Conclusion and the Ecstatic Psychopath are just the sort of places a poorly programmed AI with artificially simple values would create.
I was very pleased to see an important Less Wrong concept had a defender in mainstream academia. I was also pleased to see that Carter had not just been content to develop the concept of the Complexity of Value. He was also able to employ in the concept in new way, successfully resolving one of the major quandaries of modern philosophy.
Footnotes
1I do not mean to imply Carter developed this theory out of thin air of course. Value pluralism has had many prominent advocates over the years, such as Isaiah Berlin and Judith Jarvis Thomson.
2Theodore Sider proposed a theory called "geometrism" in 1991 that also focused on diminishing returns, but geometrism is still a monist theory, it had geometric diminishing returns for the people in the scenario, rather than the values creating the people was trying to fulfill.
Edited - To remove a reference to Aumann's Agreement Theorem that the commenters convinced me was unnecessary and inaccurate.