Mundane Mandy: ordinary conception of what a “good world” looks like, i.e. your friends and family living flourish lives in their biological bodies, with respect for “sacred” goods
Galaxy-brain Gavin: transhumanist, longtermist, scope-sensitive, risk-neutral, substrate-indifferent, impartial

I think Mundane Mandy should have the proximal lightcone (anything within 1 billion light years) and Galaxy-brain Gavin should have the distal lightcone (anything 1-45 B ly). This seems like a fair trade.

[-]Nina Panickssery7mo98

Not a fair trade, but also present-day "Mundane Mandy" does not want to risk everything she cares about to give "Galaxy-brain Gavin" the small chance of achieving his transhumanist utopia.

[-]cubefox7mo53

If Galaxy-brain Gavin's theory has serious counterintuitive results, like ignoring the preference of current humans that humanity be not replaced by AI in the future, then Gavin's theory is not galaxy-brained enough.

[-]ryan_greenblatt7mo51

Does mundane mandy care about stuff outside the solar system? Let alone stuff which is over 1 million light years away.

(Separately, I think the distal light cone is more like 10 B ly than 45 B ly as we can only reach a subset of the observable universe.)

[-]Cleo Nardo7mo40

Yep, you might be right about the distal/proximal cut-off. I think that the Galaxy-brained value systems will end up controlling most of the distant future simply because they have a lower time-preference for resources. Not sure where the cut-off will be.

For similar reasons, I don't think we should do a bunch of galaxy-brained acausal decision theory to achieve our mundane values, because the mundane values don't care about counterfactual worlds.

[-]Ishual5mo10

Is there actually (only) a small number of moral worldviews?

My own moral worldview cares about the journey and has a bunch of preferences for "not going too fast", "not losing important stuff", "not cutting lives short", "not forcing people to grow up much faster than they would like". But my own moral worldview also cares about not having the destination artificially limited. From my vantage point (which is admittedly just intuitions barely held together with duck tape), it seems plausible that there is a set of intermediate preferences between MM and GG, somewhat well-indexed by a continuous "comfortable speed". Here are some questions on which I think people might differ according to their "comfortable speed":
- how far beyond the frontier are you entitled to remain and still live a nice life? (ie how much should we subsidize people who wait for the second generation of upload tech before uploading?)
- how much risk are you allowed to take in pushing the frontier?
- how much consensus do we require to decide the path forward to greater capabilities? (eg choosing which of the following is legitimate: genetic edits, uploading, or artificial intelligence)
- how much control do we want to exert over future generations?
- how much do you value a predictible future?
- how comfortable are you with fast change?

If my "comfortable speed" model is correct, then maybe you would want to assign regions of the lightcone to various preferences according to some gradient.

There can also be preferences over how much the present variance within humanity keeps interacting in the future.

[-]Vladimir_Nesov7mo1513

Utilitarianism implies that if we build an AI that successfully maximizes utility/value, we should be ok with it replacing us

Under the premise of successful alignment, the "replacement" only happens if it's the right thing to do, which doesn't seem at all likely (depending on what "replacing" means). Optimization of the world doesn't destructively overwrite it with optimal stuff, it instead moves it in optimal ways starting with its actual initial state, which happens to include humanity.

[-]Nina Panickssery7mo62

There's no reason for me to think that my personal preferences (e.g. that my descendants exist) are related to the "right thing to do", and so there's no reason for me to think that optimizing the world for the "right things" will fulfil my preference.

I think most people share similar preferences to me when it comes to their descendants existing, which is why I expect my sentiment to be relatable and hope to collaborate with others on preventing humanity's end.

[-]AnthonyC7mo1210

There's no reason for me to think that my personal preferences (e.g. that my descendants exist) are related to the "right thing to do", and so there's no reason for me to think that optimizing the world for the "right things" will fulfil my preference.

This, and several of the passages in your original post such as, "I agree such a definition of moral value would be hard to justify," seem to imply some assumption of moral realism that I sometimes encounter as well, but have never really found convincing arguments for. I would say that the successionists you're talking to are making a category error, and I would not much trust their understanding of 'should'-ness outside normal day-to-day contexts.

In other words: it sounds like you don't want to be replaced under any conditions you can foresee.. You have judged. What else is there?

I can't really imagine a scenario where I "should" or would be ok with currently-existing-humans going extinct, though that doesn't mean none could exist. I can, however, imagine a future where humanity chooses to cease (most?) natural biological reproduction in favor of other methods of bringing new life into the world, whether biological or artificial, which I could endorse (especially if we become biologically or otherwise immortal as individuals). I can further imagine being ok with those remaining biological humans each changing (gradually or suddenly) various aspects of their bodies, their minds, and the substrates their minds run on, until they are no longer meat-based and/or no longer 'human' in various ways most people currently understand the term.

[-]Nina Panickssery7mo*40

This, and several of the passages in your original post such as, "I agree such a definition of moral value would be hard to justify," seem to imply some assumption of moral realism that I sometimes encounter as well, but have never really found convincing arguments for. I would say that the successionists you're talking to are making a category error, and I would not much trust their understanding of 'should'-ness outside normal day-to-day contexts.

I broadly agree.

I am indeed being a bit sloppy with the moral language in my post. What I mean to say is something like "insofar as you're trying to describe a moral realist position with a utility function to be optimized for, it'd be hard to justify valuing your specific likeness".

In a similar fashion, I prefer and value my family more than your family but it'd be weird for me to say that you also should prefer my family to your own family.

However, I expect our interests and preferences to align when it comes to preferring that we have the right to prefer our own families, or preferring that our species exists.

(Meta: I am extremely far from an expert on moral philosophy or philosophy in general, I do aspire to improve how rigorously I am able to articulate my positions.)

[-]Vladimir_Nesov7mo80

I expect it's the right thing to do, for the world to let you pursue your personal preferences. Values in decision theory sense are about the way the world develops, not about the way any given person lives their life, any more than laws of physics are about the way a person lives their life, even though it's certainly going to be in strict accordance with the laws of physics.

So decision theoretic values can easily stipulate that people should keep their differing preferences, have conflicts and disagreements, make mistakes and cause disasters, and so on. And that people with these properties should be part of the world simply because they already are, and not because they necessarily needed to exist in this form, with these preferences and with these conflicts and disagreements, if there was no humanity in the world from the outset.

[-]Cole Wyeth7mo3-1

The right thing to do is subjective, and what we’re trying for is to get our actual, individual or collective human values into AI systems. Specifying that is TBD and I think pretty confusing and hard. However, it seems that if we are all killed off and replaced, we would presumably have failed, since that is not something most of us want.

[-]Wei Dai7mo44

Nina is worried not just about humans getting killed and replaced, but also about humans not being allowed to have unenhanced children. It seems plausible that most humans, after reflection, would endorse some kind of "successionist" philosophy/ideology, and decide that intentionally creating an unenhanced human constitutes a form of child abuse (e.g., due to risk of psychological tendency to suffer, or having a much worse life on expectation than what's possible). It seems reasonable for Nina to worry about this, if she thinks her own values (current or eventual or actual) are different.

[-]Mikhail Samin7mo33

(btw i expect we’ll really want enhanced humans to have the capacity to suffer, because we have preferences around future people being able to experience the kinds of feelings we experience when we read stories, including very sad stories. Some suffering is reflectively endorsed and we enjoy it/wouldn’t want it to not happen; and it seems fine to want new humans and enhanced current humans to also have it, although maybe with more access to some control over it.)

[-]Cole Wyeth7mo30

Certainly an aligned AI can be a serious threat if you have sufficiently unusual values relative to whoever does the aligning. That worries me a lot - I think many possible “positive” outcomes are still somewhat against my interests and are also undemocratic, stripping agency from many people.
However, if this essay were capable of convincing “humanity” that they shouldn’t value enhancement, CEV should already have that baked in?

[-]Wei Dai7mo74

No, because power/influence dynamics could be very different in CEV compared to the current world and it seems reasonable to distrust CEV in principle or in practice, and/or CEV is sensitive to initial conditions implying a lot of leverage to influencing opinions before it starts.

[-]Satya Benson7mo125

why is your definition of value so arbitrary as to stipulate that biological meat-humans are necessary

In this kind of conversation, it's important that different people will inevitably want fundamentally different things in a way that cannot be fully reconciled.

Most people recognize that humans in their current form will not exist forever, but our preferences about what comes next are really varied. Many people want to minimize change and want the future to be something which they recognize and understand. Many others see this kind of future as both unrealistic and undesirable, and focus on caring about the manner in which we are replaced, or preserving certain qualities of humans in our successors.

Unfortunately, it's common to assume that there is some correct thing to value, or that a certain class of values (e.g. evolutionary preferences) are well justified or widely accepted.

This isn't a productive framework.

Perhaps I am overly pessimistic, but I see preferences as so varied on this question that I'd guess there is no possible outcome that is desirable to more than half the population.

[-]cubefox7mo61

It seems highly likely that the majority of humans prefer humanity not to be replaced by AI in the foreseeable future. So I don't think there is that much variance here.

[-]Vladimir_Nesov7mo53

Many different things can happen to many different people at the same time, in the same world. For most disagreements about what happens to people personally, or to like-minded communities of people, to a first approximation there is no conflict.

[-]Satya Benson7mo21

The question of 'what should replace humanity and how should this occur' isn't something we can answer with a patchwork of different microworlds.

[-]Vladimir_Nesov7mo20

That different things can happen to different people illustrates the point that even significant variations in preference don't directly lead to conflict about desirable outcomes, certainly not negative outcomes for half the population. I'm objecting specifically to the argument that variations in preference result in bad outcomes for a significant fraction of people.

The more settled form of the future is of course very different, but the least we should do is avoid catastrophic mistakes on the very first steps there.

[-]ThomasCederborg7mo10

If I understand your comment correctly, then I think that the claim about preference variation made in the second sentence is wrong. Specifically, I think that a significant fraction of people do have strong preferences, such that fulfilling those preferences would be very bad for a significant fraction of people. Some people would for example strongly prefer that earths ecosystems be destroyed (as a way of preventing animal suffering). Others would strongly prefer to protect earths ecosystems. The partitioning idea that you mention would be irrelevant to this value difference. There also exists no level of resources that would make this disagreement go away. (This seems to me to be in contradiction with the claim that is introduced in the second sentence of your comment).

I don't think that this is an unusual type of value difference. I think that there exists a wide range of value differences such that (i): fulfilling a strong preference of a significant fraction of people leads to a very bad outcome for another significant fraction of people, (ii): partitioning is irrelevant, and (iii): no amount of resources would help.

Let's for example examine the primary value difference discussed in the Archipelago text that you link to in your earlier comment. Specifically the issue of what religion future generations will grow up in. And what religious taboos future generations will be allowed to violate. Consider a very strict religious community that view apostasy as a really bad thing. They want to have children, and they reject non-natural ways of creating or raising children. One possibility is an outcome where children are indoctrinated to ensure that essentially all children born in this community will be forced to adapt to their very strict rules (which is what would happen if the communities of a partitioned world have autonomy). This would be very bad for everyone who cares about the well-being of those children that would suffer under such an arrangement. To prevent such harm, one would have to interfere in a way that would be very bad for most members of this community (for example by banning them from having children. Or by forcing most members of this community to live with the constant fear of loosing a child to apostasy (in the Archipelago world described in your link, this would for example happen because of the mandatory UniGov lectures that all children must attend)).

We can also consider a currently discussed, near-term, and pragmatic, ethics question: what types of experiences are acceptable for what types of artificial minds? I see no reason to think that people will agree on which features of an artificial mind, implies that this mind must be treated well (or agree on what it means for different types of artificial minds to be treated well). This is partly an empirical issue, so some disagreements might be dissolved by better understanding. But it also has a value component. And there will presumably be a wide range of positions on the value component. Partitioning is again completely irrelevant to these value differences. (And additional resources actually makes the problem worse, not better).

Extrapolated humans will presumably have value differences along these lines about a wide range of issues that have not yet been considered.

There thus seems to exist many cases where variations in preference will in fact result in bad outcomes for a significant fraction of people (in the sense that many cases exists, where fulfilling a strong preference held by a significant fraction, would be very bad for a significant fraction of people).

More straightforwardly, but less exactly, one could summarise my main point as: for many people, many types of bad outcomes cannot be compensated for by giving someone the high-tech equivalent of a big pile of gold. There is no amount of gold that will make Bob ok with the destruction of earths ecosystems. And there is no amount of gold that will make Bill ok with animals suffering indefinitely in protected ecosystems. And the way things are partitioned would rarely be relevant for these types of conflicts. And if resources do happen to matter in some way, then additional resources would usually make things worse. Even more straightforwardly: you cannot compensate for one part of the universe being filled with factory farms by filling a different part of the universe with vegans.

In conclusion, it seems to me that there exists a wide range of strong preferences (each held by a significant fraction of people), such that fulfilling this preference would be very bad for a significant fraction of people. It seems to me that this falsifies the additional claim that you introduce in the second sentence of your comment.

(Some preference differences can indeed be solved with partitioning. So it is indeed possible for a given set of minds to have preference differences without having any conflicts. Which indeed means that the existence of preference differences does not, on its own, conclusively prove the existence of conflict. I am however only discussing the additional claim that you introduce in the second sentence of this comment. In other words: I am narrowly focusing on the novel claim that you seem to be introducing in the second sentence. I am just picking out this specific claim. And pointing out that it is false)

Technical note on scope:

My present comment is strictly limited to exploring the question of how one fraction of the population would feel, if a preference held by another fraction of the population were to be satisfied. I stay within these bounds, partly because any statement regarding what would happen, if an AI were to implement The Outcome that Humanity would like an AI to implement, would be a nonsense statement (because any such statement would implicitly assume that there is some form of free floating Group entity, out there, that has an existence that is separate from any specific mapping or set of definitions (see for example this post and this comment for more on this type of "independently-existing-G-entity" confusion)).

A tangent:

I think that from the perspective of most people, humans actively choosing an outcome would be worse than this outcome happening by accident. Humans deciding to destroy ecosystems will usually be seen as worse than ecosystems being destroyed by accident (similarly to how most people would see a world with lots of murders as worse than a world with lots of unpreventable accidents). The same goes for deciding to actively protect ecosystems despite this leading to animal suffering (similarly to how most people would see a world with lots of torture as worse than a world with lots of suffering from unpreventable diseases).

[-]Seth Herd7mo120

Instead, my opposition to AI successionism comes from a preference toward my own kind. This is hardwired in me from biology. I prefer my family members to randomly-sampled people with similar traits

I have a biologically hardwired preference for defeating and hurting those who oppose me vigorously. I work very hard to sideline that biologically hardwired preference.

To be human is to be more than human.

You and all of us struggle against some of our hardwired impulses while embracing others.

Separately, the wiser among successionist advocates may be imagining a successor for whom they'd feel the same sort of love you feel for your grandchildren. Humans are not limited to loving their biological kin, although we are certainly biased toward loving them more (do ask someone from a really bad family how strong their familial instincts are).

Before I'd even consider accepting some sort of successor, I'd want to meet them and know they spark joy in my heart like seeing a young human playing and learning does. I find it entirely possible that we could create beings who would evoke more love from humans than humans do, becasue they are genuinely more worthy of it.

[-]Davidmanheim7mo92

I have a biologically hardwired preference for defeating and hurting those who oppose me vigorously. I work very hard to sideline that biologically hardwired preference.

This seems like a very bad analogy, which is misleading in this context. We can usefully distinguish between evolutionarily beneficial instrumental strategies which are no longer adaptive and actively sabotage our other preferences in the modern environment, and preferences that we can preserve without sacrificing other goals.

[-]Seth Herd7mo5-2

This seems at best like an overstatement of how bad that analogy is. At worst it's pointing to an important aspect of the logic here: having an innate drive doesn't mean consciously endorsing it. There's a second step.

I think the analogy is actually directly on.

My preference for humans over other types of sapient/sentient beings does conflict with my other goals: furthering happiness/joy in all its embodiments, and furthering diversity of thought and experience. If I want a world full of humans and make no room for sentient AIs, those other goals may well be sacrificed for my preference for humans.

I feel I should clarify once again that I am not a successorist; right now I'd prefer a world with both lots of humans and posthumans, and lots of types of sentient AIs (and non-sentient AIs to do work that nobody sentient wants to do).

But I'm highly uncertain, and merely trying to contribute to the logic of this question.

I haven't thought about this a ton because I consider it far more pressing to figure out alignment so we can have some measure of control over the future. Non-sentient AI taking over the lightcone is something almost no one wants on reflection, including, I think, most "successorists" who are motivated more by trolling than considered sincere beliefs. Where I've followed their logic (eg. Beff Jezos) they have actually indicated that they're expecting sentient AI and would be unhappy with a future without sentience in some form.

[-]Mikhail Samin7mo119

a large fraction of the AI safety and alignment community endorses humanity being replaced by a sufficiently aligned and advanced AI, and would prefer this to a future where our actual descendants spread over the planets, albeit at a slower pace and with fewer total objective "utils"

I don’t think this is true.

To the extent we, on reflection, have preferences about discovering, solving, colonizing the universe ourselves—solving physics, building technology, etc.—a CEV-aligned superintelligence just refuses to help us with those things! It will stop deaths immediately, and it will probably send probes to distant galaxies immediately (on average, our future lightcone loses approximately one galaxy every year), but then it will probably leave us reflect and do stuff, because this is what’s fun.

“Utils” are not some objective thing. It is what we want. To the extent we do want to do meaningful, cool, and interesting stuff, this is what “utils” end up being mainly about.

A sufficiently aligned and advanced AI likely prevents loss of huge amounts of value and then approximately turns off.

[-]cousin_it7mo42

Yeah, I had similar thoughts. And it's even funnier, the AI will not just refuse to solve these problems, but also stop us from creating other AIs to solve these problems.

[-]plex7mo54

You seem to be in a different bubble to me in remarkably many ways, given the overlap of topics.

[-]Anon User7mo*31

I think this is also a burden of proof issue. Somebody who argues I ought to sacrifice my/my children's future for the benefit of some extremely abstract "greater good" has IMHO an overwhelming burden of proof that they are not making a mistake in their reasoning. And frankly I do not think the current utilitarian frameworks are precise enough / universally accepted enough to be capable of truly meeting that burden of proof in any real sense.

[-]cubefox7mo31

I agree with this. Just one quibble here:

Utilitarianism implies that if we build an AI that successfully maximizes utility/value, we should be ok with it replacing us.

That is true for some forms of utilitarianism (total or average utilitarianism) but not necessarily for some others (person affecting preference utilitarianism). For the reason you outline, I think the latter kind of utilitarianism is correct. Its notion of utility/value incorporates what currently existing people want, and I believe that most humans want that humanity will not be replaced by AI in the future.

One way to think about it: AI succesionism is bad for a similar reason we regard being dead as bad: We don't want it to happen. We don't want to die, and we also don't want humanity to become extinct through sterilization or other means. Perhaps we have these preferences for biological reasons, but it doesn't matter why: final goals need no justification, they just exist and have moral weight in virtue of existing. The fact that we wouldn't be around if it (being dead, or humanity being extinct) happens, and that we therefore would not suffer from it, doesn't invalidate the current weight of our current preferences.

[-]RogerDearnaley7mo20

As someone who often passionately rants against the AI successionist line of thinking, the most common objection I hear is "why is your definition of value so arbitrary as to stipulate that biological meat-humans are necessary." This is missing the crux—I agree such a definition of moral value would be hard to justify.

I don't agree. I follow the philosophical school of evolutionary ethics, which derives ethical value from evolutionary theory about the behavior of social organisms (which will tend to evolve things like a sense of fairness). This gives a clear, non-religious solution to the ought-from-is problem otherwise inherent to moral realism: suffering is bad because it's a close proxy for things that decreases the well-being and evolutionary fitness of a living being that is subject to evolutionary forces. And in my view as a human, it's particularly bad when it applies to other humans — that's the way my moral sense was evolved, to apply to other members of the same coalition-of-tribes (rationally I'm willing to expand this to at least the entire human species — if we ever meet an alien sapient species I'd need to assess whether they're utility monsters).

As such, it's pretty easy to derive from evolutionary ethics a corollary that if something is non-living and not evolved, so not subject to evolutionary fitness, then it is not a moral patient (unless, for some reason, it would benefit humans for us to treat it as one, as we do for pets). In that moral framework, no current AI is a moral patient, and the utilitarian value of a world containing only AI and no living beings is zero.

[-]RogerDearnaley7mo20

Note that under this moral framework the moral status of an uploaded human is a much more complicated question: they're a product of evolution, even though they're no longer subject to it.

[-]wonder7mo*20

Note: this is a completely sidebar (and apologize in advance).

I prefer my family members to randomly-sampled people with similar traits. I would certainly not elect to sterilize or kill my family members so that they could be replaced with smarter, kinder, happier people.

Out of curiosity, in hypothetical scenarios:

Would you replace randomly-sampled people with smarter, kinder, happier people? (To clarify I hope the answer is no)
Would you say you would (or humans) prefer your family members over other randomly-sampled people, in a resource scarce situation and when you have the power to choose who to survive?

Context: I have no issue with this statement. I am asking because I have also been thinking about this human nature, and trying to find the pros and cons of it.

[-]Mo Putera7mo20

Importantly, I value every intermediate organism in this chain

An interesting and personally relevant variant of this is if the approval only goes one direction in time. This happened to me: 2025!Mo is vastly different from 2010!Mo in large part due to step-changes in my "coming of age" story that would've left 2010!Mo horrified (indeed he tried to fight the step-changes for months) but that 2025!Mo retrospectively fully endorses post-reflective equilibrium.

So when I read something like Anders Sandberg's description here

There is a kind of standard argument you sometimes hear if you’re a transhumanist — like I am — that talks about life extension, where somebody cleverly points out that you would change across your lifetime. If it’s long enough, you will change into a different person. So actually you don’t get an indefinitely extended life; you just get a very long life thread. I think this is actually an interesting objection, but I’m fine with turning into a different future person. Anders Prime might have developed from Anders in an appropriate way — we all endorse every step along the way — and the fact that Anders Prime now is a very different person is fine. And then Anders Prime turns into Anders Biss and so on — a long sequence along a long thread.

I think: it's not all that likely that I'm done with the whole "coming of age" reflective equilibrium thing, so I find it very likely that there are more step-changes I'll experience that 2025!Mo would find horrifying but Future!Mo would fully endorse, contra Anders' "we all endorse every step along the way". It's not just the outcomes that Past!Mos disendorse: reflection changes what changes are endorsed too.

This is the sort of retrospection that makes me sympathetic to what Scott said in his review of Hanson's Age of Em:

A short digression: there’s a certain strain of thought I find infuriating, which is “My traditionalist ancestors would have disapproved of the changes typical of my era, like racial equality, more open sexuality, and secularism. But I am smarter than them, and so totally okay with how the future will likely have values even more progressive and shocking than my own. Therefore I pre-approve of any value changes that might happen in the future as definitely good and better than our stupid hidebound present.”
I once read a science-fiction story that depicted a pretty average sci-fi future – mighty starships, weird aliens, confederations of planets, post-scarcity economy – with the sole unusual feature that rape was considered totally legal, and opposition to such as bigoted and ignorant as opposition to homosexuality is today. Everybody got really angry at the author and said it was offensive for him to even speculate about that. Well, that’s the method by which our cheerful acceptance of any possible future values is maintained: restricting the set of “any possible future values” to “values slightly more progressive than ours” and then angrily shouting down anyone who discusses future values that actually sound bad. But of course the whole question of how worried to be about future value drift only makes sense in the context of future values that genuinely violate our current values. Approving of all future values except ones that would be offensive to even speculate about is the same faux-open-mindedness as tolerating anything except the outgroup.
Hanson deserves credit for positing a future whose values are likely to upset even the sort of people who say they don’t get upset over future value drift. I’m not sure whether or not he deserves credit for not being upset by it. Yes, it’s got low-crime, ample food for everybody, and full employment. But so does Brave New World. The whole point of dystopian fiction is pointing out that we have complicated values beyond material security. Hanson is absolutely right that our traditionalist ancestors would view our own era with as much horror as some of us would view an em era. He’s even right that on utilitarian grounds, it’s hard to argue with an em era where everyone is really happy working eighteen hours a day for their entire lives because we selected for people who feel that way. But at some point, can we make the Lovecraftian argument of “I know my values are provincial and arbitrary, but they’re my provincial arbitrary values and I will make any sacrifice of blood or tears necessary to defend them, even unto the gates of Hell?”

Except that I'm pretty sure Future!Mos won't be defending Past!Mos' "provincial and arbitrary values", the way 2025!Mo doesn't defend and in fact flatly rejects a lot 2010!Mo's core values. I'm not sure how to think of all this.

[-]Stephen Martin7mo20

if me and my family were uploaded to a computer and our existences and evolution simulated at enormous speed, I should be ok with our descendants coming out of the simulation and repopulating the world

What if you and your family are uploaded to a computer and just never come out because it's way better in there?

Technically this would fall under the umbrella of:

the path of human extinction and replacement with "superior" beings

From my end I would consider that meaningfully different from an outcome where humanity is just killed off and never gets to upload. Would you also be against this outcome if done en masse by humanity, a voluntary cedeing of "meatspace" to digital intelligence "property managers"?

[-]cubefox7mo2-1

Instead, my opposition to AI successionism comes from a preference toward my own kind. This is hardwired in me from biology. I prefer my family members to randomly-sampled people with similar traits. I would certainly not elect to sterilize or kill my family members so that they could be replaced with smarter, kinder, happier people.

I would go further. Most people probably also prefer humans existing in the future to AIs existing in the future. It's not just about direct personal descendants, but about a preference for our own species continuing to exist.

Is that preference guilty of "speciesism" and therefore immoral? No, it's just a final preference about what should happen in the future. It's not immoral, since possible future AIs do not exist yet and as such have no preferences (like coming into existence) which could be violated. So it isn't bad when we don't make them exist. This is called the procreation asymmetry.

[-]Seth Herd7mo20

I don't think this is properly considered a final preference. It's just a preference. We could change it, slowly or quickly, if we for some reason decided AIs were more lovable/valuable than humans (see my other comment if you want).

You could use the same argument about existing beings against longtermism, but I just don't think it carves reality at its joints. Your responsibility toward possible beings is no different than your responsibility toward existant beings. You could make things better or worse for either of them and they'd love or hate that if you did.

[-]cubefox7mo31

Well, like any preference it could change over time, but "final" is the opposite of "instrumental", not of "variable".

You could use the same argument about existing beings against longtermism, but I just don't think it carves reality at its joints. Your responsibility toward possible beings is no different than your responsibility toward existant beings. You could make things better or worse for either of them and they'd love or hate that if you did.

The important point is: If we don't let them come into existence then they would not hate that. Merely possible beings don't exist and therefore have no preference for coming to be. So it isn't bad if they never get to exist. See here for details.

[-]Seth Herd7mo31

Got it, I was just confused on terminology on the first point.

On not valuing beings who have yet to exist, I'm familiar with the standard arguments and they just don't make sense to me. I have a huge preference for existing, and if we took actions to allow similar beings to exist, they would really appreciate it. You're right that they'll never hate it if we don't bring them into existence but that only takes care of half of the argument. Unless you've got something unique in that link this has always struck me as a very convenient position but not one that hangs together logically.

[-]cubefox7mo-11

You can think of what is good for someone as preferences. And you can think of preferences as being about potentially switching from some state X (where you currently are) to Y. There can be no preferences about switching from X to Y if you don't exist in X, since there is nobody who would have such a preference. Of course you can have preferences once you exist in Y, e.g. about not switching back to X (not dying). But that only means that switching from Y to X is bad, not that switching from X to Y was good.

[-]Seth Herd7mo20

You are casting preference to only extend into the future. I guess that is the largest usage of "preference.". But people also frequently say things like "I wish that hadn't happened to me" so it's also frequently used about the past.

It seems like this isn't preference utilitarianism. It does fit negative utilitarianism, which I'm even more sure isn't right or at least intuitively appealing to the vast majority of considered opinions.

Utilitarianism basically means (to me) that since I like happiness for myself, I also like it for other beings who feel similar happiness. Whether or not that happiness exists in the future and whether or not my actions created them to be happy or just increased their happiness, I do them because I like happiness.

(or utils or whatever your precise definition is).

You can decide your utilitarianism doesn't extend into creating new beings to be if you like but then I don't even know why you're utilitarian. It's the same sort of stuff and I personally will either like it or not in all circumstances.

I'll check out your link to see if there are arguments I haven't thought about.

[-]cubefox7mo*31

You are casting preference to only extend into the future. I guess that is the largest usage of "preference.". But people also frequently say things like "I wish that hadn't happened to me" so it's also frequently used about the past.

I thought about this, and I agree "regret" wishes like this are clearly possible. We can think of wishes (desires, "wants") as preferences, namely as preferring something we don't have to something we do have. Or more precisely and generally, preferring being certain that something is true to the actual current case of being less than certain that it is true. This means someone can wish that they hadn't been born. Which means they prefer not having been born to the actual case of having been born. More technically, this would mean preferring a certainty (100% probability) of not having been born to the current 0% belief of not having been born.

However, this means we cannot wish for anything we already have. Because that would be preferring having something to having it. We cannot prefer being certain that X is true to our current degree of belief of X being true, if that current degree of belief is such that we are already certain that X is true. In short, we can't prefer being certain that X to our current state of being certain that X.^[1]

Which means I can wish that I had never been born (regret being born), but I cannot wish that I have been born. So it can actually be bad for someone to come into existence, namely when they, after coming into existence, regret having been born (e.g. because of very bad life conditions). But it cannot similarly be good for them to have come into existence, as there is no possible preference of this sort. If they have come to exist, they can't prefer that to having come to exist (as elaborated above), and if they don't come to exist, they can't regret that they didn't come to exist, because someone who doesn't exist can't form any preference, including preferring to exist over the actual current state of not existing.

In fact, this point has been argued before in the context of decision theory. In chapter 4.4 of Richard Jeffrey's book The Logic of Decision he writes (emphasis in the original):

Socrates argues (Symposium, 200) that Love is not a god because to desire something is to be in want of it: you cannot desire what you already have. The point is also made in contemporary dictionaries. Thus, The Concise Oxford Dictionary defines a desire as an unsatisfied appetite, longing, wish, or craving. This account of the matter seems roughly right, although it needs some refinement. Thus, to vary the example of the Symposium, since people do not always know when they are loved, it is entirely possible to desire someone's love when you already have it. Therefore, it seems better to say that you cannot desire what you think you have. Taking propositions as the objects of desire, the doctrine becomes: one who believes that a proposition is true cannot desire that it be true.

I would clarify that strictly speaking you can believe something to be true but still wish it to be true, as long as your belief (as is normally the case) falls short of certainty. Thus, if there is a thunderstorm, you may be 99% sure you won't be struck by lighting, but you can still desire not to be struck by lightning, insofar you (more or less strongly) prefer a 100% certainty not be be struck to your current 99% belief that you will not be struck. Another way of looking at this: if being struck by lightning is "extremely bad", say , then the expected utility of it for a 1% chance/belief is $P (I'm struck by lightning) \times U (I'm struck by lightning) = 1 % \times 1, 000, 000 = - 10, 000$ . Which may still be "very bad".

Of course we can e.g. still wish for (being certain of) things happening in the future, insofar we are not already certain they will happen. ↩︎

[-]Vladimir_Nesov7mo1-2

possible beings don't exist and therefore have no preference for coming to be

That doesn't follow, if you describe that which doesn't exist in sufficient detail, it becomes possible to answer any question about it. They may have a preference, it's just more easily ignored and less likely to have influence with this world.

[-]cubefox7mo30

But from a consistent description doesn't follow concrete existence. Otherwise an ontological argument could imply the existence of God. And if someone doesn't exist, their parts don't exist either. A preference is a part of someone, like an arm or a leg, and they wouldn't exist. A description could only answer what preference someone would have if they existed. But that's only a hypothetical preference, not one that exists.

[-]Vladimir_Nesov7mo30

We can estimate that an arm or a leg still have such and such weight, even for a hypothetical person who doesn't exist, and plan how to build bridges that won't collapse if that person who doesn't exist were to come into existence and walk on them.

The question is preference, a thing that tells you whether X is better than Y, which is abstract data that doesn't depend on a bearer of it existing in some place or other. If you can point to preference, you can answer such questions, including about existence of instances of some bearer of that preference.

But that's only a hypothetical preference, not one that exists.

Certainly, but it answers the same questions in the same way, so why should that matter at all? We are asking what this preference says about its bearer existing in some world, so we can ask the hypothetical preference and get the same answer, just as good as the real one. An abstract program computes the same results as one actually running on a physical computer, it answers all the same questions in the same way.

[-]cubefox7mo20

In a preference utilitarian calculus it does matter which possible preferences actually exist and which don't. We don't count hypothetical votes in an election either.

[-]Vladimir_Nesov7mo30

Whether you count the hypothetical votes or not, doesn't address the question of what those votes say. I'm objecting to the claim that the votes are saying nothing, not necessarily to the claim that they are to be ignored.

[-]Htarlov5mo10

I think there are multiple moral worldviews that are rational and based on some values. Likely the whole continuum.

The thing is that we have values that are in conflict in edge cases, and those conflicts need to be taken into account and resolved when building a worldview as a whole. You can resolve them in many ways. Some might be simple like "always prefer X", some might be more complex like "in such and such circumstances or precoditions prefer X over Y, in some other preconditions prefer Z over Y, in some other ...". It might be threshold-based when you try to measure the levels of things and weight them mathematically or quasi-mathematically.

At the most basic level, it is about how you weigh the values in relation to each other (which is often hard, as we often do not have good measures), and also how important for you it is to you to be right and exact vs being more efficient, quick, being able to spare more of your mental energy or capacity or time for other things than devising exact worldview.

If your values are not simple (which is often the case for humans) and often collide with each other, complex worldviews have the advantage of being closer to applying your values in different situations in a way that is consistent. On the other hand, simple worldviews have the advantage of being easy and fast to follow, and are technically internally consistent, even if not always feeling right. You don't need as much thinking beforehand, and on the spot when you need to decide.

Now, you can reasonably prefer some rational middle ground. A worldview that isn't as simple as basic utilitarianism or ethical egoism or others, but is also not as complex as thinking out each possible moral dilemma and possible decision to work out how to weigh and apply own values in each of them.
It might be threshold-based or/and patchwork-based, and in such values can be built in a way that different ones have different weights in different subspaces of the whole space of moral situations. You may actually want to zero out some values in some subspaces to simplify and not take in components, that are already too small or would incentivize focus on unimportant progress.

In practical terms to show an example - you may be utilitarian in broad area of circumstances, but in any circumstances when it would make you have relatively high effort for a very small change in lowering total suffering or heightening total happiness, then you might zero out that factor and fall back to choosing in accordance of what is better for yourself (ethical egoism).

BTW I believe it is also a way to devise value systems for AI - by having them purposely only take into account values when the change in the total value function between decissions taken from that value are not too small. If it is very small, it should not care, it should not take it into account about that minuscule change. On the meta-level, it is also based on another value - valuing own time and energy to have a sensible impact.

Yes, I know this comment is a bit off-topic from the article. What is important for the topic - there are people, me included, who have consequentialist quasi-utilitarian beliefs, but won't see why we would like to have strict value-maximising (even if that value is total happiness) or replace them with entities that are such maximizers.

Also, I don't value complexity reduction, so I don't value systems that maximize happiness and reduce the world to simpler forms, where situations when other values matter simply don't happen. On the contrary, I prefer preserving complexity and the ability for the world to be interesting.

[-]Sean Pan7mo10

I feel that it may not be in LessWrong standards to simply reply with a broad sense of deep, empathetic agreement so I will instead quote what I feel is a truly beautiful reflection from Erik Hoel on this, borrowing from his "How to prevent the coming inhuman future":

"Personally, I’m on the side of Shakespeare. For it is not such a bad future, in the hands of humans. Humans after all, invented space travel and ended slavery and came up with antibiotics and snow plows and baby formula. We patch holes in clothing. We sing along to the radio in cars. We’re cute and complicated and fucked-up and kind. We kill, yes, but rarely, and we love far more. We’re not perfect. But we’re the best it’s going to get. Anything else will be unrecognizable and immoral, except by its own incommensurate and alien standards. So give me humans, fallible humans, poetic humans, funny humans, free humans, humans with their animal-like cunning, humans with their ten fingers and ten toes, human babies looking out the portholes of space stations just as they looked out over the hills of Pompeii, humans with their brains unfettered, humans colonizing the galaxy as individuals and nations and religions and collectives and communes and families, humans forever.

Humans forever! Humans forever! Humans forever!"

https://www.theintrinsicperspective.com/p/how-to-prevent-the-coming-inhuman

[+]Logan Zoellner6mo-8-1

Moderation Log

LESSWRONG
LW

LESSWRONG
LW

68

Why I am not a successionist

68

68

Technical note on scope:

A tangent: