A tentative dialogue with a Friendly-boxed-super-AGI on brain uploads

You must be using the words "friendly" and "aligned" in a sense I am not familiar with.

Possibly. I said this AGI is “safer and more aligned”, implying that it is a matter of degree – while I think most people would regard these properties as discrete: either you are aligned or unaligned. But then I can just replace it with “more likely to be regarded as safe, friendly and aligned”, and the argument remains the same. Moreover, my standard of comparison was Celest-IA, who convinces people to do brain uploading by creating a “race to the bottom” scenario (i.e., as more and more people go to the simulation, human extinction becomes more likely – until there’s nobody to object turning the Solar System into computronium), and adapts their simulated minds so they enjoy being ponnies; my AGI is way "weaker" than that.

I still think it’s not inappropriate to call my AGI “Friendly”, since its goals are defined by a consistent social welfare function; and it’s at least tempting to call it “safe”, as it is law-abiding and does obey explicit commands. Plus, it is strictly maximizing the utility of the agents it is interacting with according to their own utility function, inferred from their brain simulation - i.e., it doesn’t even require general interpersonal comparison of utility. I admit I did add a tip of a perverse sense of humor (e.g., the story of the neighbors), but that's pretty much irrelevant for the overall argument.

But I guess arguing over semantics is beyond the point, right? I was targeting people who think one can “solve alignment” without “solving value”. Thus, I concede that, after reading the story, you and I can agree that the AGI is not aligned - and so could B, in hindsight; but it’s not clear to me how this AGI could have been aligned in the first place. I believe the interesting discussion to have here is why it ends up displaying unaligned behaviour.

I suspect the problem is that B has (temporally and modally) inconsistent preferences, such that, after the brain upload, the AI can consistently disregard the desire of original-B-in-the-present (even though it still obeys original-B’s explicit commands), because they conflict with simulated-B’s preferences (which weigh more, since sim-B can produce more utility with less resources) and past-B’s preferences (who freely opted for brain upload). As I mentioned above, one way to deflect my critique is to bite the bullet: like a friend of mine replied, one can just say that they would not want to survive in the real world after a brain upload – they can consistently say that it’d be a waste of resources. Another way to avoid the specific scenario in my story would be by avoiding brain simulation, or by not regarding simulations as equivalent to oneself, or, finally, by somehow becoming robust against evidential blackmail.

I don’t think that is an original point, and I now see I was sort of inspired by things I read from debates on coherent extrapolated volition long ago. But I think people still underestimate the idea that value is a hard problem: no one has a complete and consistent system of preferences and beliefs (except the Stoic Sages, who “are more rare than the Phoenix”), and it’s hard to see how we could extrapolate from the way we usually cope with that (e.g., through social norms and satisficing behavior) to AI alignment - as superintelligences can do way worse than Dutch books.

[-]Garrett Baker3y2-1

The reason why I saw Friendship Is Optimal as a utopia was because it seemed like lots of value in the world was preserved, and lots of people seemed satisfied with the result. Like, if I could choose that world, or this world as it currently is, I would choose that world. Similarly with the world you describe.

This is different from saying it's the best possible world. It's just, like, a world which makes me compromise on comparatively few values I hold dear compared to the expected outcome of this world.

This may come down to differing definitions of utopia/dystopia. So I'd recommend against using those words in future replies.

[-]Ramiro P.3y10

Thanks, I believe you are right. I really regret how much time and resources are wasted arguing over the extension / reference of a word.

I'd like to remark though that I was just trying to explain what I see as problematic in FiO. I wouldn't only say that its conclusion is suboptimal (and I believe it is bad, and many people would agree); I also think that, given what Celestia can do, they got lucky (though lucky is not the adequate word when it comes to narratives) it didn't end up in worse ways.

As I point out in a reply to shminux, I think it's hard to see how an AI can maximize B's preferences in an aligned way if B's preferences and beliefs are inconsistent (temporally or modally). If B actually regards sim-B as another self, then its sacrifice is required; I believe that people who bite this bullet will tend to agree that FiO ends in a good way, even though they dislike "the process".

[-]bokov3y10

What I like about this story is that it makes more accessible the (to me) obvious fact that, in the absence of technology to synchronize/reintegrate memories from parallel instances, uploading does not solve any problems for you-- it at best spawns a new instance of you that doesn't have those problems, but you still do.

Yet uploading is so much easier than fixing death/illness/scarcity in the physical world that people want to believe it's the holy grail. And may resist evidence to the contrary.

Destructive uploads are murder and/or suicide.

[-]Algon3y50

Wait, why are destructive uploads murder/suicide? A copy of you ceases to exist and then another copy comes into existence with the exact same sense of memories/continuity of self etc. That's like going to sleep and waking up. Non-destructive uploads are plausibly like murder/suicide, but you don't need to do down that route.

[-]bokov3y10

A copy of you ceases to exist and then another copy comes into existence with the exact same sense of memories/continuity of self etc. That's like going to sleep and waking up.

Even when it becomes possible to do this at sufficient resolution, I see no reason it won't be like going to sleep and never waking up.

It's not as if there is a soul to transfer or share between the two instances. No way to sync the experiences of the two instances.

So I don't see a fundamental difference between "You go to sleep and an uploaded you wakes up" vs "You go to sleep and an uploaded somebody else wakes up". In either case it will be a life in which I am not a participant and experiences I will not be able to access.

Non-destructive uploads could be benign, provided they are not used as an excuse for not improving the lives of the original instances.

[-]Shamash3y21

Consider the following thought experiment: You discover that you've just been placed into a simulation, and that every night at midnight you are copied and deleted instantaneously, and in the next instant your copy is created where the original once was. Existentially terrified, you go on an alcohol and sugary treat binge, not caring about the next day. After all, it's your copy who has to suffer the consequences, right? Eventually you fall asleep.

The next day you wake up hungover as all hell. After a few hours of recuperation, you consider what has happened. This feels just like waking up hungover before you were put into the simulation. You confirm that the copy and deletion did occur. It is confirmed. Are you still the same person you were before?

You're right that it's like going to sleep and never waking up, but Algon was also right about it being like going to sleep and waking up in the morning, because from the perspective of "original" you those are both the same experience.

[-]green_leaf3y10

Your instance is the pattern, and the pattern is moved to the computer.

Since consciousness is numerically identical to the pattern (or, more precisely, the pattern being processed), the question of how to get my consciousness in the computer after the pattern is already there doesn't make sense. The consciousness is already there, because the consciousness is the pattern, and the pattern is already there.

[-]bokov3y10

Now, if synchronizing minds is possible, it would address this problem.

But I don't see nearly as much attention being put into that as into uploading. Why?

[-]Noosphere893y11

I note a distributional shift issue, in that the concept of a single, continuous you only exists due to limitations of biology, and once digital uploads can happen, the concept of personality can get very weird indeed. The real question can be, does it matter then? Well, that's a question that won't be solved by philosophers.

So the real thing that is a lesson is be wary of distributional shift mucking up your consciousness.

I'm also biting the bullet and saying that this is probably what we should aim for, barring pivotal acts because I see AGI development as mostly inevitable, and there are far worse outcomes than this.

[-]bokov3y0-1

I'm also biting the bullet and saying that this is probably what we should aim for, barring pivotal acts because I see AGI development as mostly inevitable, and there are far worse outcomes than this.

Dead is dead, whether due to AGI or due to a sufficient percentage of smart people convincing themselves that destructive uploading is good enough and continuity is a philosophical question that doesn't matter.

LESSWRONG
Petrov Day
LW

LESSWRONG
Petrov Day
LW

1

A tentative dialogue with a Friendly-boxed-super-AGI on brain uploads

1

1