Impossible moral problems and moral authority

[-]cousin_it6yΩ150

It seems to me that with meals, there's a fact of the matter that AI could help with. After all, if two copies of you went and had the different meals, one of them would probably be happier than the other.

Though that happiness might not depend only on the chosen meal. For example, if one meal is a cake that looks exactly like Justin Bieber, that might be actually not as fun as it sounds. But if you skipped it in favor of an ordinary burrito, you'd forever regret that you didn't get to see the Justin Bieber cake.

[-]Dagon6y*70

I think you still hit the multidimension comparison problem. One of the branches may be happier, the other more satisfied. One might spend the rest of the evening doing some valuable work, the other didn't have as much time. One consumed a slightly better mix of protein and fat, the other got the right amount of carbs and micronutrients.

Without a utility function to tell you the coefficients, there _IS_ no truth of the matter which is better.

Edit: this problem also hits intertemporally, even if you solve the comparability problem for multiple dimensions of utility. The one that's best when eating may be different than the one that you remember most pleasurably afterward, which may be different than the one you remember most pleasantly a year from now.

[-]Gordon Seidoh Worley6yΩ250

I think all of your reasons for how a human comes to have moral authority boil down to something like having a belief that doing things that this authority says are expected to be good (have positive valence, in my current working theory of values). This perhaps gives a way of reframing alignment as the problem of constructing an agent to whom you would give moral authority to decide for you, rather than as we normally do as an agent that is value aligned.

[-]Mass_Driver4y40

I'm curious about the source of your intuition that we are obligated to make an optimal selection. You mention that the utility difference between two plausibly best meals could be large, which is true, especially when we drop the metaphor and reflect on the utility difference between two plausibly best FAI value schemes. And I suppose that, taken literally, the utilitarian code urges us to maximize utility, so leaving any utility on the table would technically violate utilitarianism.

On a practical level, though, I'm usually not in the habit of nitpicking people who do things for me that are sublimely wonderful yet still marginally short of perfect, and I try not to criticize people who made a decision that was plausibly the best available decision simply because some other decision was also plausibly the best available decision. If neither of us can tell for sure which of two options is the best, and our uncertainty isn't of the kind that seems likely to be resolvable by further research, then my intuition is that the morally correct thing to do is just pick one and enjoy it, especially if there are other worse options that might fall upon us by default if we dither for too long.

I agree with you that a trusted moral authority figure can make it easier for us to pick one of several plausibly best options...but I disagree with you that such a figure is morally necessary; instead, I see them as useful moral support for an action that can be difficult due to a lack of willpower or self-confidence. Ideally, I would just always pick a plausibly best decision by myself; since that's hard and I am a human being who sometimes experiences angst, it's nice when my friends and my mom help me make hard decisions. So the role of the moral authority, in my view, isn't that they justify a hard decision, causing it to become correct where it was not correct prior to their blessing; it's that the moral authority eases the psychological difficulty of making a decision that was hard to accept but that was nevertheless correct even without the authority's blessing.

[-]Charlie Steiner4y20

Yes, I agree with everything you said... until the last sentence ;)

In-parable, your mom is mostly just there as moral support. Neither of you is doing cognitive work the other couldn't. But an aligned AI might have to do a lot of hard work figuring out what some candidate options for the universe even are, and if we want to check its work it will probably have to break big complicated visions of the future into human-comprehensible mouthfuls. So we really will need to extend trust to it - not as in trusting that whatever it picks is the only right thing, but trusting it to make decent decisions even in domains that are too complicated for human oversight.

[-]Victor Novikov3y*20

This seems to me like a "you do not understand your own values well enough" problem, not a "you need a higher moral authority to decide for you" problem.

Or, if we dissolve the idea of your values as something that produces some objective preference ordering (which I suppose is the point of this post): you lack a process that allows you to make decisions when your value system is in a contradictory state.

[-]Mary Chernyshenko6y10

But it is not important. If it were important, you wouldn't think about it as "what do I want". For example, if you want "world peace" and you have two ways to achieve it, and you can't choose because both are so great, it means either that you have no dice or there is a reason why both would fail.

[-]Charlie Steiner6y30

Yes, I hope that my framing of the problem supports this sort of conclusion :P

An alternate framing where it still seems important would be "moral uncertainty". Where when we don't know what to do, it's because we are lacking some facts, maybe even key facts. So I'm sort of sneakily arguing against that frame.

LESSWRONG
LW

LESSWRONG
LW

23

Impossible moral problems and moral authority

23

Ω 8

23

Ω 8