Followup toMixed Reference: The Great Reductionist Project

Humans need fantasy to be human.

"Tooth fairies? Hogfathers? Little—"

Yes. As practice. You have to start out learning to believe the little lies.

"So we can believe the big ones?"

Yes. Justice. Mercy. Duty. That sort of thing.

"They're not the same at all!"

You think so? Then take the universe and grind it down to the finest powder and sieve it through the finest sieve and then show me one atom of justice, one molecule of mercy.

- Susan and Death, in Hogfather by Terry Pratchett

Suppose three people find a pie - that is, three people exactly simultaneously spot a pie which has been exogenously generated in unclaimed territory. Zaire wants the entire pie; Yancy thinks that 1/3 each is fair; and Xannon thinks that fair would be taking into equal account everyone's ideas about what is "fair".

I myself would say unhesitatingly that a third of the pie each, is fair. "Fairness", as an ethical concept, can get a lot more complicated in more elaborate contexts. But in this simple context, a lot of other things that "fairness" could depend on, like work inputs, have been eliminated or made constant. Assuming no relevant conditions other than those already stated, "fairness" simplifies to the mathematical procedure of splitting the pie into equal parts; and when this logical function is run over physical reality, it outputs "1/3 for Zaire, 1/3 for Yancy, 1/3 for Xannon".

Or to put it another way - just like we get "If Oswald hadn't shot Kennedy, nobody else would've" by running a logical function over a true causal model - similarly, we can get the hypothetical 'fair' situation, whether or not it actually happens, by running the physical starting scenario through a logical function that describes what a 'fair' outcome would look like:

So am I (as Zaire would claim) just assuming-by-authority that I get to have everything my way, since I'm not defining 'fairness' the way Zaire wants to define it?

No more than mathematicians are flatly ordering everyone to assume-without-proof that two different numbers can't have the same successor. For fairness to be what everyone thinks is "fair" would be entirely circular, structurally isomorphic to "Fzeem is what everyone thinks is fzeem"... or like trying to define the counting numbers as "whatever anyone thinks is a number". It only even looks coherent because everyone secretly already has a mental picture of "numbers" - because their brain already navigated to the referent.  But something akin to axioms is needed to talk about "numbers, as opposed to something else" in the first place. Even an inchoate mental image of "0, 1, 2, ..." implies the axioms no less than a formal statement - we can extract the axioms back out by asking questions about this rough mental image.

Similarly, the intuition that fairness has something to do with dividing up the pie equally, plays a role akin to secretly already having "0, 1, 2, ..." in mind as the subject of mathematical conversation. You need axioms, not as assumptions that aren't justified, but as pointers to what the heck the conversation is supposed to be about.

Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself - if somebody thinks of Euclidean geometry when you utter the sound "num-berz" they're not doing anything false, they're associating the sound to a different logical thingy. It's not about words with intrinsically rigid referential power, it's that the words are window dressing on the underlying entities. I want to talk about a particular logical entity, as it might be defined by either axioms or inchoate images, regardless of which word-sounds may be associated to it.  If you want to call that "rigid designation", that seems to me like adding a level of indirection; I don't care about the word 'fair' in the first place, I care about the logical entity of fairness.  (Or to put it even more sharply: since my ontology does not have room for physics, logic, plus designation, I'm not very interested in discussing this 'rigid designation' business unless it's being reduced to something else.)

Once issues of justice become more complicated and all the contextual variables get added back in, we might not be sure if a disagreement about 'fairness' reflects:

  1. The equivalent of a multiplication error within the same axioms - incorrectly dividing by 3.  (Or more complicatedly:  You might have a sophisticated axiomatic concept of 'equity', and incorrectly process those axioms to invalidly yield the assertion that, in a context where 2 of the 3 must starve and there's only enough pie for at most 1 person to survive, you should still divide the pie equally instead of flipping a 3-sided coin.  Where I'm assuming that this conclusion is 'incorrect', not because I disagree with it, but because it didn't actually follow from the axioms.)
  2. Mistaken models of the physical world fed into the function - mistakenly thinking there's 2 pies, or mistakenly thinking that Zaire has no subjective experiences and is not an object of ethical value.
  3. People associating different logical functions to the letters F-A-I-R, which isn't a disagreement about some common pinpointed variable, but just different people wanting different things.

There's a lot of people who feel that this picture leaves out something fundamental, especially once we make the jump from "fair" to the broader concept of "moral", "good", or "right".  And it's this worry about leaving-out-something-fundamental that I hope to address next...

...but please note, if we confess that 'right' lives in a world of physics and logic - because everything lives in a world of physics and logic - then we have to translate 'right' into those terms somehow.

And that is the answer Susan should have given - if she could talk about sufficiently advanced epistemology, sufficiently fast - to Death's entire statement:

You think so? Then take the universe and grind it down to the finest powder and sieve it through the finest sieve and then show me one atom of justice, one molecule of mercy. And yet — Death waved a hand. And yet you act as if there is some ideal order in the world, as if there is some ... rightness in the universe by which it may be judged.

"But!" Susan should've said.  "When we judge the universe we're comparing it to a logical referent, a sort of thing that isn't in the universe!  Why, it's just like looking at a heap of 2 apples and a heap of 3 apples on a table, and comparing their invisible product to the number 6 - there isn't any 6 if you grind up the whole table, even if you grind up the whole universe, but the product is still 6, physico-logically speaking."


If you require that Rightness be written on some particular great Stone Tablet somewhere - to be "a light that shines from the sky", outside people, as a different Terry Pratchett book put it - then indeed, there's no such Stone Tablet anywhere in our universe.

But there shouldn't be such a Stone Tablet, given standard intuitions about morality.  This follows from the Euthryphro Dilemma out of ancient Greece.

The original Euthryphro dilemma goes, "Is it pious because it is loved by the gods, or loved by the gods because it is pious?" The religious version goes, "Is it good because it is commanded by God, or does God command it because it is good?"

The standard atheist reply is:  "Would you say that it's an intrinsically good thing - even if the event has no further causal consequences which are good - to slaughter babies or torture people, if that's what God says to do?"

If we can't make it good to slaughter babies by tweaking the state of God, then morality doesn't come from God; so goes the standard atheist argument.

But if you can't make it good to slaughter babies by tweaking the physical state of anything - if we can't imagine a world where some great Stone Tablet of Morality has been physically rewritten, and what is right has changed - then this is telling us that...

(drumroll)

...what's "right" is a logical thingy rather than a physical thingy, that's all.  The mark of a logical validity is that we can't concretely visualize a coherent possible world where the proposition is false.

And I mention this in hopes that I can show that it is not moral anti-realism to say that moral statements take their truth-value from logical entities.  Even in Ancient Greece, philosophers implicitly knew that 'morality' ought to be such an entity - that it couldn't be something you found when you ground the Universe to powder, because then you could resprinkle the powder and make it wonderful to kill babies - though they didn't know how to say what they knew.


There's a lot of people who still feel that Death would be right, if the universe were all physical; that the kind of dry logical entity I'm describing here, isn't sufficient to carry the bright alive feeling of goodness.

And there are others who accept that physics and logic is everything, but who - I think mistakenly - go ahead and also accept Death's stance that this makes morality a lie, or, in lesser form, that the bright alive feeling can't make it.  (Sort of like people who accept an incompatibilist theory of free will, also accept physics, and conclude with sorrow that they are indeed being controlled by physics.)

In case anyone is bored that I'm still trying to fight this battle, well, here's a quote from a recent Facebook conversation with a famous early transhumanist:

No doubt a "crippled" AI that didn't understand the existence or nature of first-person facts could be nonfriendly towards sentient beings... Only a zombie wouldn't value Heaven over Hell. For reasons we simply don't understand, the negative value and normative aspect of agony and despair is built into the nature of the experience itself. Non-reductionist? Yes, on a standard materialist ontology. But not IMO within a more defensible Strawsonian physicalism.

It would actually be quite surprisingly helpful for increasing the percentage of people who will participate meaningfully in saving the planet, if there were some reliably-working standard explanation for why physics and logic together have enough room to contain morality.  People who think that reductionism means we have to lie to our children, as Pratchett's Death advocates, won't be much enthused about the Center for Applied Rationality.  And there are a fair number of people out there who still advocate proceeding in the confidence of ineffable morality to construct sloppily designed AIs.

So far I don't know of any exposition that works reliably - for the thesis for how morality including our intuitions about whether things really are justified and so on, is preserved in the analysis to physics plus logic; that morality has been explained rather than explained away.  Nonetheless I shall now take another stab at it, starting with a simpler bright feeling:


When I see an unusually neat mathematical proof, unexpectedly short or surprisingly general, my brain gets a joyous sense of elegance.

There's presumably some functional slice through my brain that implements this emotion - some configuration subspace of spiking neural circuitry which corresponds to my feeling of elegance.  Perhaps I should say that elegance is merely about my brain switching on its elegance-signal?  But there are concepts like Kolmogorov complexity that give more formal meanings of "simple" than "Simple is whatever makes my brain feel the emotion of simplicity."  Anything you do to fool my brain wouldn't make the proof really elegant, not in that sense.  The emotion is not free of semantic content; we could build a correspondence theory for it and navigate to its logical+physical referent, and say:  "Sarah feels like this proof is elegant, and her feeling is true."  You could even say that certain proofs are elegant even if no conscious agent sees them.

My description of 'elegance' admittedly did invoke agent-dependent concepts like 'unexpectedly' short or 'surprisingly' general.  It's almost certainly true that with a different mathematical background, I would have different standards of elegance and experience that feeling on somewhat different occasions.  Even so, that still seems like moving around in a field of similar referents for the emotion - much more similar to each other than to, say, the distant cluster of 'anger'.

Rewiring my brain so that the 'elegance' sensation gets activated when I see mathematical proofs where the words have lots of vowels - that wouldn't change what is elegant.  Rather, it would make the feeling be about something else entirely; different semantics with a different truth-condition.

Indeed, it's not clear that this thought experiment is, or should be, really conceivable.  If all the associated computation is about vowels instead of elegance, then from the inside you would expect that to feel vowelly, not feel elegant...

...which is to say that even feelings can be associated with logical entities.  Though unfortunately not in any way that will feel like qualia if you can't read your own source code.  I could write out an exact description of your visual cortex's spiking code for 'blue' on paper, and it wouldn't actually look blue to you.  Still, on the higher level of description, it should seem intuitively plausible that if you tried rewriting the relevant part of your brain to count vowels, the resulting sensation would no longer have the content or even the feeling of elegance.  It would compute vowelliness, and feel vowelly.


My feeling of mathematical elegance is motivating; it makes me more likely to search for similar such proofs later and go on doing math.  You could construct an agent that tried to add more vowels instead, and if the agent asked itself why it was doing that, the resulting justification-thought wouldn't feel like because-it's-elegant, it would feel like because-it's-vowelly.

In the same sense, when you try to do what's right, you're motivated by things like (to yet again quote Frankena's list of terminal values):

"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."

If we reprogrammed you to count paperclips instead, it wouldn't feel like different things having the same kind of motivation behind it.  It wouldn't feel like doing-what's-right for a different guess about what's right.  It would feel like doing-what-leads-to-paperclips.

And I quoted the above list because the feeling of rightness isn't about implementing a particular logical function; it contains no mention of logical functions at all; in the environment of evolutionary ancestry nobody has heard of axiomatization; these feelings are about life, consciousness, etcetera.  If I could write out the whole truth-condition of the feeling in a way you could compute, you would still feel Moore's Open Question:  "I can see that this event is high-rated by logical function X, but is X really right?" - since you can't read your own source code and the description wouldn't be commensurate with your brain's native format.

"But!" you cry.  "But, is it really better to do what's right, than to maximize paperclips?"  Yes!  As soon as you start trying to cash out the logical function that gives betterness its truth-value, it will output "life, consciousness, etc. >B paperclips".  And if your brain were computing a different logical function instead, like makes-more-paperclips, it wouldn't feel better, it would feel moreclippy.

But is it really justified to keep our own sense of betterness?  Sure, and that's a logical fact - it's the objective output of the logical function corresponding to your experiential sense of what it means for something to be 'justified' in the first place.  This doesn't mean that Clippy the Paperclip Maximizer will self-modify to do only things that are justified; Clippy doesn't judge between self-modifications by computing justifications, but rather, computing clippyflurphs.

But isn't it arbitrary for Clippy to maximize paperclips?  Indeed; once you implicitly or explicitly pinpoint the logical function that gives judgments of arbitrariness their truth-value - presumably, revolving around the presence or absence of justifications - then this logical function will objectively yield that there's no justification whatsoever for maximizing paperclips (which is why I'm not going to do it) and hence that Clippy's decision is arbitrary. Conversely, Clippy finds that there's no clippyflurph for preserving life, and hence that it is unclipperiffic.  But unclipperifficness isn't arbitrariness any more than the number 17 is a right triangle; they're different logical entities pinned down by different axioms, and the corresponding judgments will have different semantic content and feel different.  If Clippy is architected to experience that-which-you-call-qualia, Clippy's feeling of clippyflurph will be structurally different from the way justification feels, not just red versus blue, but vision versus sound.

But surely one shouldn't praise the clippyflurphers rather than the just?  I quite agree; and as soon as you navigate referentially to the coherent logical entity that is the truth-condition of should - a function on potential actions and future states - it will agree with you that it's better to avoid the arbitrary than the unclipperiffic.  Unfortunately, this logical fact does not correspond to the truth-condition of any meaningful proposition computed by Clippy in the course of how it efficiently transforms the universe into paperclips, in much the same way that rightness plays no role in that-which-is-maximized by the blind processes of natural selection.

Where moral judgment is concerned, it's logic all the way down.  ALL the way down.  Any frame of reference where you're worried that it's really no better to do what's right then to maximize paperclips... well, that really part has a truth-condition (or what does the "really" mean?) and as soon as you write out the truth-condition you're going to end up with yet another ordering over actions or algorithms or meta-algorithms or something.  And since grinding up the universe won't and shouldn't yield any miniature '>' tokens, it must be a logical ordering.  And so whatever logical ordering it is you're worried about, it probably does produce 'life > paperclips' - but Clippy isn't computing that logical fact any more than your pocket calculator is computing it.

Logical facts have no power to directly affect the universe except when some part of the universe is computing them, and morality is (and should be) logic, not physics.

Which is to say:

The old wizard was staring at him, a sad look in his eyes. "I suppose I do understand now," he said quietly.

"Oh?" said Harry. "Understand what?"

"Voldemort," said the old wizard. "I understand him now at last. Because to believe that the world is truly like that, you must believe there is no justice in it, that it is woven of darkness at its core. I asked you why he became a monster, and you could give no reason. And if I could ask him, I suppose, his answer would be: Why not?"

They stood there gazing into each other's eyes, the old wizard in his robes, and the young boy with the lightning-bolt scar on his forehead.

"Tell me, Harry," said the old wizard, "will you become a monster?"

"No," said the boy, an iron certainty in his voice.

"Why not?" said the old wizard.

The young boy stood very straight, his chin raised high and proud, and said: "There is no justice in the laws of Nature, Headmaster, no term for fairness in the equations of motion. The universe is neither evil, nor good, it simply does not care. The stars don't care, or the Sun, or the sky. But they don't have to! We care! There is light in the world, and it is us!"

 

Part of the sequence Highly Advanced Epistemology 101 for Beginners

Next post: "Standard and Nonstandard Numbers"

Previous post: "Mixed Reference: The Great Reductionist Project"

New Comment
941 comments, sorted by Click to highlight new comments since: Today at 5:36 PM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Is this a fair summary?

The answer to the clever meta-moral question, “But why should we care about morality?” is just “Because when we say morality, we refer to that-which-we-care-about - and, not to belabor the point, but we care about what we care about. Whatever you think you care about, which isn’t morality, I’m calling that morality also. Precisely which things are moral and which are not is a difficult question - but there is no non-trivial meta-question.”

There is a non-trivial point in this summary, which is the meaning of "we." I could imagine a possible world in which the moral intuitions of humans diverge widely enough that there isn't anything that could reasonably be called a coherent extrapolated volition of humanity (and I worry that I already live there).

0Dues8y
Humans value some things more than others. Survival is the bedrock human value (yourself, your family, your children, your species). Followed by things like pleasure and the lives of others and the lives of animals. Every human weighs the things a little differently, and we're all bad at the math. But on average most humans weigh the important things about the same. There is a reason Elizer is able to keep going back to the example of saving a child.

If we reprogrammed you to count paperclips instead, it wouldn't feel like different things having the same kind of motivation behind it. It wouldn't feel like doing-what's-right for a different guess about what's right. It would feel like doing-what-leads-to-paperclips.

Um, how do you know?

7chaosmosis11y
It would depend on what exactly what we reprogrammed within you, I expect.
6Alicorn11y
Exactly. I mean, you could probably make it have its own quale, but you could also make it not, and I don't see why that would be in question as long as we're postulating brain-reprogramming powers.
9Eliezer Yudkowsky11y
Assume the subject of reprogramming is an existing human being, otherwise minimally altered by this reprogramming, i.e., we don't do anything that isn't necessary to switch their motivation to paperclips. So unless you do something gratuitiously non-minimal like moving the whole decision-action system out of the range of introspective modeling, or cutting way down on the detail level of introspective modeling, or changing the empathic architecture for modeling hypothetical selves, the new person will experience themselves as having ineffable 'qualia' associated with the motivation to produce paperclips. The only way to make it seem to them like their motivational quales hadn't changed over time would be to mess with the encoding of their previous memories of motivation, presumably in a structure-destroying way since the stored data and their introspectively exposed surfaces will not be naturally isomorphic. If you carry out the change to paperclip-motivation in the obvious way, cognitive comparisions of the retrieved memories to current thoughts will return 'unequal ineffable quales', and if the memories are visualized in different modalities from current thoughts, 'incomparable ineffable quales'. Doing-what-leads-to-paperclips will also be a much simpler 'quale', both from the outside perspective looking at the complexity of cognitive data, and in terms of the internal experience of complexity - unless you pack an awful lot of detail into the question of what constitutes a more preferred paperclip. Otherwise, compared to the old days when you thought about justice and fairness, introspection will show that less questioning and uncertainty is involved, and that there are fewer points of variation among the motivational thought-quales being considered. I suppose you could put in some extra work to make the previous motivations map in cognitively comparable ways along as many joints as possible, and try to edit previous memories without destroying their structure s

I think you and Alicorn may be talking past each other somewhat.

Throughout my life, it seems that what I morally value has varied more than what rightness feels like - just as it seems that what I consider status-raising has changed more than what rising in status feels like, and what I find physically pleasurable has changed more than what physical pleasures feel like. It's possible that the things my whole person is optimizing for have not changed at all, that my subjective feelings are a direct reflection of this, and that my evaluation of a change of content is merely a change in my causal model of the production of the desiderata (I thought voting for Smith would lower unemployment, but now I think voting for Jones would, etc.) But it seems more plausible to me that

1) the whole me is optimizing for various things, and these things change over time,
2) and that the conscious me is getting information inputs which it can group together by family resemblance, and which can reinforce or disincentivize its behavior.

Imagine a ship which is governed by an anarchic assembly beneath board and captained by an employee of theirs whom they motivate through in-kind bonuses. So the assembly... (read more)

This comment expands how you'd go about reprogramming someone in this way with another layer of granularity, which is certainly interesting on its own merits, but it doesn't strongly support your assertion about what it would feel like to be that someone. What makes you think this is how qualia work? Have you been performing sinister experiments in your basement? Do you have magic counterfactual-luminosity-powers?

I think Eliezer is simply suggesting that qualia don't in fact exist in a vacuum. Green feels the way it does partly because it's the color of chlorophyll. In a universe where plants had picked a different color for chlorophyll (melanophyll, say), with everything else (per impossibile) held constant, we would associate an at least slightly different quale with green and with black, because part of how colors feel is that they subtly remind us of the things that are most often colored that way. Similarly, part of how 'goodness' feels is that it imperceptibly reminds us of the extension of good; if that extension were dramatically different, then the feeling would (barring any radical redesigns of how associative thought works) be different too. In a universe where the smallest birds were ten feet tall, thinking about 'birdiness' would involve a different quale for the same reason.

7khafra11y
It sounds to me like you don't think the answer had anything to do with the question. But to think that, you'd pretty much have to discard both the functionalist and physicalist theories of mind, and go full dualist/neutral monist; wouldn't you?
2Eliezer Yudkowsky11y
I think I'll go with this as my reply - "Well, imagine that you lived in a monist universe - things would pretty much have to work that way, wouldn't they?"
1Nick_Tarleton11y
Possibly (this is total speculation) Eliezer is talking about the feeling of one's entire motivational system (or some large part of it), while you're talking about the feeling of some much narrower system that you identify as computing morality; so his conception of a Clippified human wouldn't share your terminal-ish drives to eat tasty food, be near friends, etc., and the qualia that correspond to wanting those things.
9Eliezer Yudkowsky11y
The Clippified human categorizes foods into a similar metric of similarity - still believes that fish tastes more like steak than like chocolate - but of course is not motivated to eat except insofar as staying alive helps to make more paperclips. They have taste, but not tastiness. Actually that might make a surprisingly good metaphor for a lot of the difficulty that some people have with comprehending how Clippy can understand your pain and not care - maybe I'll try it on the other end of that Facebook conversation.
8DaFranker11y
The metaphor seems like it could lose most of its effectiveness on people who have never applied the outside view to how taste and tastiness feel from inside - they've never realized that chocolate tastes good because their brain fires "good taste" when it perceives the experience "chocolate taste". The obvious resulting cognitive dissonance (from "tastes bad for others") predictions match my observations, so I suspect this would be common among non-rationalists. If the Facebook conversation you mention is with people who haven't crossed that inferential gap yet, it might prove not that useful.

Consider Bob. Bob, like most unreflective people, settles many moral questions by "am I disgusted by it?" Bob is disgusted by, among other things, feces, rotten fruit, corpses, maggots, and men kissing men. Internally, it feels to Bob like the disgust he feels at one of those stimuli is the same as the disgust he feels at the other stimuli, and brain scans show that they all activate the insula in basically the same way.

Bob goes through aversion therapy (or some other method) and eventually his insula no longer activates when he sees men kissing men.

When Bob remembers his previous reaction to that stimuli, I imagine he would remember being disgusted, but not be disgusted when he remembers the stimuli. His positions on, say, same-sex marriage or the acceptability of gay relationships have changed, and he is aware that they have changed.

Do you think this example agrees with your account? If/where it disagrees, why do you prefer your account?

I think this is really a sorites problem. If you change what's delicious only slightly, then deliciousness itself seems to be unaltered. But if you change it radically — say, if circuits similar to your old gustatory ones now trigger when and only when you see a bright light — then it seems plausible that the experience itself will be at least somewhat changed, because 'how things feel' is affected by our whole web of perceptual and conceptual associations. There isn't necessarily any sharp line where a change in deliciousness itself suddenly becomes perceptible; but it's nevertheless the case that the overall extension of 'delicious' (like 'disgusting' and 'moral') has some effect on how we experience deliciousness. E.g., deliciousness feels more foodish than lightish.

9Vaniver11y
When I look at the problem introspectively, I can see that as a sensible guess. It doesn't seem like a sensible guess when I look at it from a neurological perspective. If the activation of the insula is disgust, then the claim that outputs of the insula will have a different introspective flavor when you rewire the inputs of the insula seems doubtful. Sure, it could be the case, but why? When we hypnotize people to make them disgusted by benign things, I haven't seen any mention that the disgust has a different introspective flavor, and people seem to reason about that disgust in the exact same way that they reason about the disgust they had before. This seems like the claim that rewiring yourself leads to something like synesthesia, and that just seems like an odd and unsupported claim to me.
4Rob Bensinger11y
Certain patterns of behavior at the insula correlate with disgust. But we don't know whether they're sufficient for disgust, nor do we know which modifications within or outside of the insula change the conscious character of disgust. There are lots of problems with identity claims at this stage, so I'll just raise one: For all we know, activation patterns in a given brain region correlate with disgust because disgust is experienced when that brain region inhibits another part of the brain; an experience could consist, in context, in the absence of a certain kind of brain activity. Hypnosis data is especially difficult to evaluate, because it isn't clear (a) how reliable people's self-reports about introspection are while under hypnosis; nor (b) how reliable people's memories-of-hypnosis are afterward. Some 'dissociative' people even give contradictory phenomenological reports while under hypnosis. That said, if you know of any studies suggesting that the disgust doesn't have at all a different character, I'd be very interested to see them! If you think my claim isn't modest and fairly obvious, then it might be that you aren't understanding my claim. Redness feels at least a little bit bloodish. Greenness feels at least a little bit foresty. If we made a clone who sees evergreen forests as everred and blood as green, then their experience of greenness and redness would be partly the same, but it wouldn't be completely the same, because that overtone of bloodiness would remain in the background of a variety of green experiences, and that woodsy overtone would remain in the background of a variety of red experiences.
1Vaniver11y
I'm differentiating between "red evokes blood" and "red feels bloody," because those seem like different things to me. The former deals with memory and association, and the second deals with introspection, and so I agree that the same introspective sensation could evoke very different memories. The dynamics of introspective sensations could plausibly vary between people, and so I'm reluctant to discuss it extensively except in the context of object-level comparisons.
1Rob Bensinger11y
I'm not sure exactly what you mean by "red evokes blood." I agree that "red feels bloody" is intuitively distinct from "I tend to think explicitly about blood when I start thinking about redness," though the two are causally related. Certain shades of green to me feel fresh, clean, 'naturey;' certain shades of red to me feel violent, hot, glaring; certain shades of blue feel cool; etc. My suggestion is that these qualia, which are part of the feeling of the colors themselves for most humans, would be experientially different even when decontextualized if we'd gone through life perceiving forests as blue, oceans as red, campfires as green, etc. By analogy, the feeling of 'virtue' may be partly independent of which things we think of under the concept 'virtuous;' but it isn't completely independent of those things.
2Vaniver11y
I am aware that many humans have this sort of classification of colors, and have learned it because of its value in communication, but as far as I can tell this isn't a significant part of my mental experience. A dark green might make it easier for me to think of leaves or forests, but I don't have any experiences that I would describe as feeling 'naturey'. If oceans and forests swapped colors, I imagine that seeing the same dark green would make it easier for me to think of waves and water, but I think my introspective experience would be the same. If I can simplify your claim a bit, it sounds like if both oceans and forests were dark green, then seeing dark green would make you think of leaves and waves / feel associated feelings, and that this ensemble would be different from your current sensation of ocean blue or forest green. It seems sensible to me that the ensembles are different because they have different elements. I'm happier with modeling that as perceptual bleedover- because forests and green are heavily linked, even forests that aren't green are linked to green, and greens that aren't on leaves are linked with forests- than I am modeling that as an atom of consciousness- the sensation of foresty greens- but if your purposes are different, a different model may be more suitable.
1Rob Bensinger11y
Part of the problem may be that I'm not so sure I have a distinct, empirically robust idea of an 'atom of consciousness.' I took for granted your distinction between 'evoking blood' and 'feeling bloody,' but in practice these two ideas blend together a great deal. Some ideas -- phonological and musical ones, for example -- are instantiated in memory by certain temporal sequences and patterns of association. From my armchair, I'm not sure how much my idea of green (or goodness, or clippiness) is what it is in virtue of its temporal and associative dispositions, too. And I don't know if Eliezer is any less confused than I.
0NancyLebovitz11y
It wouldn't surprise me if the sensation of disgust has some variation from one person to another, and even for the same person, from one object to another.
4adamisom11y
I just wanted to tell everyone that it is great fun to read this in the voice of that voice actor for the Enzyte commercial :)
1FeepingCreature11y
I think this is easier because disgust is relatively arbitrary to begin with, in that it seems to implement a function over the world-you relation (roughly, things that are bad for you to eat/be near). We wouldn't expect that relation to have much coherence to begin with, so there'd be not much loss of coherence from modifying it - though, arguably, the same thing could be said for most qualia - elegance is kind of the odd one out.
5Armok_GoB11y
I wouldn't be all that suprised if the easiest way to get a human maximizing papperclips was to make it believe paperclips had epiphenomenal consciousnesses experiencing astronomical amounts of pleasure. edit: or you could just give them a false memory of god telling them to do it.
3FeepingCreature11y
The Enrichment Center would like to remind you that the Paperclip cannot speak. In the event that the Paperclip does speak, the Enrichment Center urges you to disregard its advice.
2MugaSofer11y
Wouldn't it be easier to have the programee remember themself as misunderstanding morality - like a reformed racist who previously preferred options that harmed minorities. I know when I gain more insight into my ethics I remember making decisions that, in retrospect, are incomprehensible (unless I deliberately keep in mind how I thought I should act.)
0Eugine_Nier11y
That depends on the details of how the human brain stores goals and memories.
2MugaSofer11y
Cached thoughts regularly supersede actual moral thinking, like all forms of thinking, and I am capable of remembering this experience. Am I misunderstanding your comment?
0Eugine_Nier11y
My point is that in order to "fully reprogram" someone it is also necessary to clear their "moral cache" at the very least.
1MugaSofer11y
Well ... is it? Would you notice if your morals changed when you weren't looking?
0Eugine_Nier11y
I probably would, but then again I'm in the habit of comparing the out of my moral intuitions with stored earlier versions of that output.
0MugaSofer11y
I guess it depends on how much you rely on cached thoughts in your moral reasoning. Of course, it can be hard to tell how much you're using 'em. Hmm...
1JoachimSchipper11y
I have no problem with this passage. But it does not seem obviously impossible to create a device that stimulates that-which-feels-rightness proportionally to (its estimate of) the clippiness of the universe - it's just a very peculiar kind of wireheading. As you point out, it'd be obvious, on reflection, that one's sense of rightness has changed; but that doesn't necessarily make it a different qualia, any more than having your eyes opened to the suffering of (group) changes your experience of (in)justice qua (in)justice.
0Gust11y
Although I think your point here is plausible, I don't think it fits in a post where you are talking about the logicalness of morality. This qualia problem is physical; whether your feeling changes when the structure of some part of your decision system changes depends on your implementation. Maybe your background understanding of neurology is enough for you to be somewhat confident stating this feeling/logical-function relation for humans. But mine is not and, although I could separate your metaethical explanations from your physical claims when reading the post, I think it would be better off without the latter.
4handoflixue11y
Speaking from personal experience, I can say that he's right. Explaining how I know this, much less sharing the experience, is more difficult. The simplest idea I can present is that you probably have multiple utility functions. If you're buying apples, you'll evaluate whether you like that type of apple, what the quality of the apple is, and how good the price is. For me, at least, these all FEEL different - a bruised apple doesn't "feel" overpriced the way a $5 apple at the airport does. Even disliking soft apples feels very different from recognizing a bruised apple, even though they both also go in to a larger basket of "no good". What's more, I can pick apples based on someone ELSE'S utility function, and actually often shop with my roommate's function in mind (she likes apples a lot more than me, but is also much pickier, as it happens). This feels different from using my own utility function. ---------------------------------------- The other side of this is that I would expect my brain to NOTICE it's actual goals. If my goal is to make paperclips, I will think "I should do this because it makes paperclips", instead of "I should do this because it makes people happy". My brain doesn't have a generic "I should do this" emotion, as near as I can tell - it just has ways of signalling that an activity will accomplish my goals. Thus, it seems reasonable to conclude that my feelings are more a combination of activity + outcome, not some raw platonic ideal. While sex, hiking, and a nice meal all make me "happy", they still feel completely different - I just lump them in to a larger category of "happiness" for some reason. I'd strongly suspect you can add make-more-paperclips to that emotional category, but I see absolutely no reason you could make me treat it the same as a nice dinner, because that wouldn't even make sense.

Speaking from personal experience, I can say that he's right.

So, you introspect the way that he introspects. Do all humans? Would all humans need to introspect that way for it to do the work that he wants it to do?

5handoflixue11y
Ooh, good call, thank you. I suppose it might be akin to visualization, where it actually varies from person to person. Does anyone here on LessWrong have conflicting anecdotes, though? Does anyone disagree with what I said? If not, it seems like a safe generalization for now, but it's still useful to remember I'm generalizing from one example :) Remembering that other people have genuinely alien minds is surprisingly tricky.

The other side of this is that I would expect my brain to NOTICE it's actual goals. If my goal is to make paperclips, I will think "I should do this because it makes paperclips", instead of "I should do this because it makes people happy". My brain doesn't have a generic "I should do this" emotion, as near as I can tell - it just has ways of signalling that an activity will accomplish my goals.

Iron deficiency feels like wanting ice. For clever, verbal reasons. Not being iron deficient doesn't feel like anything. My brain did not notice that it was trying to get iron - it didn't even notice it was trying to get ice, it made up reasons according to which ice was an instrumental value for some terminal goal or other.

4shminux11y
Other people? I find my own mind quite alien below the thin layer accessible to my introspection. Heck, most of the time I cannot even tell if my introspection lies to me.
2asparisi11y
I think I have a different introspection here. When I have a feeling such as 'doing-whats-right' there is a positive emotional response associated with it. Immediately I attach semantic content to that emotion: I identify it as being produced by the 'doing-whats-right' emotion. How do I do this? I suspect that my brain has done the work to figure out that emotional response X is associated with behavior Y, and just does the work quickly. But this is maleable. Over time, the emotional response associated with an act can change and this does not necessarily indicate a change in semantic content. I can, for example, give to a charity that I am not convinced is good and I still will often get the 'doing-whats-right' emotion even though the semantic content isn't really there. I can also find new things I value, and occasionally I will acknowledge that I value something before I get positive emotional reinforcement. So in my experience, they aren't identical. I strongly suspect that if you reprogrammed my brain to value counting paperclips, it would feel the same as doing what is right. At very least, this would not be inconsistent. I might learn to attach paperclippy instead of good to that emotional state, but it would feel the same.
2MugaSofer11y
... they do? For what values of "alien"?
1handoflixue11y
Because I'm not sure how else to capture a "scale of alien-ness": I once wrote a sci-fi race that was a blind, deaf ooze, but extremely intelligent and very sensitive to tactile input. Over the years, and with the help of a few other people, I've gotten a fairly good feel for their mindset and how they approach the world. There's a distinct subset of humans which I find vastly more puzzling than these guys.
4A1987dM11y
From Humans in Funny Suits:
1handoflixue11y
The race was explicitly designed to try and avoid "humans in funny suits", and have a culture that's probably more foreign than the 1960s. But I'm only 29, and haven't traveled outside of English-speaking countries, so take that with a dash of salt! On a 0-10 scale, with myself at 0, humans in funny suits at 1, and the 1960s at 2, I'd rate my creation as a 4, and a subset of humanity exists in the 4-5 range. Around 5, I have trouble with the idea that there's coherent intelligent reasoning happening, because the process is just completely lost on me, and I don't think I'd be able to easily assign anything more than a 5, much less even speculate on what a 10 would look like. Trying to give a specific answer to "how alien is it" is a lot harder than it seems! :)
4IlyaShpitser11y
If I may make a recommendation, if you are concerned about "alien aliens", read a few things by Stanislaw Lem. The main theme of Lem's scifi, I would say, is alien minds, and failure of first contact. "Solaris" is his most famous work (but the adaptation with Clooney is predictably terrible).
0handoflixue11y
Not sure if I've read Lem, but I'll be sure to check it out. I have a love for "truly alien" science fiction, which is why I had to try my hand at making one of my own :)
3Eugine_Nier11y
Well reading fiction (and non-fiction) for which English speakers of your generation weren't the target audience is a good way to start compensating.
1handoflixue11y
I've got a lot of exposure to "golden age" science fiction and fantasy, so going back a few decades isn't hard for me. I just don't get exposed to many other good sources. The "classics" seem to generally fail to capture that foreignness. If you have recommendations, especially a broader method than just naming a couple authors, I'd love to hear it. Most of my favourite authors have a strong focus on foreign cultures, either exploring them or just having characters from diverse backgrounds.
2beoShaffer11y
Anime&Manga, particularly the older stuff is a decent source.
0handoflixue11y
... it is really sad that I completely forgot that anime and manga isn't English. I grew up around it, so it's just a natural part of my culture. Suffice to say, I've had a lot of exposure -- but not to anything older than I am. Any recommendations for OLD anime or manga, given I don't speak/read Japanese? :)
0beoShaffer11y
You're probably best of asking on a manga/forum, but Barefoot Gen is a good, and depressing, start.
0Eugine_Nier11y
Which time period do you mean by this? "Golden age of science fiction" typically refers to the 1940's and 1950's, "golden age of fantasy" to the late 1970's and early 1980's. If you mean the latter time period, read stuff from the former as a start. Also try going back at least a century to the foundational fantasy authors, e.g., Edgar Rice Burroughs, William Morris's The Well at the World's End. Go even further back to things like Treasure Island, or The Three Musketeers. Or even further back to the days when people believed the stuff in their "fantasy" could actually happen. Read Dante's Divine Comedy, Thomas Moore's Utopia, an actual chivalric romance (I haven't read any so I can't give recommendations). A good rule of thumb is that you should experience values dissonance while reading them. A culture whose values don't make you feel uncomfortable isn't truly alien. Also for this reason, avoid modern adaptations as these tend to do their best clean up the politically incorrect parts and otherwise modernize the worldview.
0kodos9611y
I'm intrigued. Do you have a link?
0handoflixue11y
Sadly not. I really should do a proper write-up, but right now they're mostly stored in the head of me and their co-creator.
0shminux11y
Secondary goals often feel like primary. Breathing and quenching thirst are means of achieving the primary goal of survival (and procreation), yet they themselves feel like primary. Similarly, a paperclip maximizer may feel compelled to harvest iron without any awareness that it wants to do it in order to produce paperclips.
6handoflixue11y
Bull! I'm quite aware of why I eat, breathe, and drink. Why in the world would a paperclip maximizer not be aware of this? Unless you assume Paperclippers are just rock-bottom stupid I'd also expect them to eventually notice the correlation between mining iron, smelting it, and shaping it in to a weird semi-spiral design... and the sudden rise in the number of paperclips in the world.
0shminux11y
I'm not sure that awareness is needed for paperclip maximizing. For example, one might call fire a very good CO2 maximizer. Actually, I'm not even sure you can apply the word awareness to non-human-like optimizers.
0handoflixue11y
"If we reprogrammed you to count paperclips instead" This is a conversation about changing my core utility function / goals, and what you are discussing would be far more of an architectural change. I meant, within my architecture (and, I assume, generalizing to most human architectures and most goals), we are, on some level, aware of the actual goal. There are occasional failure states (Alicorn mentioned iron deficiencies register as a craving for ice o.o), but these tend to tie in to low-level failures, not high-order goals like "make a paperclip", and STILL we tend to manage to identify these and learn how to achieve our actual goals.
5Nornagest11y
Survival and procreation aren't primary goals in any direct sense. We have urges that have been selected for because they contribute to inclusive genetic fitness, but at the implementation level they don't seem to be evaluated by their contributions to some sort of unitary probability-of-survival metric; similarly, some actions that do contribute greatly to inclusive genetic fitness (like donating eggs or sperm) are quite rare in practice and go almost wholly unrewarded by our biology. Because of this architecture, we end up with situations where we sate our psychological needs at the expense of the factors that originally selected for them: witness birth control or artificial sweeteners. This is basically the same point Eliezer was making here. It might be meaningful to treat supergoals as intentional if we were discussing an AI, since in that case there would be a unifying intent behind each fitness metric that actually gets implemented, but even in that case I'd say it's more accurate to talk about the supergoal as a property not of the AI's mind but of its implementors. Humans, of course, don't have that excuse.
0shminux11y
All good points. I was mostly thinking about an evolved paperclip maximizer, which may or may not be a result of a fooming paperclip-maximizing AI.
0[anonymous]11y
Evolved creatures as we know them (at least the ones with complex brains) are reward-center-reward maximizers, which implicitly correlates with being offspring maximizers. (Actual, non-brainy organisms are probably closer to offspring maximizers).
0Eugine_Nier11y
An evolved agent wouldn't evolve to maximize paper clips.
1MugaSofer11y
It could if the environment rewarded paperclips. Admittedly this would require an artificial environment, but that's hardly impossible.
0Luke_A_Somers11y
So far as I can tell, he chose to carve the world at this joint when making the definition of 'right'. In short, by definition. This is hardly the first time. Not too long ago, and perhaps in this sequence, there was a post about rightness and multiple-place-functions that justified the utility of this definition.
0endoself11y
I think he's talking about the obvious fact that you'd be able to think to yourself "it seems I'm trying to maximize paperclips", as well as the other differences in your experience that would occur for similar reasons.

The standard religious reply to the baby-slaughter dilemma goes something like this:

Sure, if G-d commanded us to slaughter babies, then killing babies would be good. And if "2+2=3" was a theorem of PA, then "2+2=3" would be true. But G-d logically cannot command us to do a bad thing, anymore than PA can prove something that doesn't follow from its axioms. (We use "omnipotent" to mean "really really powerful", not "actually omnipotent" which isn't even a coherent concept. G-d can't make a stone so heavy he can't lift it, draw a square circle, or be evil.) Religion has destroyed my humanity exactly as much as studying arithmetic has destroyed your numeracy. (Please pay no attention to the parts of the Bible where G-d commands exactly that.)

8lavalamp11y
But that's just choosing the other horn of the dilemma, no? I.e., "god commands thing because they are moral." And of course the atheist response to that is, Not that anyone here didn't already know this, of course. The wikipedia page lists some theistic responses that purport to evade both horns, but I don't recall being convinced that they were even coherent when I last looked at it.

It does choose a horn, but it's the other one, "things are moral because G-d commands them". It just denies the connotation that there exists a possible Counterfactual!G-d which could decide that Real!evil things are Counterfactual!good; in all possible worlds, G-d either wants the same thing or is something different mistakenly called "G-d". (Yeah, there's a possible world where we're ruled by an entity who pretends to be G-d and so we believe that we should kill babies. And there's a possible world where you're hallucinating this conversation.)

Or you could say it claims equivalence. Is this road sign a triangle because it has three sides, or does it have three sides because it is a triangle? If you pick the latter, does that mean that if triangles had four sides, the sign would change shape to have four sides? If you pick the former, does that mean that I can have three sides without being a triangle? (I don't think this one is quite fair, because we can imagine a powerful creator who wants immoral things.)

Three possible responses to the atheist response:

  • Sure. Not believing has bad consequences - you're wrong as a matter of fact, you don't get special believ

... (read more)

Obvious further atheist reply to the denial of counterfactuals: If God's desires don't vary across possible worlds there exists a logical abstraction which only describes the structure of the desires and doesn't make mention of God, just like if multiplication-of-apples doesn't vary across possible worlds, we can strip out the apples and talk about the multiplication.

8dspeyer11y
I think that's pretty close to what a lot of religious people actually believe in. They just like the one-syllable description.
0Alejandro111y
The obvious theist counter-reply is that the structure of God's desires is logically related to the essence of God, in a way that you can't have the goodness without the God nor more than God without the goodness, they are part of the same logical structure. (Aquinas: "God is by essence goodness itself") I think this is a self-consistent metaethics as metaethics goes. The problem is that God is at the same time part of the realm of abstract logical structures like "goodness", and a concrete being that causes the world to exist, causes miracles, has desires, etc. The fault is not in the metaethics, it is in the confused metaphysics that allows for a concrete being to "exist essentially" as part of its logical structure. ETA: of course, you could say the metaethics is self-consistent but also false, because it locates "goodness" outside ourselves (our extrapolated desires) which is where it really is. But for the Thomist I am currently emulating, "our extrapolated desires" sound a lot like "our final cause, the perfection to which we tend by our essence" and God is the ultimate final cause. The problem is again the metaphysics (in this case, using final causes without realizing they are mind projecting fallacy), not the metaethics.
5DaFranker11y
My mind reduces all of this to "God = Confusion". What am I missing?
0Alejandro111y
Well, I said that the metaphysics is confused, so we agree. I just think the metaethics part of religious philosophy can be put in order without falling into Euthyphro, the problem is in its broader philosophical system.
0DaFranker11y
Not quite how I'd put it. I meant that in my mind the whole metaethics part implies that "God" is just a shorthand term for "whatever turns out to be 'goodness', even if we don't understand it yet", and that this resolves to the fact that "God" serves no other purposes than to confuse morality with other things within this context. I think we still agree, though.
6MixedNuts11y
Using the word also implies that this goodness-embodying thing is sapient and has superpowers.
2dspeyer11y
Or that it is sometimes useful to tell metaphorical stories about this goodness-embodying thing as if it were sapient and had superpowers. Or as if the ancients thought it was sapient and had superpowers. They were wrong about that, but right about enough important things that we still value their writings.
3Eugine_Nier11y
As I explained here, it's perfectly reasonable to describe mathematical abstractions as causes.
0MixedNuts11y
How would a theist (at least the somewhat smart theist I'm emulating) disagree with that? That sounds a lot like "If all worlds contain a single deity, we can talk about the number one in non-theological contexts".
5lavalamp11y
It seems like you're claiming an identity relationship between god and morality, and I find myself very confused as to what that could possibly mean. I mean, it's sort of like I just encountered someone claiming that "friendship" and "dolphins" are really the same thing. One or both of us must be very confused about what the labels "friendship" and/or "dolphins" signify, or what this idea of "sameness" is, or something else...
6MixedNuts11y
See Alejandro's comment. Define G-d as "that which creates morality, and also lives in the sky and has superpowers". If you insist on the view of morality as a fixed logical abstraction, that would be a set of axioms. (Modus ponens has the Buddha-nature!) Then all you have to do is settle the factual question of whether the short-tempered creator who ordered you to genocide your neighbors embodies this set of axioms. If not, well, you live in a weird hybrid universe where G-d intervened to give you some sense of morality but is weaker than whichever Cthulhu or amoral physical law made and rules your world. Sorry.
5shminux11y
Out of curiosity, why do you write G-d, not God? The original injunction against taking God's name in vain applied to the name in the old testament, which is usually mangled in the modern English as Jehovah, not to the mangled Germanic word meaning "idol".
9MixedNuts11y
People who care about that kind of thing usually think it counts as a Name, but don't think there's anything wrong with typing it (though it's still best avoided in case someone prints out the page). Trying to write it makes me squirm horribly and if I absolutely need the whole word I'll copy-paste it. I can totally write small-g "god" though, to talk about deities in general (or as a polite cuss). I feel absolutely silly about it, I'm an atheist and I'm not even Jewish (though I do have a weird cultural-appropriatey obsession). Oh well, everyone has weird phobias.
2kodos9611y
Thought experiment: suppose I were to tell you that every time I see you write out "G-d", I responded by writing "God", or perhaps even "YHWH", on a piece of paper, 10 times. Would that knowledge alter your behavior? How about if I instead (or additionally) spoke it aloud? Edit: downvote explanation requested.
3MixedNuts11y
It feels exactly equivalent to telling me that every time you see me turn down licorice, you'll eat ten wheels of it. It would bother me slightly if you normally avoided taking the Name in vain (and you didn't, like, consider it a sacred duty to annoy me), but not to the point I'd change my behavior. Which I didn't know, but makes sense in hindsight (as hindsight is wont to do); sacredness is a hobby, and I might be miffed at fellow enthusiasts Doing It Wrong, but not at people who prefer fishing or something.
0shminux11y
Why should s/he care about what you choose to do?
4kodos9611y
I don't know. That's why I asked.
-1Eugine_Nier11y
1) I don't believe you. 2) I don't respond to blackmail.
4kodos9611y
What???!!! Are you suggesting that I'm actually planning on conducting the proposed thought experiment? Actually, physically, getting a piece of paper and writing out the words in question? I assure you, this is not the case. I don't even have any blank paper in my home - this is the 21st century after all. This is a thought experiment I'm proposing, in order to help me better understand MixedNuts' mental model. No different from proposing a thought experiment involving dust motes and eternal torture. Are you saying that Eliezer should be punished for considering such hypothetical situations, a trillion times over?
2Eugine_Nier11y
Yes I know, and my comment was how I would respond in your thought experiment. (Edited: the first version accidentally implied the opposite of what I intended.)
-1kodos9611y
??? Ok, skipping over the bizarre irrationality of your making that assumption in the first place, now that I've clarified the situation and told you in no uncertain terms that I am NOT planning on conducting such an experiment (other than inside my head), are you saying you think I'm lying? You sincerely believe that I literally have a pen and paper in front of me, and I'm going through MixedNuts's comment history and writing out sacred names for each occurance of "G-d"? Do you actually believe that? Or are you pulling our collective leg? In the event that you do actually believe that, what kind of evidence might I provide that would change your mind? Or is this an unfalsifiable belief?
0Eugine_Nier11y
Oops. See my edit.
3wedrifid11y
My usual response to reading 2) is to think 1). I wonder if you really wouldn't respond to blackmail if the stakes were high and you'd actually lose something critical. "I don't respond to blackmail" usually means "I claim social dominance in this conflict".
4kodos9611y
Not in general, but in this particular instance, the error is in seeing any "conflict" whatsoever. This was not intended as a challenge, or a dick-waving contest, just a sincerely proposed thought experiment in order to help me better understand MixedNuts' mental model.
3wedrifid11y
(My response was intended to be within the thought experiment mode, not external. I took Eugine's as being within that mode too.)
0kodos9611y
Thanks, I apppreciate that. My pique was in response to Eugine's downvote, not his comment.
0A1987dM11y
“In practice, virtually everyone seems to judge a large matter of principle to be more important than a small one of pragmatics, and vice versa — everyone except philosophers, that is.” (Gary Drescher, Good and Real)
0[anonymous]11y
Also: 0) The laws of Moses aren't even binding on Gentiles.
-1Decius11y
Isn't blackmail a little extreme?
3kodos9611y
Yes, which is why I explicitly labled it as only a thought experiment. This seems to me to be entirely in keeping with the LW tradition of thought experiments regarding dust particles and eternal torture.... by posing such a question, you're not actually threatening to torture anybody. Edit: downvote explantion requested.
0Decius11y
Or put a dost mote in everybody's eye. Withdrawn.
1shminux11y
How interesting. Phobias are a form of alief, which makes this oddly relevant to my recent post.
3MixedNuts11y
I don't think it's quite the same. I have these sinking moments of "Whew, thank... wait, thank nothing" and "Oh please... crap, nobody's listening", but here I don't feel like I'm being disrespectful to Sky Dude (and if I cared I wouldn't call him Sky Dude). The emotion is clearly associated with the word, and doesn't go "whoops, looks like I have no referent" upon reflection. What seems to be behind it is a feeling that if I did that, I would be practicing my religion wrong, and I like my religion. It's a jumble of things that give me an oxytocin kick, mostly consciously picked up, but it grows organically and sometimes plucks new dogma out of the environment. ("From now on Ruby Tuesday counts as religious music. Any questions?") I can't easily shed a part, it has to stop feeling sacred of its own accord.
-8kodos9611y
0Nisan11y
You can eliminate inconvenient phobias with flooding. I can personally recommend sacrilege. EDIT: It sounds like maybe it's not just a phobia.
0[anonymous]11y
Step 1: learn Italian; step 2: google for "Mario Magnotta" or "Germano Mosconi" or "San Culamo".
2lavalamp11y
I think there's a bug in your theist-simulation module ^^ I've yet to meet one that could have spontaneously come up with that statement. Anyway, more to the point... in the definition of god you give, it seems to me that the "lives in sky with superpowers" part is sort of tacked on to the "creates morality" part, and I don't see why I can't talk about the "creates morality" part separate from the tacked-on bits. And if that is possible, I think this definition of god is still vulnerable to the dilemma (although it would seem clear that the second horn is the correct one; god contains a perfect implementation of morality, therefore what he says happens to be moral).
9MugaSofer11y
Hi there.
2lavalamp11y
Are you a real theist or do you just like to abuse the common terminology (like, as far as I can tell, user:WillNewsome)? :)
4MugaSofer11y
A real theist. Even a Christian, although mostly Deist these days.
1lavalamp11y
So you think there's a god, but it's conceivable that the god has basically nothing to do with our universe? If so, I don't see how you can believe this while giving a similar definition for "god" as an average (median?) theist. (It's possible I have an unrepresentative sample, but all the Christians I've met IRL who know what deism is consider it a heresy... I think I tend to agree with them that there's not that much difference between the deist god and no god...)
0MugaSofer11y
That "mostly" is important. While there is a definite difference between deism and atheism (it's all in the initial conditions) it would still be considered heretical by all major religions except maybe Bhuddism because they all claim miracles. I reckon Jesus and maybe a few others probably worked miracles, but that God doesn't need to do all that much; He designed this world and thus presumably planned it all out in advance (or rather from outside our four-dimensional perspective.) But there were still adjustments, most importantly Christianity, which needed a few good miracles to demonstrate authority (note Jesus only heals people in order to demonstrate his divine mandate, not just to, well, heal people.)
3Oligopsony11y
That depends on the Gospel in question. The Johannine Jesus works miracles to show that he's God; the Matthean Jesus is constantly frustrated that everyone follows him around, tells everyone to shut up, and rejects Satan's temptation to publicly show his divine favor as an affront to God.
0MugaSofer11y
He works miracles to show authority. That doesn't necessarily mean declaring you're the actual messiah, at least at first.
3Peterdjones11y
So you can have N>1 miracles and still have deism? I always thought N was 0 for that.
6MixedNuts11y
I think (pure) deism is N=1 ("let's get this thing started") and N=0 is "atheism is true but I like thinking about epiphenomena".
5MugaSofer11y
I'm not actually a deist. I'm just more deist than the average theist.
1kodos9611y
And also, to occasionally demonstrate profound bigotry, as in Matthew 15:22-26: Was his purpose in that to demonstrate that "his divine mandate" applied only to persons of certain ethnicities?
-2MugaSofer11y
One, that's NOT using his powers. Two, she persuaded him otherwise. And three, I've seen it argued he knew she would offer a convincing argument and was just playing along. Not sure how solid that argument is, but ... it does sound plausible.
0lavalamp11y
OK, you've convinced me you're (just barely) a theist (and not really a deist as I understand the term). To go back to the original quotation (http://lesswrong.com/lw/fv3/by_which_it_may_be_judged/80ut): So you consider the "factual question" above to be meaningful? If so, presumably you give a low probability for living in the "weird hybrid universe"? How low?
-4MugaSofer11y
About the same as 2+2=3. The universe exists; gotta have a creator. God is logically necessary so ...
5lavalamp11y
OK; my surprise was predicated on the hypothetical theist giving the sentence a non-negligible probability; I admit I didn't express this originally, so you'll have to take my word that it's what I meant. Thanks for humoring me :) On another note, you do surprise me with "God is logically necessary"; although I know that's at least a common theist position, it's difficult for me to see how one can maintain that without redefining "god" into something unrecognizable.
4drnickbone11y
This "God is logically necessary" is an increasingly common move among philosophical theists, though virtually unheard of in the wider theistic community. Of course it is frustratingly hard to argue with. No matter how much evidence an atheist tries to present (evolution, cosmology, plagues, holocausts, multiple religions, psychology of religious experience and self-deception, sociology, history of religions, critical studies of scriptures etc. etc.) the theist won't update an epistemic probability of 1 to anything less than 1, so is fundamentally immovable. My guess is that this is precisely the point: the philosophical theist basically wants a position that he can defend "come what may" while still - at least superficially - playing the moves of the rationality game, and gaining a form of acceptance in philosophical circles.
2MugaSofer11y
Who said I have a probability of 1? I said the same probability (roughly) as 2+2=3. That's not the same as 1. But how exactly are those things evidence against God (except maybe plagues, and even then it's trivially easy to justify them as necessary.) Some of them could be evidence against (or for) Christianity, but not God. I'm much less certain of Christianity than God, if it helps.
2drnickbone11y
OK, so you are in some (small) doubt whether God is logically necessary or not, in that your epistemic probability of God's existence is 2+2-3, and not exactly 1:-) Or, put another way, you are able to imagine some sort of "world" in which God does not exist, but you are not totally sure whether that is a logically impossible world (you can imagine that it is logically possible after all)? Perhaps you think like this: 1. God is either logically necessary or logically impossible 2. I'm pretty sure (probability very close to 1) that God's existence is logically possible So: 3. I'm pretty sure (probability very close to 1) that God's existence is logically necessary. To support 1, you might be working with a definition of God like St Anselm's (a being than which a greater cannot be conceived) or Alvin Plantinga's (a maximally great being, which has the property of maximal excellence - including omnipotence, omniscience and moral perfection - in every possible world). If you have a different sort of God conception then that's fine, but just trying to clear up misunderstanding here.
0MugaSofer11y
Yup. It's the Anslem one, in fact.
5Peterdjones11y
Well, it's not like there's a pre-existing critique of that, or anything.
1drnickbone11y
Yeah, there's only about 900 years or so of critique... But let's cut to the chase here. For sake of argument, let's grant that there is some meaningful "greater than" order between beings (whether or not they exist) that there is a possible maximum to the order (rather than an unending chain of ever-greater beings), that parodies like Gaunilo's island fail for some unknown reason, that existence is a predicate, that there is no distinction between conceivability and logical possibility, that beings which exist are greater than beings which don't, and a few thousand other nitpicks. There is still a problem that premises 1) and 2) don't follow from Anselm's definition. We can try to clarify the definition like this: (*) G is a being than which a greater cannot be conceived iff for every possible world w where G exists, there is no possible world v and being H such that H in world v is greater than G in world w No difficulty there... Anselm's "Fool" can coherently grasp the concept of such a being and imagine a world w where G exists, but can also consistently claim that the actual world a is not one of those worlds. Premise 1) fails. Or we can try to clarify it like this: (**) G is a being than which a greater cannot be conceived iff there are no possible worlds v, w and no being H such that H in world v is greater than G in world w That is closer to Plantinga's definition of maximal greatness, and does establish Premise 1). But now Premise 2) is implausible, since it is not at all obvious that any possible being satisfies that definition. The Fool is still scratching his head trying to understand it...
0MugaSofer11y
I am no longer responding to arguments on this topic, although I will clarify my points if asked. Political argument in an environment where I am already aware of the consensus position on this topic is not productive. It bugs the hell out of me not to respond to comments like this, but a lengthy and expensive defence against arguments that I have already encountered elsewhere just isn't worth it.
0drnickbone11y
Sorry my comment wasn't intended to be political here. I was simply pointing out that even if all the classical criticisms of St Anselm's OA argument are dropped, this argument still fails to establish that a "being than which a greater cannot be conceived" is a logically necessary being rather than a logically contingent being. The argument just can't work unless you convert it into something like Alvin Plantinga's version of the OA. Since you were favouring St A's version over Plantinga's version, I thought you might not be aware of that. Clearly you are aware of it, so my post was not helpful, and you are not going to respond to this anyway on LW. However, if you wish to continue the point by email, feel free to take my username and add @ gmail.com.
0MugaSofer11y
Fair enough. I was indeed aware of that criticism, incidentally.
1Jayson_Virissimo11y
Or counters to those pre-existing critiques, etc...
3Peterdjones11y
The phil. community is pretty close to consensus , for once, on the OA.
3Jayson_Virissimo11y
Yeah, as far as the "classical ontological arguments" are concerned, virtually no philosopher considers them sound. On the other hand, I am under the impression that the "modern modal ontological arguments" (Gödel, Plantinga, etc...) are not well known outside of philosophy of religion and so there couldn't be a consensus one way or the other (taking philosophy as a whole).
1MugaSofer11y
I am no longer responding to arguments on this topic, although I will clarify my points if asked. Political argument in an environment where I am already aware of the consensus position on this topic is not productive. It bugs the hell out of me not to respond to comments like this, but a lengthy and expensive defense against arguments that I have already encountered elsewhere just isn't worth it.
0MugaSofer11y
Source?
0MugaSofer11y
I have read the critiques, and the critiques of the critiques, and so on and so forth. If there is some "magic bullet" argument I somehow haven't seen, LessWrong does not seem the place to look for it. I will not respond to further attempts at argument. We all have political stakes in this; LessWrong is generally safe from mindkilled dialogue and I would like it to stay that way, even if it means accepting a consensus I believe to be inaccurate. Frankly, I have nothing to gain from fighting this point. So I'm not going to pay the cost of doing so.
1drnickbone11y
P.S. On a simple point of logic P(God exists) = P(God exists & Christianity is true) + P(God exists and Christianity is not true). Any evidence that reduces the first term also reduces the sum. In any case, the example evidences I cited are general evidence against any sort of omni being, because they are *not the sorts of things we would expect to observe if there were such a being, but are very much what we'd expect to observe if there weren't.
5wedrifid11y
No it doesn't. Any evidence that reduces the first term by a greater degree than it increases the second term also reduces the sum. For example if God appeared before me and said "There is one God, Allah, and Mohammed is My prophet" it would raise p(God exists), lower p(God exists & Christianity is true) and significantly raise p(psychotic episode).
3A1987dM11y
ITYM "lower p(God exists & Christianity is true)".
1wedrifid11y
Thanks.
2drnickbone11y
Good point... What I was getting at here is that evidence which reduces the probability of the Christian God but leaves probability of other concepts of God unchanged still reduces P(God). But you are correct, I didn't quite say that.
2wedrifid11y
Your point is a valid one!
2MugaSofer11y
For example? Bearing in mind that I am well aware of all your "example evidences" and they do not appear confusing - although I have encountered other conceptions of God that would be so confused (for example, those who don't think God can have knowledge about the future - because free will - might be puzzled by His failure to intervene in holocausts.) EDIT:
3DaFranker11y
Despite looking for some way to do so, I've never found any. I presume you can't. Philosophical theists are happy to completely ignore this issue, and gaily go on to conflate this new "god" with their previous intuitive ideas of what "god" is, which is (from the outside view) obviously quite confused and a very bad way to think and to use words.
0MugaSofer11y
Well, my idea of what "God" is would be an omnipotent, omnibenevolent creator. That doesn't jive very well with notions like hell, at first glance, but there are theories as to why a benevolent God would torture people. My personal theory is too many inferential steps away to explain here, but suffice to say hell is ... toned down ... in most of them.
0MugaSofer11y
Oh, OK. I just meant it sounds like something I would say, probably in order to humour an atheist. The traditional method is the Ontological argument, not to be confused with two other arguments with that name; but it's generally considered rather ... suspect. However, it does get you a logically necessary, omnipotent, omnibenevolent God; I'm still somewhat confused as to whether it's actually valid.
1Decius11y
So it is trivially likely that the creator of the universe (God) embodies the set of axioms which describe morality? God is not good? I handle that contradiction by pointing out that the entity which created the universe, the abstraction which is morality, and the entity which loves genocide are not necessarily the same.
0MugaSofer11y
There certainly seems to be some sort of optimisation going on. But I don't come to LW to debate theology. I'm not here to start arguments. Certainly not about an issue the community has already decided against me on.
0Decius11y
The universe probably seems optimized for what it is; is that evidence of intelligence, or anthropic effect?
1MugaSofer11y
I am no longer responding to arguments on this topic, although I will clarify my points if asked. Political argument in an environment where I am already aware of the consensus position on this topic is not productive. It bugs the hell out of me not to respond to comments like this, but a lengthy and expensive defense against arguments that I have already encountered elsewhere just isn't worth it.
-1MixedNuts11y
It is logically necessary that the cause of the universe be sapient?
0MugaSofer11y
Define "sapient". An optimiser, certainly.
-2Peterdjones11y
"Creation must have a creator" is about as good as "the-randomly-occuring-totailty randomly occurred".
0MugaSofer11y
OK, firstly, I'm not looking for a debate on theology here; I'm well aware of what the LW consensus thinks of theism. Secondly, what the hell is that supposed to mean?
2Peterdjones11y
You seem to have started one. That one version of the First Cause argument begs the question by how it describes the universe.
1MugaSofer11y
I clarified a probability estimate. I certainly didn't intend an argument:( As ... created. Optimized? It's more an explanation, I guess.
-1Sengachi11y
Deism is essentially the belief that an intelligent entity formed, and then generated all of the universe, sans other addendums, as opposed to the belief that a point mass formed and chaoticly generated all of the universe.
1lavalamp11y
Yes, but those two beliefs don't predict different resulting universes as far as I can tell. They're functionally equivalent, and I disbelieve the one that has to pay a complexity penalty.
-2Decius11y
I typically don't accept the mainstream Judeo-Christian text as metaphorical truth, but if I did I can settle that question in the negative: The Jehovah of those books is the force that forbade knowledge and life to mankind in Genesis, and therefore does not embody morality. He is also not the creator of morality nor of the universe, because that would lead to a contradiction.
2MixedNuts11y
I dunno, dude could have good reasons to want knowledge of good and evil staying hush-hush. (Forbidding knowledge in general would indeed be super evil.) For example: You have intuitions telling you to eat when you're hungry and give food to others when they're hungry. And then you learn that the first intuition benefits you but the second makes you a good person. At this point it gets tempting to say "Screw being a good person, I'm going to stuff my face while others starve", whereas before you automatically shared fairly. You could have chosen to do that before (don't get on my case about free will), but it would have felt as weird as deciding to starve just so others could have seconds. Whereas now you're tempted all the time, which is a major bummer on the not-sinning front. I'm making this up, but it's a reasonable possibility. Also, wasn't the tree of life totally allowed in the first place? We just screwed up and ate the forbidden fruit and got kicked out before we got around to it. You could say it's evil to forbid it later, but it's not that evil to let people die when an afterlife exists. Also there's an idea (at least one Christian believes this) that G-d can't share his power (like, polytheism would be a logical paradox). Eating from both trees would make humans equal to G-d (that part is canon), so dude is forced to prevent that. You can still prove pretty easily that the guy is evil. For example, killing a kid (through disease, not instant transfer to the afterlife) to punish his father (while his mother has done nothing wrong). Or ordering genocides. (The killing part is cool because afterlife, the raping and enslaving part less so.) Or making a bunch of women infertile because it kinda looked like the head of the household was banging a married woman he thought was single. Or cursing all descendents of a guy who accidentally saw his father streaking, but being A-OK with raping your own father if there are no marriageable men available. Or... well,
-1MugaSofer11y
You sure? They believed in a gloomy underworld-style afterlife in those days.
0MixedNuts11y
Well, it's not as bad as it sounds, anyway. It's forced relocation, not murder-murder. How do you know what they believed? Mordern Judaism is very vague about the afterlife - the declassified material just mumbles something to the effect of "after the Singularity hits, the righteous will be thawed and live in transhuman utopia", and the advanced manual can't decide if it likes reincarnation or not. Do we have sources from back when?
-2MugaSofer11y
As I said, that's debatable; most humans historically believed that's what "death" consisted of, after all. That's not to say it's wrong. Just debatable. Eh? Google "sheol". It's usually translated as "hell" or "the grave" these days, to give the impression of continuity.
3MixedNuts11y
There's something to be said against equating transhumanism with religious concepts, but the world to come is an exact parallel. I don't know much about Kabbalah because I'm worried it'll fry my brain, but Gilgul is a thing. I always interpreted sheol as just the literal grave, but apparently it refers to an actual world. Thanks.
-2MugaSofer11y
Well, it is if you expect SAIs to be able to reconstruct anyone, anyway. But thanks for clarifying. Huh. You learn something new every day.
-1Decius11y
No, the Tree of Life and the Tree of Knowledge (of Good and Evil) were both forbidden. My position is that suppressing knowledge of any kind is Evil. The contradiction is that the creator of the universe should not have created anything which it doesn't want. If nothing else, can't the creator of the universe hex-edit it from his metauniverse position and remove the tree of knowledge? How is that consistent with morality?
9MixedNuts11y
Genesis 2:16-2:17 looks pretty clear to me: every tree which isn't the tree of knowledge is okay. Genesis 3:22 can be interpreted as either referring to a previous life tree ban or establishing one. If you accept the next gen fic as canon, Revelations 22:14 says that the tree will be allowed at the end, which is evidence it was just a tempban after the fall. Where do you get that the tree of life was off-limits? Sheesh. I'll actively suppress knowledge of your plans against the local dictator. (Isn't devil snake guy analogous?) I'll actively suppress knowledge of that weird fantasy you keep having where you murder everyone and have sex with an echidna, because you're allowed privacy. Standard reply is that free will outweighs everything else. You have to give people the option to be evil.
4BerryPick611y
There is no reason an omnipotent God couldn't have created creatures with free will that still always choose to be good. See Mackie, 1955.
3MixedNuts11y
Yeah, or at least put the option to be evil somewhere other than right in the middle of the garden with a "Do not eat, or else!" sign on it for a species you created vulnerable to reverse psychology.
1Eugine_Nier11y
My understanding is that the vulnerability to reverse psychology was one of the consequences of eating the fruit.
0MugaSofer11y
That's an interesting one. I hadn't heard that.
1Decius11y
There is a trivial argument against an omniscient, omnipotent, benevolent god. Why would a god with up to two of those three characteristics make creatures with free will that still always choose to be good?
0MugaSofer11y
Well, that depends on your understanding of "free will", doesn't it? Most people here would agree with you, but most people making that particular argument wouldn't.
2drnickbone11y
The most important issue is that however the theist defines "free will", he has the burden of showing that free will by that very definition is supremely valuable: valuable enough to outweigh the great evil that humans (and perhaps other creatures) cause by abusing it, and so valuable that God could not possibly create a better world without it. This to my mind is the biggest problem with the Free Will defence in all its forms. It seems pretty clear that free will by some definition is worth having; it also seems pretty clear that there are abstruse definitions of free will such that God cannot both create it and ensure it is used only for good. But these definitions don't coincide. One focal issue is whether God himself has free will, and has it in all the senses that are worth having. Most theist philosophers would say that God does have every valuable form of free will, but also that he is not logically free : there is no possible world in which God performs a morally evil act. But a little reflection shows there are infinitely many possible people who are similarly free but not logically free (so they also have exactly the same valuable free will that God does). And if God creates a world containing such people, and only such people, he necessarily ensure the existence of (valuable) free will but without any moral evil. So why doesn't he do that? See Quentin Smith for more on this. You may be aware of Smith's argument, and may be able to point me at an article where Plantinga has acknowledged and refuted it. If so, please do so.
1Legolan11y
I think this is an excellent summary. Having read John L. Mackie's free will argument and Plantinga's transworld depravity free will defense, I think that a theodicy based on free will won't be successful. Trying to define free will such that God can't ensure using his foreknowledge that everyone will act in a morally good way leads to some very odd definitions of free will that don't seem valuable at all, I think.
-2MugaSofer11y
Well sure. But that's a separate argument, isn't it? My point is that anyone making this argument isn't going to see Berry's argument as valid, for the same reason they are making this (flawed for other reasons) argument in the first place. Mind you, it's still an accurate statement and a useful observation in this context.
0BerryPick611y
It was my understanding that Alvin Plantinga mostly agreed that Mackie had him pinned with that response, so I'm calling you on this one.
-2MugaSofer11y
Most people making that argument, in my experience, believe that for free will to be truly "free" God cannot have decided (or even predicted, for some people) their actions in advance. Of course, these people are confused about the nature of free will. If you could show me a link to Plantinga conceding, that might help clear this up, but I'm guessing Mackie's argument (or something else) dissolved his confusion on the topic. If we had access to someone who actually believes this, we could test it ... anyone want to trawl through some theist corner of the web? Unless I'm misunderstanding your claim, of course; I don't believe I've actually read Mackie's work. I'm going to go see if I can find it free online now.
0BerryPick611y
Maybe I have gotten mixed up and it was Mackie who conceded to Plantinga? Unfortunately, I can't really check at the moment. Besides, I don't really disagree with what you said about most people who are making that particular argument.
-1MugaSofer11y
Fair enough. Well, having looked into it, it appears that Plantinga wasn't a compatibilist, while Mackie was. Their respective arguments assume their favored version of free will. Wikipedia thinks that Plantinga's arguments are generally agreed to be valid if* you grant incompatibilism, which is a big if; the LW consensus seems to be compatibilist for obvious reasons. I can't find anything on either of them conceding, I'm afraid.
0Decius11y
No, if I give the creator free will, he doesn't have to give anyone he creates the option. He chose to create the option or illusion, else he didn't exercise free will. It seems like you require a reason to suppress knowledge; are you choosing the lesser of two evils when you do so?
0MixedNuts11y
I meant free will as a moral concern. Nobody created G-d, so he doesn't necessarily have free will, though I think he does. He is, however, compelled to act morally (lest he vanish in a puff of logic). And morality requires giving people you create free will, much more than it requires preventing evil. (Don't ask me why.) Sure, I'm not Kant. And I'm saying G-d did too. People being able but not allowed to get knowledge suppresses knowledge, which is a little evil; people having knowledge makes them vulnerable to temptation, which is worse; people being unable to get knowledge deprives them of free will and also suppresses knowledge, which is even worse; not creating people in the first place is either the worst or impossible for some reason.
1Decius11y
I disagree with your premise that the actions taken by the entity which preceded all others are defined to be moral. Do you have any basis for that claim?
0MixedNuts11y
It says so in the book? (Pick any psalm.) I mean if we're going to disregard that claim we might as well disregard the claims about a bearded sky dude telling people to eat fruit. Using your phrasing, I'm defining G-d's actions as moral (whether this defines G-d or morality I leave up to you). The Bible claims that the first entity was G-d. (Okay, it doesn't really, but it's fanon.) It hardly seems fair to discount this entirely, when considering whether an apparently evil choice is due to evilness or to knowing more than you do about morality.
0Decius11y
Suppose that the writer of the book isn't moral. What would the text of the book say about the morality of the writer? Or we could assume that the writer of the book takes only moral actions, and from there try to construct which actions are moral. Clearly, one possibility is that it is moral to blatantly lie when writing the book, and that the genocide, torture, and mass murder didn't happen. That brings us back to the beginning again. The other possibility is too horrible for me to contemplate: That torture and murder are objectively the most moral things to do in noncontrived circumstances.
-1Eugine_Nier11y
Taboo "contrived".
1Decius11y
No. But I will specify the definition from Merriam-Webster and elaborate slightly: Contrive: To bring about with difficulty. Noncontrived circumstances are any circumstances that are not difficult to encounter. For example, the credible threat of a gigantic number of people being tortured to death if I don't torture one person to death is a contrived circumstance. 0% of exemplified situations requiring moral judgement are contrived.
-3MugaSofer11y
Taboo "difficult".
0Decius11y
Torture and murder are not the most moral things to do in 1.00000 00000 00000*10^2% of exemplified situations which require moral judgement. Are you going to taboo "torture" and "murder" now?
8Eugine_Nier11y
Well, that's clearly false. Your chances of having to kill a member of the secret police of an oppressive state are much more than 1/10^16, to say nothing of less clear cut examples.
1Decius11y
Do the actions of the secret police of an oppressive state constitute consent to violent methods? If so, they cannot be murdered in the moral sense, because they are combatants. If not, then it is immoral to kill them, even to prevent third parties from executing immoral acts. You don't get much less clear cut than asking questions about whether killing a combatant constitutes murder.
2A1987dM11y
Well, if you define “murder” as ‘killing someone you shouldn't’ then you should never murder anyone -- but that'd be a tautology and the interesting question would be how often killing someone would not be murder.
0Decius11y
"Murder" is roughly shorthand for "intentional nonconsensual interaction which results in the intended outcome of the death of a sentient." If the secret police break down my door, nothing done to them is nonconsensual.
5Eugine_Nier11y
Any half-way competent secret police wouldn't need to. You seem to have a very non-standard definition of "nonconsensual".
0Decius11y
I meant in the non-transitive sense. Being a combatant constitutes consent to be involved in the war. How is that non-standard?
0nshepperd11y
Being involved in the war isn't equivalent to being killed. I find it quite conceivable that I might want to involve myself in the war against, say, the babyeaters, without consenting to being killed by the babyeaters. I mean, ideally the war would go like this: we attack, babyeaters roll over and die, end. I'm not really sure what is the use of a definition of "consent" whereby involving myself in war causes me to automatically "consent" to being shot at. The whole point of fighting is that you think you ought to win.
4Nornagest11y
Well, I think consent sort of breaks down as a concept when you start considering all the situations where societies decide to get violent (or for that matter to involve themselves in sexuality; I'd rather not cite examples for fear of inciting color politics). So I'm not sure I can endorse the general form of this argument. In the specific case of warfare, though, the formalization of war that most modern governments have decided to bind themselves by does include consent on the part of combatants, in the form of the oath of enlistment (or of office, for officers). Here's the current version used by the US Army: Doesn't get much more explicit than that, and it certainly doesn't include an expectation of winning. Of course, a lot of governments still conscript their soldiers, and consent under that kind of duress is, to say the least, questionable; you can still justify it, but the most obvious ways of doing so require some social contract theory that I don't think I endorse.
4wedrifid11y
Indeed. Where the 'question' takes the form "Is this consent?" and the answer is "No, just no."
0Decius11y
Duress is a problematic issue- conscription without the social contract theory supporting it is immoral. So are most government policies, and I don't grok the social contract theory well enough to justify government in general.
1wedrifid11y
At the same time it should be obvious that there is something---pick the most appropriate word---that you have done by trying to kill something that changes the moral implications of the intended victim deciding to kill you first. This is the thing that we can clearly see that Decius is referring to. The 'consent' implied by your action here (and considered important to Decius) is obviously not directly consent to be shot at but rather consent to involvement in violent interactions with a relevant individual or group. For some reason of his own Decius has decided to grant you power such that a specific kind of consent is required from you before he kills you. The kind of consent required is up to Decius and his morals and the fact that you would not grant a different kind of consent ('consent to be killed') is not relevant to him.
0ArisKatsaris11y
"violence" perhaps or "aggression" or "acts of hostility". Not "consent". :-)
0Decius11y
Did all of the participants in the violent conflict voluntarily enter it? If so, then they have consented to the outcome.
1Eugine_Nier11y
Generally not, actually.
0Decius11y
Those who engage in an action in which not all participants enter of their own will is immoral. Yes, war is generally immoral in the modern era.
1nshepperd11y
A theory of morality that looks nice on paper but is completely wrong. In a war between Good and Evil, Good should win. It doesn't matter if Evil consented.
0Decius11y
You're following narrative logic there. Also, using the definitions given, anyone who unilaterally starts a war is Evil, and anyone who starts a war consents to it. It is logically impossible for Good to defeat Evil in a contest that Evil did not willingly choose to engage in.
0Eugine_Nier11y
What if Evil is actively engaged in say torturing others?
0Decius11y
Acts like constitute acts of the 'war' between Good and Evil that you are so eager to have. Have at them.
0nshepperd11y
Right, just like it's logically impossible for Good to declare war against Evil to prevent or stop Evil from doing bad things that aren't war.
2Decius11y
Exactly. You can't be Good and do immoral things. Also, abstractions don't take actions.
1A1987dM11y
Er, that kind-of includes asking a stranger for the time.
0Decius11y
Now we enter the realm of the social contract and implied consent.
2wedrifid11y
Decius, you may also be interested in the closely related post Ethical Inhibitions. It describes actions like, say, blatant murder, that could in principle (ie. in contrived circumstances) be actually the consequentialist right thing to do but that nevertheless you would never do anyway as a human since you are more likely to be biased and self-deceiving than to be correctly deciding murdering was right.
-3Decius11y
Correctly deciding that 2+2=3 is equally as likely as correctly deciding murdering was right.
0wedrifid11y
Ok, you're just wrong about that.
0Decius11y
In past trials, each outcome has occurred the same number of times.
2wedrifid11y
This could be true and you'd still be totally wrong about the equal likelihood.
-2ChristianKl11y
Murder is unlawful killing. If you are a citizen of the country you are within it's laws. If the oppressive country has a law against killing members of the secret police than it's murder.
4Decius11y
Murder (law) and murder (moral) are two different things; I was exclusively referring to murder (moral). I will clarify: There can be cases where murder (law) is either not immoral or morally required. There are also cases where an act which is murder (moral) is not illegal. My original point is that many of the actions of Jehovah constitute murder (moral).
2Eugine_Nier11y
What's your definition of murder (moral)?
0Decius11y
Roughly "intentional nonconsensual interaction which results in the intended outcome of the death of a sentient". To define how I use 'nonconsensual', I need to describe an entire ethics. Rough summary: Only every action which is performed without the consent of one or more sentient participant(s) is immoral. (Consent need not be explicit in all cases, especially trivial and critical cases; wearing a military uniform identifies an individual as a soldier, and constitutes clearly communicating consent to be involved in all military actions initiated by enemy soldiers.)
0BerryPick611y
This may be the word for which I run into definitional disputes most often. I'm glad you summed it up so well.
-3MugaSofer11y
I'm pretty sure they would say no, if asked. Just like, y'know, a non-secret policeman (the line is blurry.)
0Decius11y
Well, if I was wondering if a uniformed soldier was a combatant, I wouldn't ask them. Why would I ask the secret police if they are active participants in violence?
0Eugine_Nier11y
So cop-killing doesn't count as murder?
0Decius11y
Murder is not a superset of cop-killing.
-1MugaSofer11y
You said "consent". That usually means "permission". It's a nonstandard usage of the word, is all. But the point about the boundary between a cop and a soldier is actually a criticism, if not a huge one.
0Decius11y
Sometimes actions constitute consent, especially in particularly minor or particularly major cases.
-1MugaSofer11y
Again, shooting someone is not giving hem permission to shoot you. That's not to say it would be wrong to shoot back, necessarily. Are you intending to answer my criticism about the cop and the soldier?
0Decius11y
I don't see your criticism about the cop and the soldier; is it in a fork that I'm not following, or did I overlook it? Assuming that the social contract requires criminals to subject themselves to law enforcement: A member of society consents to be judged according to the laws of that society and treated appropriately. The criminal who violates their contract has already consented to the consequences of default, and that consent cannot be withdrawn. Secret police and soldiers act outside the law enforcement portion of the social contract. Does that cover your criticism?
2Eugine_Nier11y
Why?
0Decius11y
There's a little bit of 'because secret police don't officially exist' and a little bit of 'because soldiers aren't police'. Also, common language definitions fail pretty hard when strictly interpreting an implied social contract. There are cases where someone who is a soldier in one context is police in another, and probably some cases where a member of the unofficial police is also a member of the police.
0Eugine_Nier11y
Well, they generally do actually. They're called 'secret' because people don't know precisely what they're up to, or who is a member. You can replace them with regular police in my hypothetical if that helps.
1MugaSofer11y
A singleminded agent with my resources could place people in such a situation. I'm guessing the same is true of you. Kidnapping isn't hard, especially if you aren't too worried about eventually being caught, and murder is easy as long as the victim can't resist. "Difficult" is usually defined with regards to the speaker, and most people could arrange such a sadistic choice if they really wanted. They might be caught, but that's not really the point. If you mean that the odds of such a thing actually happening to you are low, "difficult" was probably the wrong choice of words; it certainly confused me. If I was uncertain what you meant by "torture" or "murder" I would certainly ask you for a definition, incidentally. (Also, refusal to taboo words is considered logically rude 'round these parts. Just FYI.)
0Decius11y
Consider the contrived situation usually used to show that consequentialism is flawed: There are ten patients in a hospital, all suffering from failure of a different organ they will die in a short time unless treated with an organ transplant, and if they receive a transplant then they will live a standard quality life. There is a healthy person who is a compatible match for all of those patients. He will live one standard quality life if left alone. Is it moral to refuse to forcibly and fatally harvest his organs to provide them to the larger number of patients? If I say that ten people dying is not a worse outcome than one person being killed by my hand, do you still think you can place someone with my values in a situation where they would believe that torture or murder is moral? Do you believe that consequentialism is objectively the accurate moral system?
1MugaSofer11y
Considering that dilemma becomes a lot easier if, say, I'm diverting a train through the one and away from the ten, I'm guessing there are other taboos there than just murder. Bodily integrity, perhaps? There IS something squicky about the notion of having surgery performed on you without you consent. Anyway, I was under the impression that you admitted that the correct reaction to a "sadistic choice" (kill him or I'll kill ten others) was murder; you merely claimed this was "difficult to encounter" and thus less worrying than the prospect that murder might be moral in day-to-day life. Which I agree with, I think.
0Decius11y
I think diverting the train is a much more complicated situation that hinges on factors normally omitted in the description and considered irrelevant by most. It could go any of three ways, depending on factors irrelevant to the number of deaths. (In many cases the murderous action has already been taken, and the decision is whether one or ten people are murdered by the murderer, and the action or inaction is taken with only the decider, the train, and the murderer as participants)
-2MugaSofer11y
Let's stipulate two scenarios, one in which the quandary is the result of a supervillain and one in which it was sheer bad luck.
-1Decius11y
Do I own the track, or am I designated by the person with ownership as having the authority to determine arbitrarily in what manner the junction may be operated? Do I have any prior agreement with regards to the operation of the junction, or any prior responsibility to protect lives at all costs? Absent prior agreements, if I have that authority to operate the track, it is neutral whether I choose to use it or not. If I were to own and control a hospital, I could arbitrarily refuse to support consensual fatal organ donations on my premises. If I have a prior agreement to save as many lives as possible at all costs, I must switch to follow that obligation, even if it means violating property rights. (Such an obligation also means that I have to assist with the forcible harvesting of organs). If I don't have the right to operate the junction according to my own arbitrary choice, I would be committing a small injustice on the owner of the junction by operating it, and the direct consequences of that action would also be mine to bear; if the one person who would be killed by my action does not agree to be, I would be murdering him in the moral sense, as opposed to allowing others to be killed. I suspect that my actual response to these contrived situations would be inconsistent; I would allow disease to kill ten people, but would cause a single event which would kill ten people without my trivial action to kill one instead (assuming no other choice existed). I prefer to believe that is a fault in my implementation of morality.
0MugaSofer11y
Nope. Oh, and the tracks join up after the people; you wont be sending a train careening off on the wrong track to crash into who knows what. I think you may be mistaking legality for morality. I'm not asking what you would have to do, I'm asking what you should do. Since prior agreements can mess with that, lets say the tracks are public property and anyone can change them, and you will not be punished for letting the people die. Murder has many definitions. Even if it would be "murder", which is the moral choice: to kill one or to let ten die? Could be. We would have to figure out why those seem different. But which of those choices is wrong? Are you saying that your analysis of the surgery leads you to change your mind about the train?
0Decius11y
The tracks are public property; walking on the tracks is then a known hazard. Switching the tracks is ethically neutral. The authority I was referencing was moral, not legal. I was actually saying that my actions in some contrived circumstances would differ from what I believe is moral. I am actually comfortable with that. I'm not sure if I would be comfortable with an AI which either always followed a strict morality, nor with one that sometimes deviated.
1[anonymous]11y
Blaming the individuals for walking on the tracks is simply assuming the not-least convenient world though. What if they were all tied up and placed upon the tracks by some evil individual (who is neither 1 of the people on the tracks nor the 1 you can push onto the tracks)?
1Decius11y
In retrospect, the known hazard is irrelevant.
-2MugaSofer11y
You still haven't answered what the correct choice is if a villain put them there. As for the rest ... bloody hell, mate. Have you got some complicated defense of those positions or are they intuitions? I'm guessing they're not intuitions.
0Decius11y
I don't think it would be relevant to the choice made in isolation what the prior events were. Moral authority is only a little bit complicated to my view, but it incorporates autonomy and property and overlaps with the very complicated and incomplete social contract theory, and I think it requires more work before it can be codified into something that can be followed. Frankly, I've tried to make sure that the conclusions follow reasonably from the premise, (all people are metaphysically equal) but it falls outside my ability to implement logic, and I suspect that it falls outside the purview of mathematics in any case. There are enough large jumps that I suspect I have more premises than I can explicate.
-2MugaSofer11y
Wait, would you say that while you are not obligated to save them, it would be better than leaving them die?
1Decius11y
I decline to make value judgements beyond obligatory/permissible/forbidden, unless you can provide the necessary and sufficient conditions for one result to be better than another.
-2MugaSofer11y
I ask because I checked and the standard response is that it would not be obligatory to save them, but it would be good.
0Decius11y
I don't have a general model for why actions are subrogatory or superogatory.
-2MugaSofer11y
I think a good way to think of this result is that leaving the switch on "kill ten people" nets 0 points, moving it from "ten" to "one" nets, say, 9 points, and moving it from "one" to "ten" loses you 9 points. I have no model that accounts for the surgery problem without crude patches like "violates bodily integrity = always bad." Humans in general seem to have difficulties with "sacred values"; how many dollars is it worth to save one life? How many hours (years?) of torture?
0Decius11y
I think that "violates bodily autonomy"=bad is a better core rule than "increases QALYs"=good.
6ArisKatsaris11y
I think I'm mostly a rule utilitarian, so I certainly understand the worth of rules... ... but that kind of rule really leaves ambiguous how to define any possible exceptions. Let's say that you see a baby about to start chewing on broken glass -- the vast majority would say that it's obligatory to stop it from doing so, of the remainder most would say that it's at least permissible to stop the baby from chewing on broken glass. But if we set up "violates bodily autonomy"=bad as an absolute rule, we are actually morally forbidden to physically prevent the baby from doing such. So what are the exceptions? If it's an issue of competence (the adult has a far better understanding of what chewing glass would do, and therefore has the right to ignore the baby's rights to bodily autonomy), then a super-intelligent AI would have the same relationship in comparison to us...
0Decius11y
Does the theoretical baby have the faculties to meaningfully enter an agreement, or to meaningfully consent to be stopped from doing harmful things? If so, then the baby is not an active moral agent, and is not considered sentient under the strict interpretation. Once the baby becomes an active moral agent, they have the right to choose for themselves if they wish to chew broken glass. Under the loose interpretation, the childcare contract obligates the caretaker to protect, educate and provide for the child and grants the caretaker permission from the child to do anything required to fulfill that role. What general rules do you follow that require or permit stopping a baby from chewing on broken glass, but prohibit forcibly stopping adults from engaging in unhealthy habits?
0[anonymous]11y
The former is an ethical injunction, the latter is a utility approximation. They are not directly comparable.
0MixedNuts11y
We do loads of things that violate children's bodily autonomy.
0Decius11y
And in doing so, we assert that children are not active moral agents. See also paternalism.
0MixedNuts11y
Yeah but... that's false. Which doesn't make the rule bad, heuristics are allowed to apply only in certain domains, but a "core rule" shouldn't fail for over 15% of the population. "Sentient things that are able to argue about harm, justice and fairness are moral agents" isn't a weaker rule than "Violating bodily autonomy is bad".
2Decius11y
Do you believe that the ability to understand the likely consequences of actions is a requirement for an entity to be an active moral agent?
0MugaSofer11y
Well, it's less well-defined if nothing else. It's also less general; QALYs enfold a lot of other values, so by maximizing them you get stuff like giving people happy, fulfilled lives and not shooting 'em in the head. It just doesn't enfold all our values,so you get occasional glitches, like killing people and selling their organs in certain contrived situations.
1Decius11y
Values also differ among even perfectly rational individuals. There are some who would say that killing people for their organs is the only moral choice in certain contrived situations, and reasonable people can mutually disagree on the subject.
-2MugaSofer11y
And your point is...?
-1Decius11y
I'm trying to develop a system which follows logically from easily-defended principles, instead of one that is simply a restatement of personal values.
0MugaSofer11y
Seems legit. Could you give me an example of "easily-defended principals", as opposed to "restatements of personal values"?
0Decius11y
"No sentient individual or group of sentient beings is metaphysically privileged over any group or individual."
-1MugaSofer11y
That seems true, but the "should" in there would seem to label it a "personal value". At least, if I've understood you correctly.
0Decius11y
I'm completely sure that I didn't understand what you meant by that.
-1MugaSofer11y
Damn. Ok, try this: where did you get that statement from, if not an extrapolation of your personal values?
0Decius11y
In addition to being a restatement of personal values, I think that it is an easily-defended principle. It can be attacked and defeated with a single valid reason why one person or group is intrinsically better or worse than any other, and evidence for a lack of such reason is evidence for that statement.
-2MugaSofer11y
It seems to me that an agent ccould coherently value people with purple eyes more than people with orange eyes. And it's arguments would not move you, nor yours it. And if you were magically convinced that the other was right, it would be near-impossible for you to defend their position; for all the agent might claim that we can never be certain if eyes are truly orange, or merely a yellowish red, and you might claim that purple eyed folk are rare, and should be preserved for diversity's sake. Am I wrong, or is this not the argument you're making? I suspect at least one of us is confused.
1Decius11y
I didn't claim that I had a universally compelling principle. I can say that someone who embodied the position that eye color granted special privilege would end up opposed to me.
-2MugaSofer11y
Oh, that makes sense. You're trying to extrapolate your own ethics. Yeah, that's how morality is usually discussed here, I was just confused by the terminology.
0Decius11y
... with the goal of reaching a point that is likely to be agreed on by as many people as possible, and then discussion the implications of that point.
-1MugaSofer11y
Shouldn't your goal be to extrapolate your ethics, then help everyone who shares those ethics (ie humans) extrapolate theirs?
1Decius11y
Why 'should' my goal be anything? What interest is served by causing all people who share my ethics (which need not include all members of the genus Homo) to extrapolate their ethics?
2BerryPick611y
Extrapolating other people's Ethics may or may not help you satisfy your own extrapolated goals, so I think that may be the only metric by which you can judge whether or not you 'should' do it. No?
3A1987dM11y
Then there might be superrational considerations, whereby if you helped people sufficiently like you to extrapolate their goals, they would (sensu Gary Drescher, Good and Real) help you to extrapolate yours.
-1MugaSofer11y
Well, people are going to extrapolate their ethics regardless. You should try to help them avoid mistakes, such as "blowing up buildings is a good thing" or "lynching black people is OK". Well sure. Psychopaths, if nothing else.
0Decius11y
Why do I care if they make mistakes that are not local to me? I get much better security return on investment by locally preventing violence against me and my concerns, because I have to handle several orders of magnitude fewer people.
0MugaSofer11y
Perhaps I haven't made myself clear. Their mistakes will, by definition, violate your (shared) ethics. For example, if they are mistakenly modelling black people as subhuman apes, and you both value human life, then their lynching blacks may never affect you - but it would be a nonpreferred outcome, under your utility function.
0Decius11y
My utility function is separate from my ethics. There's no reason why everything I want happens to be something which is moral. It is a coincidence that murder is both unethical and disadvantageous to me, not tautological.
1Peterdjones11y
You may have some non-ethical values, as many do, but if your ethics are no part of your values, you are never going to act on them.
0Decius11y
I am considering taking the position that I follow my ethics irrationally; that I prefer decisions which are ethical even if the outcome is worse. I know that position will not be taken well here, but it seems more accurate than the position that I value my ethics as terminal values.
0MugaSofer11y
No, I'm not saying it would inconvenience you, I'm saying it would be a Bad Thing, which you, as a human (I assume,) would get negative utility from. This is true for all agents whose utility function is over the universe, not eg their own experiences. Thus, say, a paperclipper should warn other paperclippers against inadvertently producing staples.
2Decius11y
Projecting your values onto my utility function will not lead to good conclusions. I don't believe that there is a universal, or even local, moral imperative to prevent death. I don't value a universe in which more QALYs have elapsed over a universe in which fewer QALYs have elapsed; I also believe that entropy in every isolated system will monotonically increase. Ethics is a set of local rules which is mostly irrelevant to preference functions; I leave it to each individual to determine how much they value ethical decisions.
-1MugaSofer11y
That wasn't a conclusion; that was an example, albeit one I believe to be true. If there is anything you value, even if you are not experiencing it directly, then it is instrumentally good for you to help others with the same ethics to understand they value it too. ... oh. It's pretty much a given around here that human values extrapolate to value life, so if we build an FAI and switch it on then we'll all live forever, and in the mean time we should sign up for cryonics. So I assumed that, as a poster here, you already held this position unless you specifically stated otherwise. I would be interested in discussing your views (known as "deathism" hereabouts) some other time, although this is probably not the time (or place, for that matter.) I assume you think everyone here would agree with you, if they extrapolated their preferences correctly - have you considered a top-level post on the topic? (Or even a sequence, if the inferential distance is too great.) Once again, I'm only talking about what is ethically desirable here. Furthermore, I am only talking about agents which share your values; it is obviously not desirable to help a babyeater understand that it really, terminally cares about eating babies if I value said babies' lives. (Could you tell me something you do value? Suffering or happiness or something? Human life is really useful for examples of this; if you don't value it just assume I'm talking about some agent that does, one of Azimov's robots or something.) [EDIT: typos.]
0Decius11y
I began to question whether I intrinsically value freedom of agents other than me as a result of this conversation. I will probably not have an answer very quickly, mostly because I have to disentangle my belief that anyone who would deny freedom to others is mortally opposed to me. And partially because I am (safely) in a condition of impaired mental state due to local cultural tradition. I'll point out that "human" has a technical definition of "members of the genus homo" and includes species which are not even homo sapiens. If you wish to reference a different subset of entities, use a different term. I like 'sentients' or 'people' for a nonspecific group of people that qualify as active or passive moral agents (respectively).
0TheOtherDave11y
Why?
0Decius11y
Because the borogroves are mimsy.
0TheOtherDave11y
There's a big difference between a term that has no reliable meaning, and a term that has two reliable meanings one of which is a technical definition. I understand why I should avoid using the former (which seems to be the point of your boojum), but your original comment related to the latter.
0Decius11y
What are the necessary and sufficient conditions to be a human in the non-taxonomical sense? The original confusion was where I was wrongfully assumed to be a human in that sense, and I never even thought to wonder if there was a meaning of 'human' that didn't include at least all typical adult homo sapiens.
-2MugaSofer11y
Well, you can have more than one terminal value (or term in your utility function, whatever.) Furthermore, it seems to me that "freedom" is desirable, to a certain degree, as an instrumental value of our ethics - after all, we are not perfect reasoners, and to impose our uncertain opinion on other reasoners, of similar intelligence, who reached different conclusions, seem rather risky (for the same reason we wouldn't want to simply write our own values directly into an AI - not that we don't want the AI to share our values, but that we are not skilled enough to transcribe them perfectly. "Human" has many definitions. In this case, I was referring to, shall we say, typical humans - no psychopaths or neanderthals included. I trust that was clear? If not, "human values" has a pretty standard meaning round here anyway.
3Decius11y
Freedom does have instrumental value; however, lack of coercion is an intrinsic thing in my ethics, in addition to the instrumental value. I don't think that I will ever be able to codify my ethics accurately in Loglan or an equivalent, but there is a lot of room for improvement in my ability to explain it to other sentient beings. I was unaware that the "immortalist" value system was assumed to be the LW default; I thought that "human value system" referred to a different default value system.
-3MugaSofer11y
The "immortalist" value system is an approximaton of the "human value system", and is generally considered a good one round here.
-3Decius11y
It's nowhere near the default value system I encounter in meatspace. It's also not the one that's being followed by anyone with two fully functional lungs and kidneys. (Aside: that might be a good question to add to the next annual poll) I don't think mass murder in the present day is ethically required, even if by doing so would be a net benefit. Even if free choice hastens the extinction of humanity, there is no person or group with the authority to restrict free choice.
0wedrifid11y
I don't believe you. Immortalists can have two fully functional lungs and kidneys. I think you are referring to something else.
-1Decius11y
Go ahead- consider a value function over the universe, that values human life and doesn't privilege any one individual, and ask that function if the marginal inconvenience and expense of donating a lung and a kidney are greater than the expected benefit.
-6MugaSofer11y
1wedrifid11y
"The kind of obscure technical exceptions that wedrifid will immediately think of the moment someone goes and makes a fully general claim about something that is almost true but requires qualifiers or gentler language."
0A1987dM11y
That doesn't help if wedrifid won't think of as obscure and noncentral exceptions with certain questions as with others. (IIRC, EY in his matching questions on OKCupid when asked whether someone is ever obliged to sex, he picked No and commented something like ‘unless I agreed to have sex with you for money, and already took the money’, but when asked whether someone should ever use a nuclear weapon (or something like that), he picked Yes and commented with a way more improbable example than that.)
-2Eugine_Nier11y
That's not helpful, especially in context.
2wedrifid11y
Apart from implying different subjective preferences to mine when it comes to conversation this claim is actually objectively false as a description of reality. The 'taboo!' demand in this context was itself a borderline (in as much as it isn't actually the salient feature that needs elaboration or challenge and the meaning should be plain to most non disingenuous readers). But assuming there was any doubt at all about what 'contrived' meant in the first place my response would, in fact, help make it clear through illustration what kind of thing 'contrived' was being used to represent (which was basically the literal meaning of the word). Your response indicates that the "Taboo contrived!" move may have had some specific rhetorical intent that you don't want disrupted. If so, by all means state it. (I am likely to have more sympathy for whatever your actual rejection of decius's comment is than for your complaint here.)
-2Eugine_Nier11y
Decius considered the possibility that In order to address this possibility, I need to know what Decius considers "contrived" and not just what the central example of a contrived circumstance is. In any case, part of my point was to force Decius to think more clearly about under what circumstances are torture and killing justified rather than simply throwing all the examples he knows in the box labeled "contrived".
4TimS11y
However Decius answers, he probably violates the local don't-discuss-politics norm. By contrast, your coyness makes it appear that you haven't done so. In short, it appears to me that you already know Decius' position well enough to continue the discussion if you wanted to. Your invocation of the taboo-your-words convention appears like it isn't your true rejection.
0A1987dM11y
I'd take “contrived circumstances” to mean ‘circumstances so rare that the supermajority of people alive have never found themselves in one of them’.
-1MugaSofer11y
Presumably the creator did want the trees, he just didn't want humans using it. I always got the impression that the trees were used by God(and angels?), who at the point the story was written was less the abstract creator of modern times and more the (a?) jealous tribal god of the early Hebrews (for example, he was physically present in the GOE.) Isn't there a line about how humanity must never reach the TOL because they would become (like) gods? EDIT: Seriously? Knowledge of any kind?
0Decius11y
Yes. Suppressing knowledge of any kind is evil. It's not the only thing which is evil, and acts are not necessarily good because they also disseminate knowledge.
0[anonymous]11y
This has interesting implications. Other more evil things (like lots of people dieing) can sometimes be prevented by doing a less evil thing (like suppressing knowledge). For example, the code for an AI that would foom, but does not have friendliness guarantees, is a prime candidate for suppression. So saying that something is evil is not the last word on whether or not it should be done, or how it's doers should be judged.
0Decius11y
Code, instructions, and many things that can be expressed as information are only incidentally knowledge. There's nothing evil about writing a program and then deleting it; there is something evil about passing a law which prohibits programming from being taught, because programmers might create an unfriendly AI.
0Eugine_Nier11y
Well, the knowledge from the tree appears to also have been knowledge of this kind.
0Decius11y
I draw comparisons between the serpent offering the apple, the Titan Prometheus, and Odin sacrificing his eye. Do you think that the comparison of those knowledge myths is unfair?
-2MugaSofer11y
Fair enough. Humans do appear to value truth. Of course, if acts that conceal knowledge can be good because of other factors, then this: ... is still valid.
4Irgy11y
This is a classic case of fighting the wrong battle against theism. The classic theist defence is to define away every meaningful aspect of God, piece by piece, until the question of God's existance is about as meaningful as asking "do you believe in the axiom of choice?". Then, after you've failed to disprove their now untestable (and therefore meaningless) theory, they consider themselves victorious and get back to reading the bible. It's this part that's the weak link. The idea that the bible tells us something about God (and therefore by extension morality and truth) is a testable and debatable hypothesis, whereas God's existance can be defined away into something that is not. People can say "morality is God's will" all they like and I'll just tell them "butterflies are schmetterlinge". It's when they say "morality is in the bible" that you can start asking some pertinent questions. To mix my metaphors, I'll start believing when someone actually physically breaks a ball into pieces and reconstructs them into two balls of the same original size, but until I really see something like that actually happen it's all just navel gazing.
6Eliezer Yudkowsky11y
Sure, and to the extent that somebody answers that way, or for that matter runs away from the question, instead of doing that thing where they actually teach you in Jewish elementary school that Abraham being willing to slaughter Isaac for God was like the greatest thing ever and made him deserve to be patriarch of the Jewish people, I will be all like, "Oh, so under whatever name, and for whatever reason, you don't want to slaughter children - I'll drink to that and be friends with you, even if the two of us think we have different metaethics justifying it". I wasn't claiming that accepting the first horn of the dilemma was endorsed by all theists or a necessary implication of theism - but of course, the rejectance of that horn is very standard atheism.

I don't think it's incompatible. You're supposed to really trust the guy because he's literally made of morality, so if he tells you something that sounds immoral (and you're not, like, psychotic) of course you assume that it's moral and the error is on your side. Most of the time you don't get direct exceptional divine commands, so you don't want to kill any kids. Wouldn't you kill the kid if an AI you knew to be Friendly, smart, and well-informed told you "I can't tell you why right now, but it's really important that you kill that kid"?

If your objection is that Mr. Orders-multiple-genocides hasn't shown that kind of evidence he's morally good, well, I got nuthin'.

You're supposed to really trust the guy because he's literally made of morality, so if he tells you something that sounds immoral (and you're not, like, psychotic) of course you assume that it's moral and the error is on your side.

What we have is an inconsistent set of four assertions:

  1. Killing my son is immoral.
  2. The Voice In My Head wants me to kill my son.
  3. The Voice In My Head is God.
  4. God would never want someone to perform an immoral act.

At least one of these has to be rejected. Abraham (provisionally) rejects 1; once God announces 'J/K,' he updates in favor of rejecting 2, on the grounds that God didn't really want him to kill his son, though the Voice really was God.

The problem with this is that rejecting 1 assumes that my confidence in my foundational moral principles (e.g., 'thou shalt not murder, self!') is weaker than my confidence in the conjunction of:

  • 3 (how do I know this Voice is God? the conjunction of 1,2,4 is powerful evidence against 3),
  • 2 (maybe I misheard, misinterpreted, or am misremembering the Voice?),
  • and 4.

But it's hard to believe that I'm more confident in the divinity of a certain class of Voices than in my moral axioms, especially if my confidenc... (read more)

7MixedNuts11y
Well, deities should make themselves clear enough that (2) is very likely (maybe the voice is pulling your leg, but it wants you to at least get started on the son-killing). (3) is also near-certain because you've had chats with this voice for decades, about moving and having kids and changing your name and whether the voice should destroy a city. So this correctly tests whether you believe (4) more than (1) - whether your trust in G-d is greater than your confidence in your object-level judgement. You're right that it's not clear why Abraham believes or should believe (4). His culture told him so and the guy has mostly done nice things for him and his wife, and promised nice things then delivered, but this hardly justifies blind faith. (Then again I've trusted people on flimsier grounds, if with lower stakes.) G-d seems very big on trust so it makes sense that he'd select the president of his fan club according to that criterion, and reinforce the trust with "look, you trusted me even though you expected it to suck, and it didn't suck".

Well, if we're shifting from our idealized post-Protestant-Reformation Abraham to the original Abraham-of-Genesis folk hero, then we should probably bracket all this Medieval talk about God's omnibenevolence and omnipotence. The Yahweh of Genesis is described as being unable to do certain things, as lacking certain items of knowledge, and as making mistakes. Shall not the judge of all the Earth do right?

As Genesis presents the story, the relevant question doesn't seem to be 'Does my moral obligation to obey God outweigh my moral obligation to protect my son?' Nor is it 'Does my confidence in my moral intuitions outweigh my confidence in God's moral intuitions plus my understanding of God's commands?' Rather, the question is: 'Do I care more about obeying God than about my most beloved possession?' Notice there's nothing moral at stake here at all; it's purely a question of weighing loyalties and desires, of weighing the amount I trust God's promises and respect God's authority against the amount of utility (love, happiness) I assign to my son.

The moral rights of the son, and the duties of the father, are not on the table; what's at issue is whether Abraham's such a good soldier-servant that he's willing to give up his most cherished possessions (which just happen to be sentient persons). Replace 'God' with 'Satan' and you get the same fealty calculation on Abraham's part, since God's authority, power, and honesty, not his beneficence, are what Abraham has faith in.

-1MugaSofer11y
If we're going to talk about what actually happened, as opposed to a particular interpretation, the answer is "probably nothing". Because it's probably a metaphor for the Hebrews abandoning human sacrifice. Just wanted to put that out there. It's been bugging me.
0BerryPick611y
[citation needed]
1MugaSofer11y
More like [original research?]. I was under the impression that's the closest thing to a "standard" interpretation, but it could as easily have been my local priest's pet theory. You've gotta admit it makes sense, though.

To my knowledge, this is a common theory, although I don't know whether it's standard. There are a number of references in the Tanakh to human sacrifice, and even if the early Jews didn't practice (and had no cultural memory of having once practiced) human sacrifice, its presence as a known phenomenon in the Levant could have motivated the story. I can imagine several reasons:

  • (a) The writer was worried about human sacrifice, and wanted a narrative basis for forbidding it.

  • (b) The writer wasn't worried about actual human sacrifice, but wanted to clearly distinguish his community from Those People who do child sacrifice.

  • (c) The writer didn't just want to show a difference between Jews and human-sacrifice groups, but wanted to show that Jews were at least as badass. Being willing to sacrifice humans is an especially striking and impressive sign of devotion to a deity, so a binding-of-Isaac-style story serves to indicate that the Founding Figure (and, by implicit metonymy, the group as a whole, or its exemplars) is willing to give proof of that level of devotion, but is explicitly not required to do so by the god. This is an obvious win-win -- we don't have to actually kill anybod

... (read more)
0BerryPick611y
I've never heard it before. After nearly a decade of studying the Old Testament, I finally decided very little of it makes sense a few years ago.
0MugaSofer11y
Huh. Well, it depends what you mean by "sense", I guess.
2Alejandro111y
The problem has the same structure for MixedNuts' analogy of the FAI replacing the Voice. Suppose you program the AI to compute explicitly the logical structure "morality" that EY is talking about, and it tells you to kill a child. You could think you made a mistake in the program (analogous to rejecting your 3), or that you are misunderstanding the AI or hallucinating it (rejecting 2). And in fact for most conjunctions of reasonable empirical assumptions, it would be more rational to take any of these options than to go ahead and kill the child. Likewise, sensible religionists agree that if someone hears voices in their head telling them to kill children, they shouldn't do it. Some of they might say however that Abraham's position was unique, that he had especially good reasons (unspecified) to accept 2 and 3, and that for him killing the child is the right decision. In the same way, maybe an AI programmer with very strong evidence for the analogies for 2 and 3 should go ahead and kill the child. (What if the AI has computed that the child will grow up to be Hitler?)
4Rob Bensinger11y
A few religious thinkers (Kierkegaard) don't think Abraham's position was completely unique, and do think we should obey certain Voices without adequate evidence for 4, perhaps even without adequate evidence for 3. But these are outlier theories, and certainly don't reflect the intuitions of most religious believers, who pay more lip service to belief-in-belief than actual service-service to belief-in-belief. I think an analogous AI set-up would be: 1. Killing my son is immoral. 2. The monitor reads 'Kill your son.' 3. The monitor's display perfectly reflects the decisions of the AI I programmed. 4. I successfully programmed the AI to be perfectly moral. What you call rejecting 3 is closer to rejecting 4, since it concerns my confidence that the AI is moral, not my confidence that the AI I programmed is the same as the entity outputting 'Kill your son.'
0Alejandro111y
I disagree, because I think the analogy between the (4) of each case should go this way: (4a) Analysis of "morality" as equivalent to a logical structure extrapolatable from by brain state (plus other things) and that an AI can in principle compute <==> (4b) Analysis of "morality" as equivalent to a logical structure embodied in a unique perfect entity called "God" These are both metaethical theories, a matter of philosophy. Then the analogy between (3) in each case goes: (3a) This AI in front of me is accurately programmed to compute morality and display what I ought to do <==> (3b) This voice I hear is the voice of God telling me what I ought to do. (3a) includes both your 3 and your 4, which can be put together as they are both empirical beliefs that, jointly, are related to the philosophical theory (4a) as the empirical belief (3b) is related to the philosophical theory (4b).
0Rob Bensinger11y
Makes sense. I was being deliberately vague about (4) because I wasn't committing myself to a particular view of why Abraham is confident in God's morality. If we're going with the scholastic, analytical, logical-pinpointing approach, then your framework is more useful. Though in that case even talking about 'God' or a particular AI may be misleading; what 4 then is really asserting is just that morality is a coherent concept, and can generate decision procedures. Your 3 is then the empirical claim that a particular being in the world embodies this concept of a perfect moral agent. My original thought simply took your 4 for granted (if there is no such concept, then what are we even talking about?), and broke the empirical claim up into multiple parts. This is important for the Abraham case, because my version of 3 is the premise most atheists reject, whereas there is no particular reason for the atheists to reject my version of 4 (or yours).
0Alejandro111y
We are mostly in agreement about the general picture, but just to keep the conversation going... I don't think (4) is so trivial or that (4a) and (4b) can be equated. For the first, there are other metaethical theories that I think wouldn't agree with the common content of (4a) and (4b). These include relativism, error theory, Moorean non-naturalism, and perhaps some naive naturalisms ("the good just is pleasure/happiness/etc, end of story"). For the second, I was thinking of (4a) as embedded in the global naturalistic, reductionistic philosophical picture that EY is elaborating and that is broadly accepted in LW, and of (4b) as embedded in the global Scholastic worldview (the most steelmanned version I know of religion). Obviously there are many differences between the two philosophies, both in the conceptual structures used and in very general factual beliefs (which as a Quinean I see as intertwined and inseparable at the most global level). In particular, I intended (4b) to include the claim that this perfect entity embodying morality actually exists as a concrete being (and, implicitly,that it has the other omni-properties attributed to God). Clearly atheists wouldn't agree with any of this.
2MugaSofer11y
I can't speak for Jewish elementary school, but surely believing PA (even when, intuitively, the result seems flatly wrong or nonsensical) would be a good example to hold up before students of mathematics? The Monty Hall problem seems like a good example of this.

I read this post with a growing sense of unease. The pie example appears to treat "fair" as a 1-place word, but I don't see any reason to suppose it would be. (I note my disquiet that we are both linking to that article; and my worry about how confused this post seems to me.)

The standard atheist reply is tremendously unsatisfying; it appeals to intuition and assumes what it's trying to prove!

My resolution of Euthryphro is "the moral is the practical." A predictable consequence of evolution is that people have moral intuitions, that those intuitions reflect their ancestral environment, and that those intuitions can be variable. Where would I find mercy, justice, or duty? Cognitive algorithms and concepts inside minds.

This article reads like you're trying to move your stone tablet from your head into the world of logic, where it can be as universal as the concept of primes. It's not clear to me why you're embarking on that particular project.

The example of elegance seems like it points the other way. If your sense of elegance is admittedly subjective, why are we supposing a Platonic form of elegance out in the world of logic? Isn't this basically the error where o... (read more)

The pie example appears to treat "fair" as a 1-place word

'Beautiful' needs 2 places because our concept of beauty admits of perceptual variation. 'Fairness' does not grammatically need an 'according to whom?' argument place, because our concept of fairness is not observer-relative. You could introduce a function that takes in a person X who associates a definition with 'fairness,' takes in a situation Y, and asks whether X would call Y 'fair.' But this would be a function for 'What does the spoken syllable FAIR denote in a linguistic community?', not a function for 'What is fair?' If we applied this demand generally, 'beautiful' would became 3-place ('what objects X would some agent Y say some agent Z finds 'beautiful'?'), as would logical terms like 'plus' ('how would some agent X perform the operation X calls "addition" on values Y and Z?'), and indeed all linguistic acts.

intuitions reflect their ancestral environment, and [...] those intuitions can be variable.

Yes, but a given intuition cannot vary limitlessly, because there are limits to what we would consider to fall under the same idea of 'fairness.' Different people may use the spoken syllables FAI... (read more)

3Vaniver11y
What? It seems to me that fairness and beauty are equally subjective, and the intuition that says "but my sense of fairness is objectively correct!" is the same intuition that says "but my sense of beauty is objectively correct!" I agree that we can logically pinpoint any specific concept; to use the pie example, Yancy uses the concept of "splitting windfalls equally by weight" and Zaire uses the concept of "splitting windfalls equally by desire." What I disagree with is the proposition that there is this well-defined and objective concept of "fair" that, in the given situation, points to "splitting windfalls equally by weight." One could put forward the axiom that "splitting windfalls equally by weight is fair", just like one can put forward the axiom that "zero is not the successor of any number," but we are no closer to that axiom having any decision-making weight; it is just a model, and for it to be used it needs to be a useful and appropriate model.
1nshepperd11y
"Fair", quoted, is a word. You don't think it's plausible that in English "fair" could refer to splitting windfalls equally by weight? (Or rather to something a bit more complicated that comes out to splitting windfalls equally by weight in the situation of the three travellers and the pie.)
2Vaniver11y
I agree that someone could mean "splitting windfalls equally by weight" when they say "fair." I further submit that words can be ambiguous, and someone else could mean "splitting windfalls equally by desire" when they say "fair." In such a case, where the word seems to adding more heat than light, I would scrap it and go with the more precise phrases.
0Rob Bensinger11y
I don't know what you mean by 'subjective.' But perhaps there is a (completely non-denoting) concept of Objective Beauty in addition to the Subjective Beauty ('in the eye of the beholder') I'm discussing, and we're talking past each other about the two. So let's pick a simpler example. 'Delicious' is clearly two-place, and ordinary English-language speakers routinely consider it two-place; we sometimes elide the 'delicious for whom?' by assuming 'for ordinary humans,' but it would be controversial to claim that speaking of deliciousness automatically commits you to a context-independent property of Intrinsic Objective Tastiness. Now, it sounded like you were claiming that fairness is subjective in much the same way as deliciousness; no claim about fairness is saturated unless it includes an argument place for the evaluater. But this seems to be false simply given how people conceive of 'fair' and 'delicious'. People don't think there's an implicit 'fairness-relative-to-a-judge-thereof' when we speak of 'fairness,' or at least it don't think it in the transparent way they think of 'deliciousness' as always being 'deliciousness-relative-to-a-taster.' ('Beauty,' perhaps, is an ambiguous case straddling these two categories.) So is there some different sense in which fairness is 'subjective'? What is this other sense? Are you claiming that Eliezer lacks any well-defined concept he's calling 'fairness'? Or are you claiming that most English-speakers don't have Eliezer's well-defined fairness in mind when they themselves use the word 'fair,' thus making Eliezer guilty of equivocation? People argue about how best to define a term all the time, but we don't generally conclude from this that any reasoning one proceeds to carry out once one has stipulated a definition for the controversial term is for that reason alone 'subjective.' There have been a number of controversies in the history of mathematics — places where people's intuitions simply could not be reconciled by
9Vaniver11y
That the judgments of "fair" or "beautiful" don't come from a universal source, but from a particular entity. I have copious evidence that what I consider "beautiful" is different from what some other people consider "beautiful;" I have copious evidence that what I consider "fair" is different from what some other people consider "fair." It is clear to me that delicious is two-place, but it seems to me that people have to learn that it is two-place, and evidence that it is two-place is often surprising and potentially disgusting. Someone who has not learned through proverbs and experience that "beauty is in the eye of the beholder" and "there's no accounting for taste" would expect that everyone thinks the same things are beautiful and tasty. There are several asymmetries between them. Deliciousness generally affects one person, and knowing that it varies allows specialization and gains from trade (my apple for your banana!). Fairness generally requires at least two people to be involved, and acknowledging that your concept of fairness does not bind the other person puts you at a disadvantage. Compare Xannon's compromise to Yancy's hardlining. People thinking that something is objective is not evidence that it is actually objective. Indeed, we have plenty of counterevidence in all the times that people argue over what is fair. No? I'm arguing that Eliezer::Fair may be well-defined, but that he has put forward no reason that will convince Zaire that Zaire::Fair should become Eliezer::Fair, just like he has put forward no reason why Zaire::Favorite Color should become Eliezer::Favorite Color. There are lots of possible geometries out there, and mathematicians can productively discuss any set of non-contradictory axioms. But only a narrow subset of those geometries correspond well with the universe that we actually live in; physicists put serious effort into understanding those, and the rest are curiosities. (I think that also answers your last two questions, but
4Peterdjones11y
But there is little upshot to people having differnt notions of beauty, since people can arrange their own environents to suit their own aesthetics. However, resources have to be apportioned one way or another. So we need, and have discussion about how to do things fairly. (Public architecture is a bit of an exception to what I said about beauty, but lo and behold, we have debates at that too).
1Rob Bensinger11y
I don't understand what this means. To my knowledge, the only things that exist are particulars. I have copious evidence that others disagree with me about ¬¬P being equivalent to P. And I have copious evidence that others disagree with me about the Earth's being more than 6,000 years old. Does this imply that my belief in Double Negation Elimination and in the Earth's antiquity is 'subjective'? If not, then what extra premises are you suppressing? Well, sure. But, barring innate knowledge, people have to learn everything at some point. 3-year-olds lack a theory of mind; and those with a new theory of mind may not yet understand that 'beautiful' and 'delicious' are observer-relative. But that on its own gives us no way to conclude that 'fairness' is observer-relative. After all, not everything that we start off thinking is 'objective' later turns out to be 'subjective.' And even if 'fairness' were observer-relative, there have to be constraints on what can qualify as 'fairness.' Fairness is not equivalent to 'whatever anyone decides to use the word "fairness" to mean,' as Eliezer rightly pointed out. Even relativists don't tend to think that 'purple toaster' and 'equitable distribution of resources' are equally legitimate and plausible semantic candidates for the word 'fairness.' That's not true. Deliciousness, like fairness, affects everyone. For instance, my roommate is affected by which foods I find delicious; it changes where she ends up going to eat. Perhaps you meant something else. You'll have to be much more precise. The entire game when it comes to as tricky a dichotomy as 'objective/subjective' is just: Be precise. The dichotomy will reveal its secrets and deceptions only if we taboo our way into its heart. What's fair varies from person to person too, because different people, for instance, put different amounts of work into their activities. And knowing about what's fair can certainly help in trade! Does not "bind" the other person? Fairness is no
0Vaniver11y
Your choice of logical system and your belief in an old Earth reside in your mind, and that you believe them only provides me rather weak evidence that they are beliefs I should hold. (I do hold both of those beliefs, but because of other evidence.) It is not clear to me that attaching the label of "subjective" or "objective" would materially improve my description. When I write the word "generally," I mean it as a qualifier that acknowledges many objections could be raised that do not materially alter the point. Generally, at restaurants, you and your roommate are not required to eat the same meal, and the effects of, say, the unpleasant-to-her smell of your meal are smaller than the effects of the pleasant-to-her taste of her meal. Of course there are meals you could eat and restaurants you could choose where that is not case, but the asymmetry between the impact of your tastes on your roommate and the impact of your sense of fairness on your roommate remains in the general case. Consider: The priest walks by the beggar without looking. The beggar calls up to him, "Matthew 25!" Matthew is not the priest's name, but still he stops, decides, and then gives the beggar his sack lunch. The practical use of moral and ethical systems is as a guide to decision-making. Moral systems typically specialize in guiding decisions in a way that increases the positive benefit to others and decreases the negative cost to them. Moral and ethical systems are only relevant insofar as they are used to make decisions. The space we live in corresponds well (obviously, not perfectly) with Euclidean space, and so it receives significant attention from physicists. The space we live in doesn't correspond well with the Poincare disk hyperbolic geometry; the most likely place a non-mathematician has seen it is M.C. Escher. Models of Euclidean geometry exist in minds, and one person's model of it may not agree with another's, but there is currently an established definition (i.e. blueprin
0Rob Bensinger11y
I didn't ask you whether my believing them gives you evidence to think they're objectively true. I asked whether other people not believing them gives me evidence to think they're merely subjective. If not, then you can't use disagreement over 'fairness,' on its own, to demonstrate the subjectivity of fairness. So is it your view that "The Earth is over 6,000 years old" is neither subjective nor objective? Why not say, then, that claims about fairness are neither subjective nor objective? That's great for you, but the fact that you believe you could meet all the objections to your assertion (if you didn't, why would you be asserting it?) doesn't give me much reason to believe what you're saying. Generally. Just like the effects of an unfair situation I'm in (generally!) impact me more than they impact my roommate. Again, it's not clear what work you're trying to do, either when you note similarities between deliciousness and fairness or when you note dissimilarities. You've provided us with no principled way to treat 'fairness' any different than we treat the old-Earth hypothesis or 0.999... = 1; and you've given us no principled way to sort the 'subjective' claims from the 'objective' ones, nor explained why it matters which of the categories we put moral concepts under. So your claim is that fairness is subjective because it has an impact on people's decision-making? Don't objective things have an impact on people's decision-making too? You didn't answer my question. Is Euclidean geometry subjective? I'm just trying to get you to make your criticism of Eliezer explicit, but every time we come close to you giving a taboo'd version of what troubles you about treating fairness in the same way we treat geometry, you shift to a different topic without explaining its relevance to the 'subjectivity!' charge. What does it take for a blueprint to be "estabished"? Does a certain percentage of the human race have to agree on the same definition of the term? Or a certai
0Vaniver11y
That is because it's not clear to me that "subjective" evokes the same concept for each of us, and so I'd rather taboo subjective and talk about object-level differences than classifications. That looks like my objection. I want to make clear that the linguistic claim is amplified by the relevance to decision-making. It is of little relevance to me whether others classify my actions as blegg or not; it is of great relevance to me whether others classify my actions as fair or not, and the same is true for enough people that putting forth an algorithm and stating "fairness points to this algorithm" is regarded as a power grab. Saying "geometry with the parallel postulate is Euclidean" is not regarded a power grab because the axioms and their consequences are useful or useless independent of the label ascribed to them. That communication is simply text; with labeling something 'fair,' there is the decision-making subtext "you should do this."
-1Rob Bensinger11y
Originally, it sounded like you were making one of these claims: * (a) There is no semantic candidate for the word 'fairness.' It's just noise attempting to goad people into behaving a certain way. * (b) No semantic candidate for 'fairness' can be rendered logically precise. * (c) Even if we precisify a candidate meaning for 'fairness,' it won't really be a Logical Idea, because the subject matter of morality is intrinsically less logicy than the subject matter of mathematics. * (d) Metaphysically speaking, there are quantitative universals or Forms in Plato's Heaven, but there are no moral universals or Forms. * (e) No semantic candidate for 'fairness' can avoid including an argument place for a judge-of-fairness. * (f) No logically precise semantic candidate for 'fairness' can avoid including such an argument place. All of these claims are implausible. But now it sounds like you're instead just claiming: (g) There are multiple semantic candidates for 'fairness,' and I'm not totally clear on which one Eliezer is talking about. So I'd appreciate it if he were a bit clearer about what he means. If (g) is all you meant to argue this whole time, then we're in agreement. Specificity is of course a virtue. Mathematical definitions are a power grab just as moral definitions are; the only difference is that people care more about the moral power-grabs than about the mathematical ones. Mathematical authorities assert their dominance, assert their right to participate in establishing General Mathematical Practice regarding definitions, inference rules, etc., every time they endorse one usage as opposed to another. It's only because their authority goes relatively unchallenged that we don't see foundational disputes over mathematical definitions as often as we see foundational disputes over moral definitions. Each constrains practice, after all.
2Wei Dai11y
This inspired me to write Morality Isn't Logical, and I'd be interested to know what you think.
0HalMorris11y
Very few mathematical definitions are about General Mathematical Practice. Euclidean and Riemannian (or projective) geometry are in perfect peaceful coexistence, and in general, new forms of mathematics expand the territory rather than fight over an existing patch of territory.
-2Rob Bensinger11y
I think you underestimate the generality of my claim. (Perhaps the phrase 'power grab' is poorly chosen.) Relatively egalitarian power grabs are still power grabs, inasmuch as they use the weight of consensus and tradition to marginalize non-egalitarian views. There is no proof that both geometries are equally 'true' or 'correct' or 'legitimate' or 'valid;' so we could equally well have decided that only Euclidean geometry is correct; or that only project geometry is; or that neither is. There is no proof that one of the latter options is superior; but nor is there a proof that one is inferior. It's a pragmatic and/or arbitrary choice, and settling such decisions depends on an initially minority viewpoint coming to exert its consensus-establishing authority over majority practice. Egalitarianism is about General Mathematical Practice. (And sometimes it's very clearly sociological, not logical, in character; for instance, the desire to treat conventional and intuitionistic systems as equally correct but semantically disjoint is a fine way to calm down human disagreement, but as a form of mathematical realism it is unmotivated, and in fact leads to paradox.) That depends a great deal on how coarse-grainedly you instantiate "forms". Mathematical results get overturned all the time; not just in the form of entire fields being rejected or revised from the ground up (like the infinitesimal calculus), and not just in the discovery of internal errors in proofs past, but in the rejection of definitions and axioms for a given discourse.
7HalMorris11y
I'm just a 2 year math Ph.D. program drop-out from 35 years ago, but I got quite a different take on it. As I experienced it, most mathematics is like "Let X be a G-space where G-space is defined as having ". and then you might spend years proving whatever those axioms imply, and defining umpteen specializations of a G-space, like a G2-space which has , and teasing out and proving the consequences of having those axioms. At no point do you say these axioms are true - that's an older, non-mathematical use of the word "axiom" as something (supposedly) self-evidently true. You just say if these axioms are true for X, then this and this and this follows. Mathematicians simply don't say that the axioms of Euclidean geometry are true. It is all about, "if an object (which is a purely mental construct) has these properties, then it most have these other properties. By the "infinitesimal calculus", being overturned, I assume you mean dropping the use of infinitesimals in favor of delta-epsilon type definitions in calculus/real analysis, it's not such a good illustration that revision from the ground up happens all the time, since really, that goes back to the late 19th century, and I really don't think such things do happen all the time though another big redefining project happened in the early 20th century.
0A1987dM11y
ZFC set theory? Peano arithmetics?
0Vaniver11y
I don't think (g) is quite right. It is clear to me which candidate Eliezer is putting forward (in this case, at least): splitting windfalls equally by weight. I think the closest of the ones you suggest is (e). Trying to put my view of my claim in similar terms to your other options, I think I would go with something like "We can create logically precise candidates for fairness, but this leaves undone the work of making those candidates relevant to decision-makers," with the motivation that the reason to have moral systems / concepts like 'fairness' is because they are relevant to decision-makers. That is, we can imagine numbers being 'prime' without a mathematician looking at them and judging them prime, but we should not imagine piles of pebbles occurring in prime numbers without some force that shifts pebbles based on their pile size.
0Rob Bensinger11y
(a) Do you think Eliezer is trying to make his terms 'relevant to decision-makers' in the requisite sense? (b) Why would adding an argument place for 'the person judging the situation as fair' help make fairness more relevant to decision-makers? I don't believe in a fundamental physical force that calculates how many pebbles are in a pile, and adds or subtracts a pebble based specifically on that fact. But I do believe that pebbles can occur in piles of 3, and that 3 is a prime number. Similarly, I don't believe in a magical Moral Force, but I do believe that people care about equitable distributions of resources, and that 'fairness' is a perfectly good word for picking out that property we care about. I still don't see any reason to add an argument place; and if there were a need for a second argument place, I still don't see why an analogous argument wouldn't force us to add a third argument to 'beautiful,' so that some third party can judge whether another person is perceiving something as beautiful. (Indeed, if we took this requirement seriously, it would produce an infinite regress, making no language expressible.)
1Vaniver11y
Do you see why a 2-place beauty would be more relevant than a 1-place beauty? I was unclear; I didn't mean "that some piles will have prime membership" but that "most or all piles of pebbles will have prime membership." Generally?
0Rob Bensinger11y
Relevant to what? I would have no objection to a one-place beautyₐ, where 'beautyₐ' is an exhaustively physically specifiable idea like 'producing feelings of net aesthetic pleasure when encountered by most human beings'. I would also have no objection to a two-place beauty₂, where 'beauty₂' means 'aesthetically appealing to some person X.' Neither one of these is more logically legitimate than the other, and neither one is less logically legitimate than the other. The only reason we prefer beauty₂ over beautyₐ is that it's (a) more user-friendly to calculate, or that it's (b) a more plausible candidate for what ordinary English language users mean when they say the word 'beauty.' What I want to see is an argument for precisely what the analogous property 'fairness₂' would look like, and why this is a more useful or more semantically plausible candidate for our word 'fairness' than a one-place 'fairnessₐ' would be. Otherwise your argument will just as easily make 'plus' three-place ('addition-according-to-someone') or 'bird' two-place ('being-a-bird-according-to-someone'). This is not only impractical, but dangerous, since it confuses us into thinking that what we want when we speak of 'objectivity' is not specificity, but merely making explicit reference to some subject. As though mathematics would become more 'objective,' and not less, if we were to relativize it to a specific mathematician or community of mathematicians. So is your worry that having a one-place 'fairness' predicate will make people think that most situations are fair, or that there's a physically real fundamental law of karma promoting fairness? In general, yes, generally.
0Vaniver11y
To decision-makers. I think I'm going to refer you to this post again. Having a beautyₐ which implicitly rather than explicitly restricts itself to humans runs the risk of being applied where its not applicable. Precision in language aids precision in thought. I think I'm also going to bow out of the conversation at this point; we have both typed a lot and it's not clear that much communication has gone on, to the point that I don't expect extending this thread is a good use of either of our times.
2A1987dM11y
Grammatically, neither does “beautiful”. “Alice is beautiful” is a perfectly grammatical English sentence.
0Rob Bensinger11y
Yes. Clearly I was being unclear. Just as saying "Eating broccoli is good" I think assumes a tacit answer to "Good for whom?" and/or "Good for what?", saying "Hamburgers are delicious" assumes a tacit "Delicious to whom?", even if the answer is "To everyone!". I have a hard time understanding what it means to visualize a possible world where everything is delicious and there are no organisms or sentients. I think of 'beauty' the same way, but perhaps not everyone does; and if some people think of 'fairness' as intrinsically -- because of the concept itself, and not just because of our metaphysical commitments or dialectical goals -- demanding an implicit argument place for a 'judge of fairness,' I'd like to hear more about why. Or is this just a metaphysical argument, not a conceptual one?
6Peterdjones11y
How do you avoid prudent predation
2dspeyer11y
I think the author of that piece needs to learn the concept of precommitment. Precommitting to one-box is not at all the same as believing that one-boxing is the dominant strategy in the general newcomb problem. Likewise, precommitting not to engage in prudent predation is not a matter of holding a counterfactual belief, but of taking a positive-expected-utility action.
0Vaniver11y
Are there moral systems used by humans that avoid prudent predation, and are not outcompeted by moral systems used by humans that make use of prudent predation? I will note that the type of predation that is prudent has varied significantly over time, and correspondingly, so have moral intuitions. Further altering the structure of society will again alter the sort of predation that is prudent, and so one can seek to restructure society so disliked behavior is less prudent and liked behavior is more prudent.
0Peterdjones11y
I find it hard to make sense of that. I don't think people go in for morality for selfish gain, and the very idea may be incoherent. Maybe. I don't see what your point is. If the moral is not the practical, and if PP is wrong, that would not imply morality is timeless, and vice versa.
0Vaniver11y
The claim is that moral intuitions exist because they were selected for, and they must have been selected for because they increased reproductive fitness. Similarly, we should expect moral behavior to the degree that morality is more rewarding than immorality. (The picture is muddied by there being both genetic and memetic evolution, but the basic idea survives.)
1Peterdjones11y
But morality isn't just moral intuitions. It includes "eat fish on friday" That doens't follow. Fitness-enahncing and gene-spreading behaviour don;t have to reward the organism concerned. What't the reward for self sacrifice? that's a considerable understatement.
0Vaniver11y
Sure. We should expect such rules to be followed to the degree that they are prudent. There are several; kin selection, reciprocal altruism, and so on. In some cases, self-sacrifice is the result of a parasitic relationship. (Kin selection appears to have a memetic analog as well, but I'm not familiar with work that develops that concept rigorously, and distinguishes it from normal alliance behaviors; it might just be a subset of that.)
0Peterdjones11y
Again, I have no idea what you mean. Morality does not predict self-centered prudence, since it enjoins self-sacrifice, and evolution doenst predict self-centered prudence in all cases. It is not selfishly prudent for bees to to defend their colony, or for male praying mantises to mate. Rewards for whom?
0Strange711y
If you pass on the idea that self-sacrifice is virtuous, in a persuasive sort of way (such as by believing it yourself), you're marginally more likely to enjoy the benefits of having someone willing to sacrifice their own interests nearby when you particularly need such a person. Of course, sometimes that meme kills you. Some people are born with sickle-cell anemia and never get the opportunity to benefit from malaria resistance; evolution doesn't play nice.
2nshepperd11y
It is exactly executing an adaption. No "just" about it though. An AI programmed to maximise paperclips is motivated by increasing the number of paperclips. It's executing its program.
1Vaniver11y
I had this post in mind. I see no reason to link behavior that 'seems moral' to the internal sensation of motivation by those terminal values, and if we're not talking about introspection about decision-making, then why are we using the word motivation? This post seems to be discussing a particular brand of moral reasoning- basically, deliberative utilitarian judgments- which seems like a rather incomplete picture of human morality as a whole, and it seems like it's just sweeping under the rug the problem of where values come from in the first place. I should make clear that first he has to describe what values are before he can describe where values come from, but if it's an incomplete description of values, that can cause problems down the line.
1SebastianGarren11y
Vaniver, I really appreciate the rigor you are bringing to this discussion. The OP struck me as very deliberative-utilitarian as well. If we want to account (or propagate) for a shared human morality, than certainly, it must be rational. But it seems to me, that the long history of searching for a rational-basis-for-morality clearly points away from the well trodden ground of this utilitarianism. From Plato and Aristotle to the Enlightenment until Nietzsche (especially to the present day), it seems the project of accounting for morality as though it were an inherent attribute of humanity, expressible through axioms and predetermined by the universe, is a bunk and, perhaps even, an irrational project. Morality, I think can only be shared, if you have a shared goal for winning life. A complete description of values requires a discussion on what makes life worth living and what is a good life, or more simply goals. Without the tools to determine and rationalize what are good goals for me, I will never be able to make a map of morality and choose the values and virtues relevant to me on my quest. Does that jive?
0Vaniver11y
Yes. I would note there is often a meaningful difference between individual and social virtues. You and I could share expectations about only our conduct when we interact and not the other's private conduct. It is easy to imagine people spending more effort on inducing their neighbors to keep their lawns pretty than their dishes pretty, for example.

Yay, I think we've finished the prerequisites to prerequisites, and started the prerequisites!

Stimulating as always! I have a criticism to make of the use made of the term 'rigid designation'.

Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself [...]

What philosophers of language ordinarily mean by calling a term a rigid designator is not that, considered purely syntactically, it intrinsically refers to anything. The property of being a rigid designator is something which can be possessed by an expression in use in a particular language-system. The distinction is between expressions-in-use whose reference we let vary across counterfactual scenarios (or 'possible worlds'), e.g. 'The first person to climb Everest', and those whose reference remains stable, e.g. 'George Washington', 'The sum of two and two'.

There is some controversy over how to apply the rigid/non-rigid distinction to general terms like 'fair' (or predicates like 'is fair') - cf. Scott Soames' book Beyond Rigidity - but I think the natural thing to say is that 'is fair' is rigid, since it is used to attribute the same property across counterfactual scenarios, in contrast with a predicate like 'possesses my favourite property'.

Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself - if somebody thinks of Euclidean geometry when you utter the sound "num-berz" they're not doing anything false, they're associating the sound to a different logical thingy. It's not about words with intrinsically rigid referential power, it's that the words are window dressing on the underlying entities.

I just wanted to agree with Tristanhaze here that this usage strikes me as non-standard. I want to put this in my own words so that Tristanhaze/Eliezer/others can correct me if I've got the wrong end of the stick.

If something is a rigid designator it means that it refers to the same thing in all possible worlds. To say it's non-rigid is to say it refers to different things in some possible worlds to others. This has nothing to do with whether different language users that use the phrase must always be referring to the same thing. So George Washington may be a rigid designator in that it refers to the same person in all possible world... (read more)

8RichardChappell11y
Correct. Eliezer has misunderstood rigid designation here.
0Eliezer Yudkowsky11y
So does that mean this: ...is your real claim here, independent of any points about language use? If so, I think I would just straightforwardly modify my paragraph above to say that my statements are not trying to talk about language use or human brains / desires, albeit that desire is both an optimization target of, and a quotation of, morality.
7RichardChappell11y
I'm not sure what you have in mind here. We need to distinguish (i) the referent of a concept from (ii) its reference-fixing "sense" or functional role. The way I understood your view, the reference-fixing story for moral terms involves our (idealized) desires. But the referent is "rigid" in the sense that it's picking out the content of our desires: the thing that actually fills the functional role, rather than the role-property itself. Since our desires typically aren't themselves about our desires, so it will turn out, on this story, that morality is not "about" desires. It's about "love, friendship," and all that jazz. But there's a story to be told about how our moral concepts came to pick out these particular worldly properties. And that's where desires come in (as I understand your view). Our moral concepts pick out these particular properties because they're the contents of our idealized desires. But that's not to say that therefore morality is "really" just about fulfilling any old desires. For that would be to neglect the part that rigid designation, and the distinction between reference and reference-fixing, plays in this story. Does that capture your view? To further clarify: the point of appealing to "rigid designation" is just to explain how desires could play a reference-fixing role without being any part of the referent of moral talk (or what it is "about"). Isn't that what you're after? Or do you have some other reference-fixing story in mind?

This all does sound good to me; but, is there a way to say the above while tabooing "reference" and avoiding talk of things "referring" to other things? Reference isn't ontologically basic, so what does it reduce to?

Basically, the main part that would worry me is a phrase like, "there's a story to be told about how our moral concepts came to pick out these particular worldly properties" which sounds on its face like, "There's a story to be told about how successorship came to pick out the natural numbers" whereas what I'd want to say is, "Of course, there's a story to be told about how moral concepts came to have the power to move us" or "There's a story to be told about how our brains came to reflect numbers".

6Qiaochu_Yuan11y
Can you give an example of a rigid designator (edit: that isn't purely mathematical / logical)? I don't understand how the concept is even coherent right now. "Issues of transworld identity" seem to be central and I don't know why you're sweeping them under the rug. More precisely, I do not understand how one goes about identifying objects in different possible worlds even in principle. I think that intuitions about this procedure are likely to be flawed because people do not consider possible worlds that are sufficiently different.
4crazy8811y
Okay, so three things are worth clarifying up front. First, this isn't my area of expertise so anything I have to say about the matter should be taken with a pinch of salt. Second, this is a complex issue and really would require 2 or 3 sequences of material to properly outline so I wouldn't read too much into the fact that my brief comment doesn't present a substantive outline of the issue. Third, I have no settled views on the issues of rigid designators, nor am I trying to argue for a substantive position on the matter so I'm not deliberately sweeping anything under the rug (my aim was to distinguish Eliezer's use of the phrase rigid designator from the standard usage and doing so doesn't require discussion of transworld identity: Eliezer was using it to refer to issues relating to different people whereas philosophers use it to refer to issues relating to a single person - or at least that's the simplified story that captures the crucial idea). All that said, I'll try to answer your question. First, it might help to think of rigid designators as cases where the thing to be identified isn't simply to be identified with its broad role in the world. So "the inventor of bifocals" is the person that plays a certain role in the world - the role of inventing bifocals. So "the inventor of bifocals" is not a rigid designator. So the heuristic for identifying rigid designators is that they can't just be identified by their role in the world. Given this, what are some examples of rigid designators? Well, the answer to this question will depend on who you ask. A lot of people, following Putnam would take "water" (and other natural kind terms) to be a rigid designator. On this view, "Water" rigidly refers to H2O, regardless of whether H20 plays the "water" role in some other possible world. So imagine a possible world where some other substance, XYZ, falls from the sky, sakes thirst, fill rivers and so on (that is, XYZ fills the water role in this possible world). On the r
4Qiaochu_Yuan11y
Thank you for the clarification. I agree that the question of what a possible world is is an important one, but the answer seems obvious to me: possible worlds are things that live inside the minds of agents (e.g. humans). Water is one of the examples I considered and found incoherent. Once you start considering possible worlds with different laws of physics, it's extremely unclear to me in what sense you can identify types of particles in one world with particles in another type of world. I could imagine doing this by making intuitive identifications step by step along "paths" in the space of possible worlds, but then it's unclear to me how you could guarantee that the identifications you get this way are independent of the choice of path (this idea is motivated by a basic phenomenon in algebraic topology and complex analysis).
0crazy8811y
As I said, these are complex issues. Yes, but almost everyone agrees with this (or at least, almost all views on possible worlds can be interpreted this way even if they can also be interpreted as claims about the existence of abstract - non-concrete - objects). There are a variety of different things that possible worlds can be even given the assumption that they exist in people's heads (almost all the disagreement about what possible worlds are is disagreement within this category rather than between this category and something else). Two things: first, the claim that "water" rigidly designates H2O doesn't imply that it must exist in all possible worlds - just that if "water" exists in a possible world then it is H2O. So if we can't identify the same particles in different worlds then this just means that water exists in almost no worlds (perhaps only in our own world). However, the view that we can't identify the same particles in other worlds is a radical one and would be a strong sign that the account of possible worlds appealed to falls short (after all, possible worlds are supposed to be about what is possible and surely there are possibilities that revolve around the particles existing in our world - ie. surely it's possible that I now be holding a glass of H2O. If your account of possible worlds can't cope with this possibility it seems to not be a very useful account of possible worlds). Further, how hard it is to identify sameness of particles across possible worlds will depend on how you take them to be "constructed" - if they are constructed by stipulation, ie. "consider the world where I am holding a glass of H2O" then it is very easy to get sameness of particles. I'm not saying there's not room for your criticisms but for them to hold requires substantial metaphysical work showing why your account, and only your account, of possible worlds works and hence that your conclusions hold.
2Qiaochu_Yuan11y
Okay. I think what I'm actually trying to say is that what constitutes a rigid designator, among other things, seems to depend very strongly on the resolution at which you examine possible worlds. When you say the phrase "imagine the possible world in which I have a glass of water in my hand" to a human, that human knows what you mean because by default humans only model the physical world at a resolution where it is easy to imagine making that intervention and only that intervention. When you say that phrase to an AI which is modeling the world at a much higher resolution, the AI does not know how to do what you ask because you haven't given it enough information. How did the glass of water get there? What happened to the air molecules that it displaced? Etc.
0crazy8811y
Okay, perhaps I can have another go at this. First thing to note, possible worlds can't be specified at different levels of detail. When doing so we are either specifying partial possible worlds or sets of possible worlds. As rigid designation is a claim about worlds, it can't be relative to the level of detail utilised as it only applies to things specified at one level of detail. Second, you still seem to be treating possible worlds as concrete things rather than something in the head (or, at least, making substantive assumptions about possible worlds and relying on these to make claims about possible worlds generally). Let's take possible worlds to be sets of propositions and truth values. In this case there's no reason to find transworld identity puzzling. H2O exists in this world just if a relevant proposition is true (like, "I am holding a glass of H2O"). There's also no room for this transworld identity to be relative to a context. Whether these things are puzzling depends on your account of possible worlds and it seems like if you're account of possible worlds can't account for transworld identity it can't do the work required of possible worlds and so it is open to the challenge that it should be abandoned in favour of some other account. Third, it's important to distinguish questions about the way worlds are from questions about how they can be specified. It's an interesting question how we should specify individual possible worlds and another interesting question whether we often do so or whether we normally specify sets of possible worlds instead. However, difficulties with specification do not undermine the concept of a rigid designator. Fourth, even if it were a relative matter whether H2O exists in a world this wouldn't undermine the concept of rigid designation. Rigid designation would simply imply that if this were the case then it would also be a relative matter whether water existed in that world. The summary of what I'm trying to get at is t
2Qiaochu_Yuan11y
I think that these two desires are contradictory. Part of what I'm trying to say is that it's a highly nontrivial problem which propositions are even meaningful, let alone true, if you specify possible worlds at a sufficiently high level of detail. For example, at an extremely high level of detail, you might specify a possible world by specifying a set of laws of physics together with an initial condition for the universe. This kind of specification of a possible world doesn't automatically allow you to interpret intuitive referents like "I," so the meaning of a statement like "I am holding a glass of water" is extremely unclear. How do you know what things are rigid designators if you neither know how to specify possible worlds nor how to determine what's in them?
0crazy8811y
I think this conversation is now well into the territory of diminishing return so I'll leave it at that.
0crazy8811y
I think this is getting past the point that I can useful contribute further though I will note that the vast literature on the topic has dealt with this sort of issue in detail (though I don't know it well enough to comment in detail). Saying that, I'll make one final contribution and then leave it at that: I suspect that you've misunderstood the idea of a rigid designator if you think it depends on the resolution at which you examine possible worlds. To say that something is a rigid designator is to say that it refers to the same thing in all possible worlds (note that this is a fact about language use). So to say that "water" rigidly denotes H2O is just to say that when we use the word water to refer to something in some possible world, we are talking about H2O. Issues of how precisely the details of the world are filled in are not relevant to this issue (for example, it doesn't matter what happens to the air molecules - this has no impact on the issue of rigid designation). The point you raise is an interesting one about how we specify possible worlds but not, to my knowledge, one that's relevant to rigid designation. But beyond that I don't think I have anything more of use to contribute (simple because we've exhausted my meagre knowledge of the topic)...
-2MugaSofer11y
I assume the AI could concoct some plausible explanation(s).
5Eliezer Yudkowsky11y
I'd like to say "sure" and then delete that paragraph, but then somebody else in the comments will say that my essay is just talking about a rigid-designation theory of morality. I mean, that's the comment I've gotten multiple times previously. Anyone got a good idea for resolving this?
6crazy8811y
You may have resolved this now by talking to Richard (who knows more about this than me) but, in case you haven't, I'll have a shot at it. First, the distinction: Richard is using rigid designation to talk about how a single person evaluates counterfactual scenarios, whereas you seem to be taking it as a comment about how different people use the same word. Second, relevance: Richard's usage allow you to respond to an objection. The objection asks you to consider the counterfactual situation where you desire to murder people and says murder must now be right so the theory is extremely subjective. You can respond that "right" is a rigid designator so it is still right to not murder in this counterfactual situation (though your counterpart here will use the word "right" differently). Suggestion: perhaps edit the paragraph so as to discuss either this objection and defence or outline why the rigid designator view so characterised is not your view.

Here's my understanding of the post:

Consider two types of possible FAI designs. A Type 1 FAI has its values coded as a logical function from the time it's turned on, either a standard utility function, or all the information needed to run a simulation of a human that is eventually supposed to provide such a function, or something like that. A Type 2 FAI tries to learn its values from its inputs. For example it might be programmed to seek out a nearby human, scan their brain, and then try to extract a utility function from the scan, going to a controlled shutdown if it encounters any errors in this process. A human is more like a Type 1 FAI than a Type 2 FAI so it doesn't matter that there is no God/Stone Tablet out in the universe that we can extract morality from.

If this is fair, I have two objections:

  1. When humans are sufficiently young they are surely more like a Type 2 FAI than a Type 1 FAI. We're obviously not born with Frankena's list of terminal values. Maybe one can argue that an adult human is like a Type 2 FAI that has completed its value learning process and has "locked down" its utility function and won't change its values or go into shutdown even if it su

... (read more)