Humans often mistakenly think they face trolley problems when they really don't. This has implications for people who believe they face a trolley problem, without directly changing what constitutes a good response by someone who actually faces a trolley problem.

I'm having trouble inferring your point here... The contrast between 'those who are dreaming think they are awake, but those who are awake know they are awake' and "I refuse to extend this reply to myself, because the epistemic state you ask me to imagine, can only exist among other kinds of... (read more)

I used some rhetoric, like using the word 'evil' and not rounding 3^^^3+1 to just 3^^^3, to highlight how the people whose fate you're choosing might perceive both the problem and how you're thinking about the problem. It's just... I have a similar reaction when thinking about a human self-righteously proclaiming 'Kill them all, God will know His own.', but I feel like it's useful that a part of me always kicks in and says I'm probably doing the same damn thing in ways that are just less obvious. But maybe it is not useful.

Umm, didn't you (non-trollishly) advocate indiscriminately murdering anyone and everyone accused of heresy as long as it's the Catholic Church doing it?

4lessdazed9yI have an uncommon relationship with the dream world, as I remember many dreams every night. I often dream within a dream, I might do this more often than most because dreams occupy a larger portion of my thoughts than they do in others, or I might just be remembering those dreams more than most do. When I wake up within a dream, I often think I am awake. On the other hand, sometimes in the middle of dreams I know I am dreaming. Usually it's not something I think about while asleep or awake. I also have hypnopompic sleep paralysis, and sometimes wake up thinking I am dead. This is like the inverse of sleep walking - the mind wakes up some time before the body can move. I'm not exactly sure if one breathes during this period or not, but it's certainly impossible to consciously breathe and one immediately knows that one cannot, so if one does not think oneself already dead (which for me is rare) one thinks one will suffocate soon. Confabulating something physical blocking the mouth or constricting the trunk can occur. It's actually not as bad to think one is dead, because then one is (sometimes) pleasantly surprised by the presence of an afterlife (even if movement is at least temporarily impossible) and one does not panic about dying - at least I don't. So all in all I'd say I have less respect for intuitions like that than most do.
1lessdazed9yOne point is that I feel very unconfused. That is, not only do I not feel confused now, I once felt confused and experienced what I thought was confusion lifting and being replaced by understanding. Which one, if just one, criteria for usefulness are you using here? It is useful for the human to have pain receptors, but there is negative utility in being vulnerable to torture (and not just from one's personal perspective). Surely you don't expect that even the most useful intuition is always right? This is similar to the Bin Laden point above, that the most justified and net-good action will almost certainly have negative consequences. I'm willing to call your intuition useful if it often saves you from being misled, and its score on any particular case is not too important in its overall value. However, its score on any particular case is indicative of how it would do in similar cases. If it has a short track record and it fails this test, we have excellent reason to believe it is a poorly tuned intuition because we know little other than how it did on this hypothetical, though its poor performance on this hypothetical should never be considered a significant factor in what makes it generally out of step with moral dilemmas regardless. This is analogous to getting cable ratings from only a few tracked boxes: we think many millions watched a show because many of the thousands tracked did, but do not think those thousand constitute a substantial portion of the audience.

Metacontrarian Metaethics

by Will_Newsome 2 min read20th May 201175 comments


Designed to gauge responses to some parts of the planned “Noticing confusion about meta-ethics” sequence, which should intertwine with or be absorbed by Lukeprog’s meta-ethics sequence at some point.

Disclaimer: I am going to leave out many relevant details. If you want, you can bring them up in the comments, but in general meta-ethics is still very confusing and thus we could list relevant details all day and still be confused. There are a lot of subtle themes and distinctions that have thus far been completely ignored by everyone, as far as I can tell.

Problem 1: Torture versus specks

Imagine you’re at a Less Wrong meetup when out of nowhere Eliezer Yudkowsky proposes his torture versus dust specks problem. Years of bullet-biting make this a trivial dilemma for any good philosopher, but suddenly you have a seizure during which you vividly recall all of those history lessons where you learned about the horrible things people do when they feel justified in being blatantly evil because of some abstract moral theory that is at best an approximation of sane morality and at worst an obviously anti-epistemic spiral of moral rationalization. Temporarily humbled, you decide to think about the problem a little longer:

"Considering I am deciding the fate of 3^^^3+1 people, I should perhaps not immediately assert my speculative and controversial meta-ethics. Instead, perhaps I should use the averaged meta-ethics of the 3^^^3+1 people I am deciding for, since it is probable that they have preferences that implicitly cover edge cases such as this, and disregarding the meta-ethical preferences of 3^^^3+1 people is certainly one of the most blatantly immoral things one can do. After all, even if they never learn anything about this decision taking place, people are allowed to have preferences about it. But... that the majority of people believe something doesn’t make it right, and that the majority of people prefer something doesn’t make it right either. If I expect that these 3^^^3+1 people are mostly wrong about morality and would not reflectively endorse their implicit preferences being used in this decision instead of my explicitly reasoned and reflected upon preferences, then I should just go with mine, even if I am knowingly arrogantly blatantly disregarding the current preferences of 3^^^3 currently-alive-and-and-not-just-hypothetical people in doing so and thus causing negative utility many, many, many times more severe than the 3^^^3 units of negative utility I was trying to avert. I may be willing to accept this sacrifice, but I should at least admit that what I am doing largely ignores their current preferences, and there is some chance it is wrong upon reflection regardless, for though I am wiser than those 3^^^3+1 people, I notice that I too am confused."

You hesitantly give your answer and continue to ponder the analogies to Eliezer’s document “CEV”, and this whole business about “extrapolation”...

(Thinking of people as having coherent non-contradictory preferences is very misleadingly wrong, not taking into account preferences at gradient levels of organization is probably wrong, not thinking of typical human preferences as implicitly preferring to update in various ways is maybe wrong (i.e. failing to see preferences as processes embedded in time is probably wrong), et cetera, but I have to start somewhere and this is already glossing over way too much.)

Bonus problem 1: Taking trolleys seriously

"...Wait, considering how unlikely this scenario is, if I ever actually did end up in it then that would probably mean I was in some perverse simulation set up by empirical meta-ethicists with powerful computers, in which case they might use my decision as part of a propaganda campaign meant to somehow discredit consequentialist reasoning or maybe deontological reasoning, or maybe they'd use it for some other reason entirely, but at any rate that sure complicates the problem...” (HT: Steve Rayhawk)