In this case: letting bias and/or intellectual laziness dominate your decision-making process.
I'm not arguing that people shouldn't decide that. I'm not arguing any kind of "should."
I'm just saying, if you do decide that, you're kind of dumb. And by analogy Eliezer was being kind of dumb in his article.
Interesting stuff. Very interesting.
Do you buy it?
That article is arguing that it's all right to value things that aren't mental states over a net gain in mental utility.[1] If, for instance, you're given the choice between feeling like you've made lots of scientific discoveries and actually making just a few scientific discoveries, it's reasonable to prefer the latter.[2]
Well, that example doesn't sound all that ridiculous.
But the logic that Eliezer is using is exactly the same logic that drives somebody who's dying of a horrible disease to refuse antibiotics, because she wants to keep her body natural. And this choice is — well, it isn't wrong, choices can't be "wrong" — but it reflects a very fundamental sort of human bias. It's misguided.
And I think that Eliezer's argument is misguided, too. He can't stand the idea that scientific discovery is only an instrument to increase happiness, so he makes it a terminal value just because he can. This is less horrible than the hippie who thinks that maintaining her "naturalness" is more important than avoiding a painful death, but it's not much less dumb.
[1] Or a net gain in "happiness," if we don't mind using that word as a catchall for "whatever it is that makes good mental states good."
[2] In this discussion we are, of course, ignoring external effects altogether. And we're assuming that the person who gets to experience lots of scientific discoveries really is happier than the person who doesn't, otherwise there's nothing to debate. Let me note that in the real world, it is obviously possible to make yourself less happy by taking joy-inducing drugs — for instance if doing so devalues the rest of your life. This fact makes Eliezer's stance seem a lot more reasonable than it actually is.
This may be a minor nit, but... is this forum collectively anti-orgasmium, now?
Because being orgasmium is by definition more pleasant than not being orgasmium. Refusing to become orgasmium is a hedonistic utilitarian mistake, full stop.[1] (Well, that's not actually true, since as a human you can make other people happier, and as orgasmium you presumably cannot. But it is at least on average a mistake to refuse to become orgasmium; I would argue that it is virtually always a mistake.)
[1] We're all hedonistic utilitarians, right?
I don't think his explanation for why a chair pushes back on your hand is quite right, either. I've mostly been told that material solidness comes from the Pauli exclusion principle, not electrostatic repulsion.
I don't know quantum mechanics, so I don't have a good perspective on the problem, but the electrostatic explanation has always seemed lacking to me. The electric charge in a neutral atom is fairly well-approximated by a symmetric sphere of negative charge with a bunch of positive charge at the center, so two atoms shouldn't experience much electrostatic repulsion until their electron clouds overlap. At which point [I've heard] the PEP should dominate the electrostatic force.
Can any physicists or physics students comment?
I have never used the word "mistake" by itself. I did say that refusing to become orgasmium is a hedonistic utilitarian mistake, which is mathematically true, unless you disagree with me on the definition of "hedonistic utilitarian mistake" (= an action which demonstrably results in less hedonic utility than some other action) or of "orgasmium" (= a state of maximum personal hedonic utility).[1]
I point this out because I think you are quite right: it doesn't make sense to tell somebody that they are mistaken in "wanting" something.
Indeed, I never argued that the dying hippie was mistaken. In fact I made exactly the same point that you're making, when I said:
What I said was that she is misguided.
The argument I was trying to make was, look, this hippie is using some suspect reasoning to make her decisions, and Eliezer's reasoning looks a lot like her's, so we should doubt Eliezer's conclusions. There are two perfectly reasonable ways to refute this argument: you can (1) deny that the hippie's reasoning is suspect, or (2) deny that Eliezer's reasoning is similar to hers.
These are both perfectly fine things to do, since I never elaborated on either point. (You seem to be trying option 1.) My comment can only possibly convince people who feel instinctively that both of these points are true.
All that said, I think that I am meaningfully right — in the sense that, if we debated this forever, we would both end up much closer to my (current) view than to your (current) view. Maybe I'll write an article about this stuff and see if I can make my case more strongly.
[1] Please note that I am ignoring the external effects of becoming orgasmium. If we take those into account, my statement stops being mathematically true.