"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.
What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.
For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:
- 3^3 = 27.
- 3^^3 = (3^(3^3)) = 3^27 = 7625597484987.
- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).
3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.
Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:
Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?
I think the answer is obvious. How about you?
The question is, of course, silly. It is perfectly rational to decline to answer. I choose to try to answer.
It is also perfectly rational to say "it depends". If you really think "a dust speck in 3^^^3 eyes" gives a uniquely defined probability distribution over different subsets of possibilityverse, you are being ridiculous. But let's pretend it did - let's pretend we had 3^^^^3 parallel Eleizers, standing on flat golden surfaces in 1G and one atmosphere, for just long enough to ask each other enough enough questions to define the problem properly. (I'm sorry, Eleizer, if by stating that possibility, I've increased the "true"ness of that part of the probabilityverse by ((3^^^3+1)/3^^^3) :) ).
You can or "I've thought about it, but I don't trust my thought processes". That is not my position.
My position is that this question does not, in fact, have an answer. I think that that fact is very important.
It's not that the numbers are meaningless. 3^^^3 is a very exact number, and you can prove any number of things about it. A different question using ridiculous numbers - say, would you rather torture 4^^^4 people for 5 minutes or 3^^^3 of them for 50 years - has a single correct answer which is very clear (of course, the 3^^^3 ones; 4^^^4 >>> (3^^^3)^2). (Unless there were very bizarre extra conditions on the problem.)
It's just that there is no universal moral utility function which inputs a probability distribution over a finite subset of the possibilityverse and outputs a number. It's more like relativistic causality - substitute "better" for "after". A is after B and B is a spacelike distance from C, but C can also be spacelike from A. The dust specks and the torture are incomparable, a spacelike distance.
I think that, philosophically, that makes a big difference. If you pilosophically can't always go around morally comparing near-infinite sets, then it's silly to try to approximate how you would behave if you could. Which means you consider the moral value of the consequences which you could possibly anticipate. So yeah, if you are working on AI, you are morally obligated to think about FAI, because that's intentional action, and you would have to be a monster to say you didn't care. But you don't get to use FAI and the singularity to trump the here-and-now, because in many ways they're just not comparable.
Which means, to me, for instance, that people can understand the singularity idea and believe it has a non-0 probability, and have abilities or resources that would be meaningful to the FAI effort, and still morally choose to simply live as "good people" in a more traditional sense (have a good life in which they make the people with whom they interact overall happier). It's not just a lack of ability to trace the consequences; it's also the possibility that the consequences of this or that outcome will be literally incomparable by any finite halting algorithm, whereas even our desperately-limited brains have decent approximations of algorithms for morally comparing the effect of, say, posting on OB versus washing the dishes.
Going to wash the dishes now.