You have to have a core of arational desires/goals/values. Otherwise, you're just a logic engine with nothing to prove.
If someone would come and tell you: "I can't fall in love. I've never fallen in love and I also think it is wrong, harmful, crazy and irrational to fall in love." What would you tell?
Most people would say opt for one of the following answers:
Falling in love has a lot of parallels to real religious conviction: It's irrational. It's not supported by evidence, still can't be argued away. It can be harmful for the person and also easily exploited and often it is not mutual. :)
One such thing could be a conscious mind. You want to be rational and efficient, and you're offered a deal by a local neurosurgeon to reprogram your brain so that you become the perfect Bayesian that does everything to further your values. No acrasia, no bias blind spots, no hindrance at all. Only downside in the transformation process is that something that enables our conscious experiences(let it be something like self-reflection, or conflicts within the system) is lost. Wanna do it?
I'm well aware that people here might disagree if this would be possible...
What would be the consequence of giving up the idea of a subjective thread of consciousness?
I wonder if believers in subjective threads of consciousness can perform a thought experiment like Chalmers' qualia-zombie thought experiment. I gather that advocates of the subjective thread hold that it is something more than just having certain clumps of matter existing at different times holding certain causal relationships with one another. (Otherwise you couldn't decide which of two future copies of yourself gets to inherit your subjective thread). So, adv...
I don't see any reason why you should give up anticipation of future experiences. It's possible that in situations involving duplication and merging of minds you should accept that anticipation is an unreliable guide for decision-making, but that's not at all the same thing. (It's more like a religious person agreeing that if he gets seriously ill he should see a doctor rather than relying on his god to cure him -- which, in fact, religious people generally do.) At least for the present, the sort of switching and copying and merging and muddling of minds that would make our usual anticipation-based thinking fail badly doesn't happen to any appreciable extent.
Suppose you ask a religious friend why he doesn’t give up religion, he might say something like “Having faith in God comforts me and I think it is a central part of the human experience. Intellectually I know it’s irrational, but I want to keep my faith anyway. My friends and the government will protect me from making any truly serious mistakes as a result of having too much faith (like falling into dangerous cults or refusing to give medical treatment to my children)."
Um, no. If you were close with that friend, and he proved himself to be pretty ...
No, that isn't what they would say. I was a Christian for many years. Most of them sincerely believe everything they're supposed to believe - at least in conservative churches, which I think comprise a large majority of US churches.
Yes, and I would say actual faith is a cognitive error more akin to deja-vu than double think, in that it is a feeling of knowledge for which adequate logical justification may not exist. A friend of mine once said, "I'm sorry that I'm so bad at explaining this [the existence of God], but I just know it, and once you do too, you'll understand."
People can have experiences of faith in non-religious contexts, such as having faith (a sense of certainty or foreknowledge) that a critically ill loved-one will pull through. Intuition and gut-feelings maybe considered faith-light, but I think certainty is part of the faith experience, and just because that certainty is false, doesn't make the feeling any less real.
Do you think an intellectual argument is all you need to self-modify so drastically? Or is this just in anticipation of some future mind-surgery technology?
Does anyone value rationality for its own sake, enough to give up anticipation if it turns out to be irrational, purely on intellectual grounds?
You don't make a conscious decision to give up something like that, if it needs giving up. You learn more, see that what you once thought was sense was in fact nonsense, and in the moment of realization, you have already lost that which you never had. Really this is the wrong way to phrase the question: you should properly ask, "If the idea of anticipation is complete nonsense and all our thoughts about it are mere helpless clinging to our own confusion, would you rather know what was really going on?" and to this I answer "Yes."
If someone offered to tell me the Real Story, saying, "Once you learn the Real Story, you will lose your grasp of that which you once called 'anticipation'; the concept will dissolve, and you will find it difficult to remember why you ever once believed such a notion could be coherent; just as you once lost 'time'," I would indeed reply "Tell me, tell me!"
Faith is easy to dismiss because it can fairly be defined as "belief without evidence".
What exactly is meant by "anticipation"?
I haven't kept up with all the decision theory stuff but can someone demonstrate to me that anticipation might be irrational and not just that anticipation has to be reduced and re-understood?
I believe this idea has been played out in numerous works of science fiction (Kurt Vonnegut's Tralfamidorians for example). It's difficult to comprehend what it would be like to know/understand the future so well as to not anticipate it, and thus I cannot say if I would 'give it up.' I think Eliezer is right though- there would probably be no dramatic choice in the matter; either we understand or we don't.
This seems analogous to giving up the belief that free will is somehow ontologically basic. The experience of having made a "choice" can be arbitrarily superimposed on pretty much any action I perform. I value the experience of choosing, but recognize it as subjective fiction. Similarly, I find your suggestion
perhaps it can run some computations on the side to generate the qualia of anticipation and continuity
to be highly intuitively acceptable, but I feel that I'm missing something, perhaps a compelling counter-example.
No notion of "anticipation"?
That's only because it subcontracts that work to a "mathematical intuition subroutine" that allows the formation of beliefs about the likely consequences of actions.
It is a better summary; and I can give no better answer than, "It's always worked that way for me before." I think the real difficulty would come for someone who was told that they had to give up anticipation, rather than seeing it for themselves in a thunderbolt of dissolving insight.
My reasoning here is that evolution in general has very limited foresight, therefore there must be a limit to human rationality somewhere that is probably far short of ideal rationality. "It's always worked that way for me before" doesn't seem like very strong evidence in comparison to that argument.
Anticipation and faith are both aspects of the human decision process, in a sense just subroutines of a larger program, but they also generate subjective experiences (qualia) that we value for their own sake. Suppose you ask a religious friend why he doesn’t give up religion, he might say something like “Having faith in God comforts me and I think it is a central part of the human experience. Intellectually I know it’s irrational, but I want to keep my faith anyway. My friends and the government will protect me from making any truly serious mistakes as a result of having too much faith (like falling into dangerous cults or refusing to give medical treatment to my children)."
Personally I've never been religious, so this is just a guess of what someone might say. But these are the kinds of thoughts I have when faced with the prospect of giving up the anticipation of future experiences (after being prompted by Dan Armak). We don't know for sure yet that anticipation is irrational, but it's hard to see how it can be patched up to work in an environment where mind copying and merging are possible, and in the mean time, we have a decision theory (UDT) that seems to work fine, but does not involve any notion of anticipation.
What would you do if true rationality requires giving up something even more fundamental to the human experience than faith? I wonder if anyone is actually willing to take this step, or is this the limit of human rationality, the end of a short journey across the space of possible minds?