People can fool themselves, hallucinate, and even go insane.
This is the key point - when people have crazy moments then compared to non crazy moments they are more likely to make extreme claims, and this is what makes it reasonable to be skeptical of extreme claims.
Academic communities can also have crazy moments, which is why we should be more skeptical of extreme published claims.
Given your emphasis on replication, it becomes extremely important to evaluate how an academic field actually practices, rewards and interprets attempts at replication. They may give lip service only.
Would you say that we're in basic agreement then that: "Extraordinary claims are always extraordinary evidence, but some claims are more extraordinary evidence than others, and some hypotheses are just too extraordinary."
Eliezer, I think saying "some hypotheses are just too extraordinary" is misleading in the sense it attributes the problem to the hypotheses, while the more fundamental problem is the tendency of some types of people to be too gullible about some types of extreme claims.
I don't know whether most scientists would subscribe it but my guess is that most theoretical physicists would, and I would, too. A solid structure of science - a solid theory extracted from a lot of repeated experiments that suddenly made sense - is more reliable than a particular observation of a particular individual, no doubt about it. In the context of physical scientists, well, experimenters are often wrong, too. Of course, if they were ideal craftsmen and if they could also perfectly evaluate all the theory that is necessary to interpret the data correctly, they would always be the most perfect source of information. But it's just not the case. Click my name for more comments about the relations between theory, experiments, and confidence.
I have a question about the problem of recursion here. If I observe a blue sky and I observe a group of scientific papers claiming that it is green, how much more likely is it that my observation of the sky is what is wrong than that my observation or understanding of the scientific articles are what is wrong?
Yes, exactly. Hallucinations and altered consciousness periods don't simply mean that your sane and usual rational mind is still there and it simply receives strange visual inputs as if you were enjoying a movie. Sometimes your very own thought processes are disturbed, it's not like a little rational homunculus can always remain skeptical. So if you then try to think about journals and science, it won't feel like a better alternative hypothesis. You will be genuinely confused and maybe imagine reading something in a journal that you didn't, or imagine that some conspiracy is out there and you're now uncovering it, etc. A real strong hallucination can be very very strong. Some on-the-fence atheists convert to a religion when they have some extreme experience of pleasure and bliss and feel that some miraculous event happened. They will know that religion is true, it will be the first and foremost truth they can imagine. It's really hard to imagine what it feels like to be someone else. Again, no rational homunculi pulling the string in our heads.
A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes.
Including my viewing of the report itself? That would be silly. Later you say that the fact that it is a good idea to trust in science is "pragmatically true," but probably better to say it's a good rule of thumb. I agree with the spirit of the post, but it goes so far into the hyperbolic that it undermines some other aspects of rationality:
What about the claim that 2 + 2 = 5?
Science cannot prove a nonsensical claim. What about the claim "A and not-A"?
I can't trust science more than the most trustworthy part of my brain, which says that 2 + 2 = 4, because I've only come to trust science as much as I do because the most trustworthy part of my brain says it's legit.
But I do trust science more than most of my brain, for example my instincts about simultaneity.
You're talking about your scientific beliefs and your brain as though they are two separate things. In reality, you have your beliefs about science as one part of your brain and your common sense intuitions as another part of your brain. You don't reject your own experiences and beliefs in favor of the external authority of Science, or, at least, you shouldn't. Instead, you should evaluate probabilities internally, using different parts of your brain and weighing their arguments against each other to come to a conclusion. Your own judgement is the only one you have.
The post annoys me because it doesn't seem to approach the problem from this direction. The problem isn't that we place too much value on experience, the problem is that we don't consider all aspects of our experience fully, even if those aspects are less visible. The fact that science works in the laboratory is something I know from my experience just as much as my intuitions are. The goal shouldn't be to reject individual experience but to see individual experience from the broadest angle possible.
I have to say that the percentage of peer-reviewed scientific journal articles that end up retracted or refuted is significantly higher then the percentage of the time that my senses have given me information that has turned out to be false.
As for the "some hypothesis are just too extraordinary" argument; I have to say that, historically speaking, prejudging what types of hypothesis are just so extraordinary that it's not even worth looking at evidence in favor of them is a type of judgement that humans, even very smart humans, seem to be quite bad at.
I have to say that the percentage of peer-reviewed scientific journal articles that end up retracted or refuted is significantly higher then the percentage of the time that my senses have given me information that has turned out to be false.
"Besides, your perceptions are fine, it's your opinions that are worthless and misleading."
"I would sooner believe that two Yankee professors would lie, than that stones would fall from heaven."
-- Thomas Jefferson, on meteors
"How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn't. It isn't going to happen."
-- Eliezer Yudkowsky, "A Technical Explanation of Technical Explanation"
"If a ship landed in my yard and LGMs stepped out, I’d push past their literature and try to find the cable that dropped the saucer on my roses. Lack of a cable or any significant burning to the flowers, I’d then grab a hammer and start knocking about in the ship till I was convinced that nothing said “Intel Inside.” Then when I discovered a “Flux Capacitor” type thing I would finally stop and say, “Hey, cool gadget!” Assuming the universal benevolence of the LGMs, I’d yank it out and demand from the nearest "Grey” (they are the tall nice ones), “where the hell did this come from?” Greys don’t talk, they communicate via telepathy, so I’d ignore the voice inside my head. Then stepping outside the saucer and sitting in a lawn chair, I’d throw pebbles at the aliens till I was sure they were solid. Then I’d look down at the “Flux Capacitor” and make sure it hadn’t morphed into my bird feeder. Finally, with proof in my hand and aliens sitting on my deck (they’d be offered beers, though I’ve heard that they absorb energy like a plant) I’d grab my cell phone and tell my doctor that I’m having a serious manic episode with full-blown visual hallucinations."
-- Peter K. Bertine, on the Extropian mailing list
We underestimate the power of science, and overestimate the power of personal observation. A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes. Our own eyes can deceive us. People can fool themselves, hallucinate, and even go insane. The controls on publication in major journals are more trustworthy than the very fabric of your brain. If you see with your own eyes that the sky is blue, and Science says it is green, then sir, I advise that you trust in Science.
This is not what most scientists will tell you, of course; but I think it is pragmatically true. Because in real life, what happens is that your eyes have a little malfunction and decide that the sky is green, and science will tell you that the sky is blue.
A replicated scientific report is a special kind of extraordinary claim, designed by the surrounding process to be more extraordinary evidence than simple verbal claims. It is more extraordinary evidence because the surrounding process - and I would place a far greater premium on the replication than on the peer review, by the way - is constructed to deny entrance to claims that are in fact false. In this way, the replicated scientific report becomes capable of overcoming greater burdens of prior improbability.
There are some burdens of prior improbability so great that simple verbal claims cannot overcome them. I would not believe someone who claimed that their coffee was disobeying conservation of angular momentum - but I might believe the same report published in Physics Today, with at least three replications. Who would believe in quantum mechanics if a stranger walked up to us on the street and whispered it to us?
Are there some burdens of prior improbability so great that science itself cannot overcome them?
What about the claim that 2 + 2 = 5?
What about journals that claim to publish replicated reports of ESP?
Sometimes, even claims deliberately constructed to be extraordinary evidence end up just not being extraordinary enough.