The following happened to me in an IRC chatroom, long enough ago that I was still hanging around in IRC chatrooms. Time has fuzzed the memory and my report may be imprecise.
So there I was, in an IRC chatroom, when someone reports that a friend of his needs medical advice. His friend says that he’s been having sudden chest pains, so he called an ambulance, and the ambulance showed up, but the paramedics told him it was nothing, and left, and now the chest pains are getting worse. What should his friend do?
I was confused by this story. I remembered reading about homeless people in New York who would call ambulances just to be taken someplace warm, and how the paramedics always had to take them to the emergency room, even on the 27th iteration. Because if they didn’t, the ambulance company could be sued for lots and lots of money. Likewise, emergency rooms are legally obligated to treat anyone, regardless of ability to pay.1 So I didn’t quite understand how the described events could have happened. Anyone reporting sudden chest pains should have been hauled off by an ambulance instantly.
And this is where I fell down as a rationalist. I remembered several occasions where my doctor would completely fail to panic at the report of symptoms that seemed, to me, very alarming. And the Medical Establishment was always right. Every single time. I had chest pains myself, at one point, and the doctor patiently explained to me that I was describing chest muscle pain, not a heart attack. So I said into the IRC channel, “Well, if the paramedics told your friend it was nothing, it must really be nothing—they’d have hauled him off if there was the tiniest chance of serious trouble.”
Thus I managed to explain the story within my existing model, though the fit still felt a little forced . . .
Later on, the fellow comes back into the IRC chatroom and says his friend made the whole thing up. Evidently this was not one of his more reliable friends.
I should have realized, perhaps, that an unknown acquaintance of an acquaintance in an IRC channel might be less reliable than a published journal article. Alas, belief is easier than disbelief; we believe instinctively, but disbelief requires a conscious effort.2
So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened. And I knew how embarrassing this was. I knew that the usefulness of a model is not what it can explain, but what it can’t. A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.
Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.
We are all weak, from time to time; the sad part is that I could have been stronger. I had all the information I needed to arrive at the correct answer, I even noticed the problem, and then I ignored it. My feeling of confusion was a Clue, and I threw my Clue away.
I should have paid more attention to that sensation of still feels a little forced. It’s one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading:
Either Your Model Is False Or This Story Is Wrong.
1 And the hospital absorbs the costs, which are enormous, so hospitals are closing their emergency rooms . . . It makes you wonder what’s the point of having economists if we’re just going to ignore them.
2 From McCluskey (2007), “Truth Bias”: “[P]eople are more likely to correctly judge that a truthful statement is true than that a lie is false. This appears to be a fairly robust result that is not just a function of truth being the correct guess where the evidence is weak—it shows up in controlled experiments where subjects have good reason not to assume truth[.]” http://www.overcomingbias.com/2007/08/truth-bias.html .
And from Gilbert et al. (1993), “You Can’t Not Believe Everything You Read”: “Can people comprehend assertions without believing them? [...] Three experiments support the hypothesis that comprehension includes an initial belief in the information comprehended.”
A valid point, Psy-Kosh, but I've seen this happen to a friend too. She was walking along the streets one night when a strange blur appeared across her vision, with bright floating objects. Then she was struck by a massive headache. I had her write down what the blur looked like, and she put down strange half-circles missing their left sides.
That point was when I really started to get worried, because it looked like lateral neglect - something that I'd heard a lot about, in my studies of neurology, as a symptom of lateralized brain damage from strokes.
The funny thing was, nobody in the medical profession seemed to think this was a problem. The medical advice line from her health insurance said it was a "yellow light" for which she should see a doctor in the next day or two. Yellow light?! With a stroke, you have to get the right medication within the first three hours to prevent permanent brain damage! So we went to the emergency room - reluctantly, because California has enormously overloaded emergency rooms - and the nurse who signed us in certainly didn't seem to think those symptoms were very alarming.
The thing is, of course, that non-doctors are legally prohibited from making diagnoses. So neither the nurse on the advice line, or the nurse who signed us into the emergency room, were allowed to say: "It's a migraine headache, you idiots."
You see, I'd heard the phrase "migraine headache", but I'd had no idea of what the symptoms of a "migraine headache" were. My studies in neurology told me about strokes and lateral brain damage, because those are very important to the study of functional neuroanatomy. So I knew about these super dangerous and rare killer events that seemed sort of like the symptoms we were encountering, but I didn't know about the common events that a doctor sees every day.
When you see symptoms, you think of lethal zebras, because those are what you read about in the newspapers. The doctor thinks of much less exciting horses. This is why the Medical Establishment has always been right, in my experience, every single time I'm alarmed and they're not.
But in answer to your question about selection effects, Psy-Kosh, I think I'd have noticed if my friend had actually had a stroke. In fact, it would have been much more likely to have been reported and repeated than the reverse case.
I had a similar experience with my girlfriend, except the symptoms were significantly more alarming. She was, among other things, unable to remember many common nouns. I would point and say 'What is that swinging room separator?" and she would be unable to figure out "door".
I was aware from the start that the symptoms might have been due to a migraine aura, having looked up the symptoms on Wikipedia, but was advised by 811 to take her to the hospital immediately. The symptoms were gone before we arrived. Five hours later (a strong hint that at least the triage people thought it wasn't an emergency), a doctor had diagnosed it as a silent migraine.