It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial.
Consider the creationist who says: "But who can really know whether evolution is correct? It is just a theory. You should be more humble and open-minded." Is this humility? The creationist practices a very selective underconfidence, refusing to integrate massive weights of evidence in favor of a conclusion he finds uncomfortable. I would say that whether you call this "humility" or not, it is the wrong step in the dance.
What about the engineer who humbly designs fail-safe mechanisms into machinery, even though he's damn sure the machinery won't fail? This seems like a good kind of humility to me. Historically, it's not unheard-of for an engineer to be damn sure a new machine won't fail, and then it fails anyway.
What about the student who humbly double-checks the answers on his math test? Again I'd categorize that as good humility.
What about a student who says, "Well, no matter how many times I check, I can't ever be certain my test answers are correct," and therefore doesn't check even once? Even if this choice stems from an emotion similar to the emotion felt by the previous student, it is less wise.
You suggest studying harder, and the student replies: "No, it wouldn't work for me; I'm not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot." This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process. If you ask someone to "be more humble", by default they'll associate the words to social modesty—which is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. You'd still double-check your calculations if you were wise.
The student says: "But I've seen other students double-check their answers and then they still turned out to be wrong. Or what if, by the problem of induction, 2 + 2 = 5 this time around? No matter what I do, I won't be sure of myself." It sounds very profound, and very modest. But it is not coincidence that the student wants to hand in the test quickly, and go home and play video games.
The end of an era in physics does not always announce itself with thunder and trumpets; more often it begins with what seems like a small, small flaw... But because physicists have this arrogant idea that their models should work all the time, not just most of the time, they follow up on small flaws. Usually, the small flaw goes away under closer inspection. Rarely, the flaw widens to the point where it blows up the whole theory. Therefore it is written: "If you do not seek perfection you will halt before taking your first steps."
But think of the social audacity of trying to be right all the time! I seriously suspect that if Science claimed that evolutionary theory is true most of the time but not all of the time—or if Science conceded that maybe on some days the Earth is flat, but who really knows—then scientists would have better social reputations. Science would be viewed as less confrontational, because we wouldn't have to argue with people who say the Earth is flat—there would be room for compromise. When you argue a lot, people look upon you as confrontational. If you repeatedly refuse to compromise, it's even worse. Consider it as a question of tribal status: scientists have certainly earned some extra status in exchange for such socially useful tools as medicine and cellphones. But this social status does not justify their insistence that only scientific ideas on evolution be taught in public schools. Priests also have high social status, after all. Scientists are getting above themselves—they won a little status, and now they think they're chiefs of the whole tribe! They ought to be more humble, and compromise a little.
Many people seem to possess rather hazy views of "rationalist humility". It is dangerous to have a prescriptive principle which you only vaguely comprehend; your mental picture may have so many degrees of freedom that it can adapt to justify almost any deed. Where people have vague mental models that can be used to argue anything, they usually end up believing whatever they started out wanting to believe. This is so convenient that people are often reluctant to give up vagueness. But the purpose of our ethics is to move us, not be moved by us.
"Humility" is a virtue that is often misunderstood. This doesn't mean we should discard the concept of humility, but we should be careful using it. It may help to look at the actions recommended by a "humble" line of thinking, and ask: "Does acting this way make you stronger, or weaker?" If you think about the problem of induction as applied to a bridge that needs to stay up, it may sound reasonable to conclude that nothing is certain no matter what precautions are employed; but if you consider the real-world difference between adding a few extra cables, and shrugging, it seems clear enough what makes the stronger bridge.
The vast majority of appeals that I witness to "rationalist's humility" are excuses to shrug. The one who buys a lottery ticket, saying, "But you can't know that I'll lose." The one who disbelieves in evolution, saying, "But you can't prove to me that it's true." The one who refuses to confront a difficult-looking problem, saying, "It's probably too hard to solve." The problem is motivated skepticism aka disconfirmation bias—more heavily scrutinizing assertions that we don't want to believe. Humility, in its most commonly misunderstood form, is a fully general excuse not to believe something; since, after all, you can't be sure. Beware of fully general excuses!
A further problem is that humility is all too easy to profess. Dennett, in "Breaking the Spell", points out that while many religious assertions are very hard to believe, it is easy for people to believe that they ought to believe them. Dennett terms this "belief in belief". What would it mean to really assume, to really believe, that three is equal to one? It's a lot easier to believe that you should, somehow, believe that three equals one, and to make this response at the appropriate points in church. Dennett suggests that much "religious belief" should be studied as "religious profession"—what people think they should believe and what they know they ought to say.
It is all too easy to meet every counterargument by saying, "Well, of course I could be wrong." Then, having dutifully genuflected in the direction of Modesty, having made the required obeisance, you can go on about your way without changing a thing.
The temptation is always to claim the most points with the least effort. The temptation is to carefully integrate all incoming news in a way that lets us change our beliefs, and above all our actions, as little as possible. John Kenneth Galbraith said: "Faced with the choice of changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof." And the greater the inconvenience of changing one's mind, the more effort people will expend on the proof.
But y'know, if you're gonna do the same thing anyway, there's no point in going to such incredible lengths to rationalize it. Often I have witnessed people encountering new information, apparently accepting it, and then carefully explaining why they are going to do exactly the same thing they planned to do previously, but with a different justification. The point of thinking is to shape our plans; if you're going to keep the same plans anyway, why bother going to all that work to justify it? When you encounter new information, the hard part is to update, to react, rather than just letting the information disappear down a black hole. And humility, properly misunderstood, makes a wonderful black hole—all you have to do is admit you could be wrong. Therefore it is written: "To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty."