Gray Area-fair point. Sorry, it's probably a terribly over-quoted book.

I really don't understand the debate. Bayesian reasoning IS the reasoning that scientists use. It is the method underlying the evolution of scientific theory. Popperian falsification is just some theory, more a prescriptive than descriptive rule. It's a pie in the sky which doesn't explain how the body of scientific knowledge evolves in time.

In practice, evidence is gathered to support or falsify a given scientific premise. Newtonian mechanics was TRUE until proven otherwise. And today's theories are more or less true based on their ability to explain reality (i.e., the same thing as positive evidence in a probabilistic sense) and not be disproved (i.e., have negative evidence against them). In reality, there are limits to our understanding and the scientist with any real sense of humility should agree with Box when he said that all models are false but some are useful.

Daniel, I think what you say about an implicit agreement that the current theory could be revised in light of new contradictory evidence, this is exactly Bayesian, a form of Bayesian model selection, where it may be that no theory or model is ever thrown out completely, just assigned a very low probability. Many evolutionary arguments are just a form of Bayesian update, conditioning on new evidence.

The idea that Bayesian decision theory being descriptive of the scientific process is very beautifully detailed in classics like Pearl's book, Causality, in a way that a blog or magazine article cannot so easily convey. In a different vein, for a very readable explanation of how "truth" changes, even in mathematics, the most pure of sciences, have a look at Imre Lakatos' book, Proofs and Refutations. In this book, Lakatos makes it clear that even mathematicians can use a Bayesian update of mathematical "evidence" for or against a given hypothesis, and that old "proofs" even by the greatest of mathematicians often have holes poked in them in time.

Now pure application of Bayes' rule may just merely give the probability that a theory/model is true. In reality, we probably do have some utility/loss function that gives us a decision rule as to whether we wish to use or discard a given theory. This loss function approach will actually allow us to use "false" theories such as Newtonian mechanics, when there is some utility to it, even though the evidence against them is immense.

What Eliezer is saying in the blog and what is said in the NS article is basically descriptive, imho, let's call a spade a spade...science is already Bayesian. Those of you who cannot really accept it and think this opens up science to the possibility of witchcraft are filled with a great idealism in how science is currently conducted behind closed doors. Either that, or like the Church fathers who silenced Galileo, you're awfully scared that the opposite of your dogma, witchcraft, might have an element of truth in it. Being honest about Bayesianism means we have to consider all the alternatives.

But, to be reassuring, I don't think we've seen a terrible amount of positive evidence for witchcraft lately....