We do ten experiments. A scientist observes the results, constructs a
theory consistent with them, and uses it to predict the results of the
next ten. We do them and the results fit his predictions. A second
scientist now constructs a theory consistent with the results of all
twenty experiments.
The two theories give different predictions for the next experiment. Which do we believe? Why?
One of the commenters links to Overcoming Bias, but as of 11PM on Sep 28th, David's blog's time, no one has given the exact answer that I would have given. It's interesting that a question so basic has received so many answers.
Both theories fit 20 data points. That some of those are predictions is irrelevant, except for the inferences about theory simplicity that result. Since likelihoods are the same, those priors are also the posteriors.
My state of belief is then represented by a certain probability that each theory is true. If forced to pick one out of the two, I would examine the penalties and payoffs of being correct and wrong, ala Pascal's wager.
David D. Friedman asks:
One of the commenters links to Overcoming Bias, but as of 11PM on Sep 28th, David's blog's time, no one has given the exact answer that I would have given. It's interesting that a question so basic has received so many answers.