Suppose you flip a coin times and get heads in a row. What is the probability the next flip will land heads?
Suppose the coin is either a fair coin with one heads or a trick coin with two heads. Let denote our training data of heads. We want to find and . Let . It follows that .
We use Bayes theorem .
I have flipped perhaps a hundred coins. One was double-headed trick coin. On the one hand, I flip trick coins with anomalously high frequency. On the other hand, double-headed trick coins are more likely than regular coins to get flipped than fair coins. I estimate the flip frequency of double-headed trick coins to be one in ten thousand.
What does it look like when we graph our probabilities with ?
For the first 5 heads you can remain confident you are flipping a regular coin. Around 10 heads the exponential takes off. You quickly become confident you are not flipping a regular coin. At 20 heads in a row you can be confident you are not flipping a regular coin.
The Inflection Point
The inflection point occurs when the probabilities are equal.
A linear increase in your data has predictive power equal to an exponential increase in the strength of your prior.