Suppose you flip a coin $n$ times and get $n$ heads in a row. What is the probability the next flip will land heads?

Suppose the coin is either a fair coin with one heads or a trick coin with two heads. Let $X$ denote our training data of $n$ heads. We want to find $P\left(\text{fair}|X\right)$ and $P\left(\text{trick}|X\right)$. Let $P\left(\text{trick}\right)=ϵ$. It follows that $P\left(\text{fair}\right)=1-ϵ$.

We use Bayes theorem $P\left(A|B\right)=\frac{P\left(B|A\right)P\left(A\right)}{P\left(B\right)}$.

$$\begin{array}{ccc}P\left(\text{fair}|X\right)& =& \frac{P\left(X|\text{fair}\right)P\left(\text{fair}\right)}{P\left(X\right)}\\ & =& \frac{P\left(X|\text{fair}\right)P\left(\text{fair}\right)}{P\left(\text{fair}\right)P\left(X|\text{fair}\right)+P\left(\text{trick}\right)P\left(X|\text{trick}\right)}\\ & =& \frac{2^{-n}(1-ϵ)}{(1-ϵ)2^{-n}+ϵ1}\\ & =& \frac{(1-ϵ)}{(1-ϵ)+2^nϵ}\end{array}$$

$$\begin{array}{ccc}P\left(\text{trick}|X\right)& =& \frac{P\left(X|\text{trick}\right)P\left(\text{trick}\right)}{P\left(X\right)}\\ & =& \frac{P\left(X|\text{trick}\right)P\left(\text{trick}\right)}{P\left(\text{trick}\right)P\left(X|\text{trick}\right)+P\left(\text{fair}\right)P\left(X|\text{fair}\right)}\\ & =& \frac{1ϵ}{ϵ1+(1-ϵ)2^{-n}}\\ & =& \frac{ϵ}{ϵ+(1-ϵ)2^{-n}}\\ & =& \frac{2^nϵ}{(1-ϵ)+2^nϵ}\end{array}$$

I have flipped perhaps a hundred coins. One was double-headed trick coin. On the one hand, I flip trick coins with anomalously high frequency. On the other hand, double-headed trick coins are more likely than regular coins to get flipped than fair coins. I estimate the flip frequency of double-headed trick coins to be one in ten thousand.

$$ϵ=\frac{\text{trick coins}}{\text{total coins}}=\frac{1}{10,000}={10}^{-4}$$

What does it look like when we graph our probabilities with $ϵ={10}^{-4}$?

For the first 5 heads you can remain confident you are flipping a regular coin. Around 10 heads the exponential takes off. You quickly become confident you are not flipping a regular coin. At 20 heads in a row you can be confident you are not flipping a regular coin.

The Inflection Point

The inflection point occurs when the probabilities are equal.

$$\begin{array}{ccc}P\left(\text{fair}|X\right)& =& P\left(\text{trick}|X\right)\\ \frac{(1-ϵ)}{(1-ϵ)+2^nϵ}& =& \frac{2^nϵ}{(1-ϵ)+2^nϵ}\\ 1-ϵ& =& 2^nϵ\\ \frac{1}{ϵ}-1& =& 2^n\\ \frac{1}{ϵ}& =& 2^n+1\\ ϵ& =& \frac{1}{2^n+1}\end{array}$$

A linear increase in your data has predictive power equal to an exponential increase in the strength of your prior.