I have a confession: for years, I lied to myself about understanding Bayes' Theorem.

I could write the formula on a whiteboard. I could pass the exams. But I didn't feel it. In my mind, it was always a hostile mess of algebra that looked like this:

$P (H | E) = \frac{P (E | H) \cdot P (H)}{P (E)}$

It felt like a chore. Especially that denominator, $P (E)$ , which asks you to sum up every possible way the universe could have turned out just to normalize a single result.

Then I discovered Bits of Evidence.

The same bits Claude Shannon defined in 1948 to measure information turn out to be the natural units for reasoning under uncertainty.

This concept has deep roots: I. J. Good, one of Alan Turing's colleagues at Bletchley Park, formalized this idea of measuring evidence in logarithmic units. Good called it the "Weight of Evidence," building on Turing's wartime work on Bayesian inference for codebreaking.

When you use the right units, the scary formula vanishes. Conditional probability inference stops being algebra and becomes a simple tally of information.

The 50/50 Intuition: What is Zero?

We use the phrase "50/50" every day without thinking. What is the operation? It's 50 cases supporting the idea and 50 cases against it.

In a truly linear scale of information, I don't know should be Zero. When people say 50/50, they are describing a state of complete ignorance. You have exactly zero bits of information pushing you in either direction.

To reach this clarity, we move to Log-Odds space and measure everything in Bits. A bit is simply the amount of information needed to choose between two equally likely possibilities.

The Case of the Rare Disease

Let's see this in action with a medical example. A patient is worried about a rare condition ( $H$ ). The question: Is he Sick, or Healthy?

Imagine a giant Balance Scale.

Left Side: Evidence for Healthy (Negative Bits)
Right Side: Evidence for Sick (Positive Bits)

1. The Starting Point: True Ignorance

Before knowing anything, the scale is perfectly level at 0 bits. This is the true prior: complete ignorance, a 50/50 state with no information pushing in either direction.

2. The Population (The Heavy Anchor)

The first piece of evidence: look at the world. This disease is rare. In a population of 10,000 people, typically 12 are sick and 9,988 are healthy.

In textbooks, this base rate is called the Prior. There's nothing magical about it. It's just the first piece of...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments