This expands on some issues I mentioned in Radical Probabilism.

A rationality pet-nitpick of mine, which is shared by almost no one, is the probable/likely distinction. I got my introduction to Bayesian thinking, in part, from Probabilistic Reasoning in Intelligent Systems by Judea Pearl. In the book, Pearl makes a simple distinction between probability and likelihood which I find to be quite wonderful: the likelihood of X given Y is just the probability of Y given X!

Probability of X given Y: $P (X | Y)$

Likelihood of X given Y: $L (X | Y) := P (Y | X)$

Why invent new terminology for something so simple?

What we're basically doing here is making a type distinction between probability and likelihoods, to help us remember that likelihoods can't be converted to probabilities without combining them with a prior.

In the context of the book, it's because Bayesian networks pass two types of messages: "probability" type messages pass prior-like information down the Bayesian network, and "likelihood" type messages pass evidence-like information up the network.

At each individual node, we need to combine a prior with evidence in order to get a posterior. However, we only start with prior information for the top nodes of the network, and we only start with evidence information for the observed nodes. In order to get the information we need for other nodes, we have to propagate information around the network via the two types of messages.

Another important aspect of the probability/likelihood distinction is that we swap our views of what varies and what remains fixed. When we regard P(A|B) as a probability function, rather than just a single probability judgement, we generally think of varying A. But when we use P(A|B) to construct a likelihood function for hypothesis testing, we think of A as the fixed evidence, and more readily vary B, which is now thought of as the hypothesis. Pearl's notation helps us track which thing we're holding steady vs which thing we're varying: we think of the first argument as what we're varying, and the second argument as what we're holding steady. Critically, probability functions sum to 1, while likelihood functions need not do so.

(Likelihood functions can sum to less than 1 if the event was improbable according to all the hypotheses, or more than 1 if the event was quite probable according to all the hypotheses.)

The downward messages in a Bayesian ...

Consequences of Logical Induction