# 2

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Fix a theory over a language . A coherent probability function is one that satisfies laws of probability theory, each coherent probability function represents a probability distribution on complete logical extensions of .

One of many equivalent definitions of coherence is that is coherent if whenever can prove that exactly one of is true.

Another very basic desirable property is that only when is provable. There have been many proposals of specific coherent probability assignments that all satisfy this basic requirement. Many satisfy stronger requirements that give bounds on how far is from 1 when is not provable.

In this post, I modify the framework slightly to instead talk about conditional probability. Consider a function which takes in a consistent theory and a sentence , and outputs a number , which represents the conditional probability of given .

We say that is coherent if:

1. whenever can prove that exactly one of is true, and

2. If proves every sentence in , then .

Theorem: There is no coherent conditional probability function such that only when proves .

Proof:

This proof will use the notation of log odds to make things simpler.

Let be a coherent conditional probability function. Fix a sentence which is neither provable nor disprovable from the empty theory. Construct an infinite sequences of theories as follows:

1. is the empty theory.

2. To construct , choose a sentence such that neither nor are provable in . If , then let . Otherwise, let .

Fix an , and without loss of generality, assume . Since is coherent we have In particular, this means that .

Observe that , and . Therefore, , so .

In the language of log odds, this means that .

Let be the union of all the . Note that by the third condition of coherence, for all , so for all

Consider and . These numbers cannot both be finite, since . Therefore, at least one of and must be 0 or 1. However neither nor prove or disprove , so this means that assigns conditional probability 1 to some statement that cannot be proven.

Open Problem: Does this theorem still hold if we leave condition 3 out of the definition of coherence?