Richard Ngo challenged me to set a time box and write down as many of the most important features of my formal epistemology as I can in one sitting. Here goes.

Motivation: Where probability distributions fail...

...to express beliefs

• There is no probability distribution over that says “”. That is a support condition about probability distributions, namely, ⨾⨾. Some s satisfy this condition and some do not; but there is no that expresses the range belief itself.
• There is no probability distribution over that says “ and are independent”. This is an equation condition about probability distributions, namely, . Some s satisfy this equation and some do not; there is no that expresses the independence belief itself.
• There is no probability distribution over that can express a conditional probability distribution , even though this is just as essential a part of a Bayesian reasoner's epistemic state as her prior. A conditional probability distribution is a family of probability distributions indexed by the condition variable, not a probability distribution. There is no single distribution that expresses it.

...to make safety tradeoffs

• Suppose there is an unfair coin, which you know to be unfair (but not exactly how much or in which direction), and you must choose between three options: bet on Heads, bet on Tails, or bet instead on a fair d20 coming up at least 12. Exercise for the reader: Every probability distribution you could possibly believe makes choosing the d20 irrational.
  • The situation doesn't even need to involve Omega for the d20 to be a permissible choice; if the coin is chosen by Omega (who adversarially determines which way the unfair coin is unfair by predicting your betting choice), the d20 is the uniquely rational choice, making the orthodox Bayesian view (that beliefs are probability distributions) clearly wrong.
  • Maximum entropy does not save you! In fact, if you follow Jaynes’ principle of maximum entropy (take the maximum entropy probability distribution consistent with your beliefs to be your belief), then the bet on Heads and the bet on Tails both strictly dominate the bet on the d20.

Beliefs, according to davidad

Definition 1. Given a state space , we define probability space as the space of all conceivable probability distributions on .

Definition 2. A belief about () is a functional

which is lower semicontinuous (meaning that ).

Following Richardson, we interpret as the level of inconsistency between the conceivable probability distribution and one's belief .

Following Lawvere, we order by (“backwards”), and pointwise, so that all conceivable beliefs about form a (thin) category of beliefs . Morphisms in can be interpreted as implications . is a commutative semicartesian monoidal category with .

Slogan: When one has multiple beliefs at the same time, one's overall belief is simply the sum of its parts.

All other known notions of belief are full subcategories

Bayesian beliefs

Definition 3. If one has a prior (big if), then one's prior belief is

Bayesian updating

Definition 4. If one has a belief over a hypothesis space, and the total state space of the situation is the product of hypothesis space and data space , then one's prior belief about is the inverse-image functor

Definition 5. An observation is a closed subset .

Example 6. If one observes that , this is the subset .

Definition 7. The belief that corresponds to an observation is

Definition 8. If one has a conditional distribution , then one's conditional belief is

Example 9. If one has a likelihood function , then the previous definition applies with and :

Definition 10. A Bayesian reasoner with prior and likelihood function has belief state

Theorem 11. The two components of a Bayesian reasoner's initial belief state, only one of which is a probability distribution, simply sum when lifted into the belief space, forming the joint belief:

However, this would be meaningless unless we could recover the Bayesian update by also summing the observations in the belief space.

Theorem 12. Whenever the Bayesian posterior is well-defined, then

with a constant denoting the level of incompatibility of the observation with the Bayesian reasoner's initial belief state, namely the prior predictive surprisal, .

Notice that the orthodox Bayesian update's “normalization” to a full-mass posterior distribution silently subtracts away the constant — the amount of incompatibility between the observed data and the statistical model as a whole — which is Deborah Mayo's criticism of Bayesian epistemology in a nutshell.

Bayesian beliefs have fully trivial categorical structure (no Bayesian belief implies any other Bayesian belief except itself), so Bayesian beliefs about are a full subcategory of , but the content of this is just that is injective (its post-inverse is ).

Infra-Bayesian beliefs (Kosoy and Appel)

Definition 13 (Kosoy and Appel). A homogenous ultracontribution is a nonempty topologically-closed convex down-closed subset of subprobability space, .

Theorem 14. Homogenous ultracontributions (ordered by ) form a full subcategory of . Specifically, given , the corresponding belief is

and given a belief , the corresponding subset of subprobability space is

and . Furthermore, is a member of iff is convex in probability space, which every satisfies.

Definition 15 (Kosoy and Appel). A homogenous ultradistribution is a homogenous ultracontribution whose intersection with the full-mass face is nonempty ().

Of course, is also a full subcategory of , since is a full subcategory of .

MWER (Halpern and Leung)

Theorem 16. Homogenous ultradistributions are exactly isomorphic to the belief states of Halpern and Leung's MWER framework (Minimax Weighted Expected Regret), via

Theorem 17. The updating process of simply adding , which is equivalent to that of Theorem 12, is also equivalent to Halpern and Leung's prescribed belief-updating procedure on the domain of their belief states.

Note: via the transform, this updating process is also consonant with the famous Multiplicative Weight Update family of algorithms (although I am not yet confident about whether e.g. AdaBoost is literally a special case of it).

Probabilistic dependency graphs (Richardson and Halpern)

Richardson and Halpern's PDGs, a common generalization of Bayes nets and factor graphs, take their semantics in functionals , and these functionals are always lower semicontinuous (though Richardson does not explicitly prove this), so therefore every PDG denotes a belief in my sense. Furthermore, the combination of PDGs with overlapping variables denotes exactly the sum of their beliefs, so the semantics of PDGs is a monoidal functor from PDGs into beliefs.

For a wide parameter regime of PDGs ( on every edge, or equivalently ) — assumed by some but not all PDG theorems [correction due to Richardson himself in the comments] — the beliefs they denote are convex, and thus immediately satisfy the exp-convexity criterion to be transformed into homogenous ultracontributions. However, even these PDGs do not typically denote homogenous ultradistributions, unless their semantics are “normalized” (by subtracting ).

PDGs can also express independence beliefs, which are non-convex while remaining lower semicontinuous.

Credal sets (Cozman)

Definition 18 (Cozman). A credal set is a nonempty closed convex set of probability distributions, .

Theorem 19. Credal sets about (ordered by ) form a full subcategory of . Specifically, given , the corresponding belief is

and given a belief , the corresponding credal set is

which is topologically closed by lower semicontinuity of .

Previsions (Goubault-Larrecq)

Definition 20 (Goubault-Larrecq). A gamble about is a Borel function . A prevision about is a functional such that and . An upper prevision is a prevision which is sub-additive: . A continuous upper prevision is an upper prevision which is Scott-continuous (for every directed family with least upper bound , ).

Definition 21. A coherent continuous upper prevision is a continuous upper prevision satisfying .

Theorem 22. Coherent continuous upper previsions about form a full subcategory of . Specifically, given the coherent continuous upper prevision , the corresponding belief is

and given a belief , the corresponding upper prevision is

The monad (Mio, Sarkis, and Vignudelli)

Definition 23 (Mio, Sarkis, and Vignudelli). The monad is defined on sets as the set of non-empty finitely-generated down-closed convex sets of subprobability distributions .

Mio, Sarkis, and Vignudelli prove that this monad is presented by the equational theory of semilattices equipped with finite probabilistic choice and , making in a strong sense the smallest semantic universe that can simultaneously interpret finite nondeterminism, finite probability, and partiality (and partiality is, in turn, needed to interpret either inconsistent beliefs or nonterminating probabilistic programs).

Theorem 24. ordered by inclusion is a full subcategory of , assuming is a Polish space. Specifically, this condition implies that finitely-generated convex sets are topologically closed, which makes a full subcategory of .