Here’s a conceptual problem David and I have been lightly tossing around the past couple days.

“A is a subset of B” we might visualize like this:

If we want a fuzzy/probabilistic version of the same diagram, we might draw something like this:

And we can easily come up with some ad-hoc operationalization of that “fuzzy subset” visual. But we’d like a principled operationalization.

Here’s one that I kinda like, based on maxent machinery.

Background Concept 1: $E[-logP[X\left]\right]\le H_P\left(X\right)$ Encodes The Same Information About $X$ As $P$ Itself

First, a background concept. Consider this maxent problem:

${\text{max}}_{P^{\prime }}-{\sum }_X{P}^{\prime }\left[X\right]log{P}^{\prime }\left[X\right]\text{s.t.}-{\sum }_X{P}^{\prime }\left[X\right]logP\left[X\right]\le -{\sum }_{X}P\left[X\right]logP\left[X\right]$

Or, more compactly

$\text{maxent}\left[X\right]\text{s.t.}E[-logP[X\left]\right]\le H_P\left(X\right)$

In English: what is the maximum entropy distribution $P^{\prime }$ for which (the average number of bits used to encode a sample from $P^{\prime }$ using a code optimized for distribution $P$) is at most (the average number of bits used to encode a sample from $P$ using a code optimized for $P$)?

The solution to this problem is just $P^{\prime }=P$.

So conceptually, leveraging the zen of maxent distributions, the constraint $E[-logP[X\left]\right]\le H_P\left(X\right)$ encodes the same information about $X$ as the distribution $P$ itself.

Background Concept 2: … So Let’s Use Maxent To Fuse Distributions?

Conceptually, if the constraint $E[-logP[X\left]\right]\le H_P\left(X\right)$ encodes all the information from $P$ into a maxent problem, and the constraint $E[-logQ[X\left]\right]\le H_Q\left(X\right)$ encodes all the information from $Q$ into a maxent problem, then solving the maxent problem with both of those constraints integrates “all the information from both $P$ and $Q$” in some sense.

Qualitatively, here’s what that looks like in an example:

$P$ says that $X$ is probably in the red oval. $Q$ says that $X$ is probably in the blue oval. So together, they conceptually say that $X$ is probably somewhere in the middle, roughly where the two intersect.

Mathematically, the first order maxent condition says $P^{\prime }\left[X\right]=\frac{1}{Z}P\left[X{]}^{\alpha }Q\right[X{]}^{\beta }$, for some $\alpha ,\beta$ (which we will assume are both positive, because I don’t want to dive into the details of that right now). For any specific $X$ value, $P[X{]}^{\alpha }$ and $Q[X{]}^{\beta }$ can be no larger than 1, but they can be arbitrarily close to 0 (they could even be 0 exactly). And since they’re multiplied, when either one is very close to 0, we intuitively expect the product to be very close to 0. Most of the probability mass will therefore end up in places where neither distribution is very close to 0 - i.e. the spot where the ovals roughly intersect, as we’d intuitively hoped. 

Notably, in the case where $P$ and $Q$ are uniform over their ovals (so they basically just represent sets), the resulting distribution is exactly the uniform distribution over the intersection of the two sets. So conceptually, $P$ says something like “$X$ is in set $S_P$”, $Q$ says something like “$X$ is in set $S_Q$”, and then throwing both of those into a maxent problem says something like “$X$ is in $S_P$ and $X$ is in $S_Q$, i.e. $X$ is in the intersection”.

So that hopefully gives a little intuition for how and why maxent can be used to combine the information “assumed in” two different distributions $P,Q$.

Something Like A Subset Relation?

What if we throw $E[-logP[X\left]\right]\le H_P\left(X\right)$ and $E[-logQ[X\left]\right]\le H_Q\left(X\right)$ into a maxent problem, but it turns out that the $Q$ constraint is nonbinding? Conceptually, that would mean that $P$ already tells us everything about $X$ which $Q$ tells us (and possibly more). Or, in hand wavy set terms, it would say that $S_P$ is a subset of $S_Q$, and therefore puts a strictly stronger bound on $X$.

In principle, we can check whether $Q$’s constraint is binding without actually running the maxent problem. We know that if $Q$’s constraint doesn’t bind, the maxent solution is $P$, so we can just evaluate $Q$’s constraint at $P$ and see if it’s satisfied. The key condition is therefore:

$E_P[-logQ[X\left]\right]\le H_Q\left(X\right)$

$Q$’s constraint is nonbinding iff that condition holds, so we can view $E_P[-logQ[X\left]\right]\le H_Q\left(X\right)$ as saying something conceptually like “The information about $X$ implicitly encoded in $Q$ is implied by the information about $X$ implicitly encoded in $P$” or, in the uniform case, “$S_P$ is a subset of $S_Q$”.

Now for an interesting check. If we’re going to think of this formula as analogous to a subset relationship, then we’d like to have transitivity: $A\subset B$ and $B\subset C$ implies $A\subset C$. So, do we have

($E_P[-logQ[X\left]\right]\le H_Q\left(X\right)$ and $E_Q[-logR[X\left]\right]\le H_R\left(X\right)$) implies $E_P[-logR[X\left]\right]\le H_R\left(X\right)$

?

Based on David’s quick computational check the answer is “no”, which makes this look a lot less promising, though I’m not yet fully convinced.