This post is a comment on Natural Latents: Latent Variables Stable Across Ontologies by John Wentworth and David Lorell. It assumes some familiarity with that work and does not attempt to explain it. Instead, I present an alternative proof that was developed as an exercise to aid my own understanding. While the original theorem and proof are written in the language of graphical models, mine instead uses the language of information theory. My proof has the advantage of being algebraically succinct, while theirs has the advantage of developing the machinery to work directly with causal structures. Very often, seeing multiple explanations of a fact helps us understand it, so I hope someone finds this post useful.

Specifically, we are concerned with their Theorem 1 (Mediator Determines Redund): both the older Iliad 1 version for stochastic latents, and the newer arXiv version for deterministic latents. I will translate each theorem into the language of information theory: Wentworth & Lorell's assumptions will imply mine, while their conclusions will be equivalent to mine. The equivalences follow from the d-separation criterion and the fact that independence is equivalent to zero mutual information.

In the restated new theorem, the latent variable $\Lambda$ is a mediator between subsets A and B of the data, meaning that it contains essentially all of the information in common between A and B, whereas the latent variable ${\Lambda }^{\prime }$ is a redund between A and B, meaning it essentially only contains information that is in common between A and B.^[1]

New Theorem 1 (deterministic latents)

Let A,B be disjoint subsets of {1,...,n}.

Suppose the random variables $X_1,\dots X_n,\Lambda ,{\Lambda }^{\prime }$ satisfy the following:

$\Lambda$ Mediation: $I(X_A:X_B\mid \Lambda )\le {ϵ}_{med}$,

${\Lambda }^{\prime }$ Redundancy: $H({\Lambda }^{\prime }\mid X_A)\le {ϵ}_{red}$ and $H({\Lambda }^{\prime }\mid X_B)\le {ϵ}_{red}$.

Then, $H({\Lambda }^{\prime }\mid \Lambda )\le {ϵ}_{med}+2{ϵ}_{red}$.

Proof

$H({\Lambda }^{\prime }\mid \Lambda )$

$=H({\Lambda }^{\prime }\mid X_B,\Lambda )+I({\Lambda }^{\prime }:X_B\mid \Lambda )$            by definition of conditional mutual information,

$\le H({\Lambda }^{\prime }\mid X_B)+I(X_A:X_B\mid \Lambda )+H({\Lambda }^{\prime }\mid X_A)$             by information theory inequalities,

$\le {ϵ}_{med}+2{ϵ}_{red}$                                                                       by Redundancy and Mediation.

Old Theorem 1 (stochastic latents)

Suppose the random variables $X_1,\dots X_n,\Lambda ,{\Lambda }^{\prime }$ satisfy the following:

Independent Latents: $I(\Lambda :{\Lambda }^{\prime }\mid X)\le {ϵ}_{ind}$,

$\Lambda$ Mediation: $I(X_j:X_{-j}\mid \Lambda )\le {ϵ}_{med}$ for all j,

${\Lambda }^{\prime }$ Redundancy: $I({\Lambda }^{\prime }:X_j\mid X_{-j})\le {ϵ}_{red}$ for all j.

Then, $I({\Lambda }^{\prime }:X\mid \Lambda )\le n({ϵ}_{ind}+{ϵ}_{med}+{ϵ}_{red})$.

Proof

First, we have

$I({\Lambda }^{\prime }:X_j\mid X_{-j})-I({\Lambda }^{\prime }:X_j\mid \Lambda ,X_{-j})$

$=I({\Lambda }^{\prime }:X_j:\Lambda \mid X_{-j})$                                  by definition of 3-way interaction information,

$=I({\Lambda }^{\prime }:\Lambda :X_j\mid X_{-j})$                                  by symmetry of 3-way interaction information,

$=I({\Lambda }^{\prime }:\Lambda \mid X_{-j})-I({\Lambda }^{\prime }:\Lambda \mid X_j,X_{-j})$

$\ge -I({\Lambda }^{\prime }:\Lambda \mid X_j,X_{-j})$

$\ge -{ϵ}_{ind}$                                                            by Independent Latents.

Therefore,

$I({\Lambda }^{\prime }:X_j\mid \Lambda )$

$\le I\left(\right({\Lambda }^{\prime },X_{-j}):X_j\mid \Lambda )$

$=I(X_{-j}:X_j\mid \Lambda )+I({\Lambda }^{\prime }:X_j\mid \Lambda ,X_{-j})$           by mutual information chain rule,

$\le I(X_{-j}:X_j\mid \Lambda )+I({\Lambda }^{\prime }:X_j\mid X_{-j})+{ϵ}_{ind}$     by the above derivation,

$\le {ϵ}_{ind}+{ϵ}_{med}+{ϵ}_{red}$                                                by Mediation and Redundancy.

The result now follows by summing over all j=1,...,n.

1. ^{^}

   Since probabilistic models are often only defined in terms of a latent structure, you might find it philosophically suspect to impose a joint distribution on all variables including the latents. If so, feel free to replace the random variables with their specific instantiations: the derivations go through almost identically with Kolmogorov complexity and algorithmic mutual information replacing the Shannon entropy and mutual information, respectively.