Natural latent maximizes sum of mutual information with each observable 

Claim: Given constraints on the entropy of a latent variable $\Lambda$, the redundancy and mediation errors are optimized exactly when the sum of mutual information ${\sum }_{i}I(X_i;\Lambda )$ is maximized.

Why is this useful?

• Early works on natural abstractions often talk about throwing away information while maintaining predictive power, but how should predictive power be measured? If you have multiple observables, you want to maximize predictive power for each of the observables you care about. In particular, you want your latent variable to include “shared information” that allows you to predict multiple variables at once. This is similar to maximizing ${\sum }_{i}I(X_i;\Lambda )$ while minimizing  $H\left(\Lambda \right)$ (as we’re trying to throw away information).
• Maximizing the sum of mutual information is likely computationally easier than directly optimizing for the mediation and redundancy conditions, so this framing could be helpful for anyone trying to find natural latents computationally 

Mediation

Mediation is one of the main conditions for naturality of latent variables. Intuitively, an (approximate) mediator $\Lambda$ captures (approximately) all the correlation between a collection of observables $X_1\dots X_n$. We will prove that a latent variable $\Lambda$ is an (optimal) approximate mediator $⟺$ it maximizes the sum of mutual information ${\sum }_{i}I(X_i;\Lambda )$ given constraint on entropy $H\left(\Lambda \right)\le k$
 

Intuition: To maximize the sum of mutual information ${\sum }_{i}I(X_i;\Lambda )$, it is beneficial to include information that is shared among multiple $X_i$ in $\Lambda$  (as that would increase multiple mutual information term). A mediator is a latent variable that captures all the shared information among the $X_i$s, so it will be selected for when maximizing ${\sum }_{i}I(X_i;\Lambda )$.

Proof:

1.  Note that the mediation error of $\Lambda$ is equivelant to the conditional total correlation $TC\left(X|\Lambda \right)=D_{KL}\left(P\right(X_1\dots X_n|\Lambda )\parallel P(X_1|\Lambda )\dots P(X_n|\Lambda \left)\right)$
2. We have  ${\sum }_{i}I(X_i;\Lambda )={\sum }_{i}H\left(X_i\right)-H\left(X_i|\Lambda \right)$, since each $H\left(X_i\right)$ term is fixed relative to $\Lambda$, maximizing the sum of mutual information is equivalent to minimizing the sum of conditional entropies ${\sum }_{i}H\left(X_i|\Lambda \right)$
3. We have  $D_{KL}\left(P\right(X_1\dots X_n|\Lambda )\parallel P(X_1|\Lambda )\dots P(X_n|\Lambda \left)\right)=TC\left(X|\Lambda \right)={\sum }_{i}H\left(X_i|\Lambda \right)-H\left(X|\Lambda \right)$, which means
   ${\sum }_{i}H\left(X_i|\Lambda \right)=H\left(X|\Lambda \right)+TC\left(X|\Lambda \right)=H\left(X\right)-I(X;\Lambda )+TC\left(X|\Lambda \right)\ge H\left(X\right)-H\left(\Lambda \right)+TC\left(X|\Lambda \right)$. For the last inequality we used $H\left(X\right)\ge I(X;\Lambda )$, we can obtain equality if $\Lambda$ is a deterministic function of $X$.  
4. Hence, if we have a constraint on entropy $H\left(\Lambda \right)\le k$, then we have the lower bound${\sum }_{i}H\left(X_i|\Lambda \right)\ge H\left(X\right)-H\left(\Lambda \right)+TC\left(X|\Lambda \right)\ge H\left(X\right)-k+TC\left(X|\Lambda \right)$. 
5. We can achieve equality on the lower bound by choosing a deterministic (w.r.t $X$) latent $\Lambda$ with entropy $H\left(\Lambda \right)=k$. For such $\Lambda$ we have ${\sum }_{i}H\left(X_i|\Lambda \right)=H\left(X\right)-k+TC\left(X|\Lambda \right)$, which means ${\sum }_{i}H\left(X_i|\Lambda \right)$ is minimized exactly when $TC\left(X|\Lambda \right)$ is minimized (as that's the only term which depends on $\Lambda$)
6. Since $TC\left(X|\Lambda \right)$ is nonnegative and equivalent to the mediation error, we conclude that ${\sum }_{i}H\left(X_i|\Lambda \right)$ is minimized exactly when the mediation error is minimized, and an exact mediator ($TC\left(X|\Lambda \right)=0$) with entropy $k$ always achieves minimal ${\sum }_{i}H\left(X_i|\Lambda \right)$ (hence maximal ${\sum }_{i}I(X_i;\Lambda )$) among latent variables with entropy $\le k$

Redundancy

The correspondence with the redundancy condition is quite simple: The sum of the redundancy errors are ${\sum }_{i}H\left(\Lambda |X_i\right)=nH\left(\Lambda \right)-{\sum }_{i}I(X_i;\Lambda )$, so if we have a constraint of the form $H\left(\Lambda \right)\ge k$, then we have ${\sum }_{i}H\left(\Lambda |X_i\right)=nH\left(\Lambda \right)-{\sum }_{i}I(X_i;\Lambda )\ge nk-{\sum }_{i}I(X_i;\Lambda )$, and the sum of redundancy errors are minimized exactly when ${\sum }_{i}I(X_i;\Lambda )$ is maximized and $H\left(\Lambda \right)=k$.

Putting it together

We've shown that given constraints on $H\left(\Lambda \right)$, both the mediation and the redundancy conditions are minimized exactly when the sum of mutual information ${\sum }_{i}I(X_i,\Lambda )$ is maximized, we can use this to simplify the search for natural latents, and while optimizing for this quantity there is no tradeoff between the redundancy and mediation errors.

However, note that mediation error increases as $H\left(\Lambda \right)$ decreases (the mediation error for the empty latent is simply total correlation), while the redundancy error increases with $H\left(\Lambda \right)$ (which is why we imposed $H\left(\Lambda \right)\le k$ for mediation but $H\left(\Lambda \right)\ge k$ for redundancy). So the entropy of the latent is exactly the parameter that represents the tradeoff between the mediation and redundancy errors.

In summary, we can picture a pareto-frontier of latent variables with maximal ${\sum }_{i}I(X_i;\Lambda )$ and different entropies, by ramping up the entropy of $\Lambda$ along the pareto-frontier we gradually increase the redundancy errors while reducing the mediation errors, and these are the only parameters relevant for latent variables that are pareto-optimal w.r.t the naturality conditions.