Latent abstractions Bootlegged.

Let be random variables distributed according to a probability distribution $p$ on a sample space $Ω$ .

Defn. A (weak) natural latent of $X_{1}, . . ., X_{n}$ is a random variable $Λ$ such that

(i) $X_{i}$ are independent conditional on $Λ$

(ii) [reconstructability] $p (Λ = λ | X_{1}, . . .,_{i}, . . ., X_{n}) = p (Λ = λ | X_{1}, . . ., X_{n})$ for all $i = 1$

[This is not really reconstructability, more like a stability property. The information is contained in many parts of the system... I might also have written this down wrong]

Defn. A strong natural latent $Λ$ additionally satisfies $p (Λ | X_{i}) = p (Λ | X_{1}, . . ., X_{n})$

Defn. A natural latent is noiseless if ?

$H (Λ) = H (X_{1}, . . ., X_{n})$ ??

[Intuitively, $Λ$ should contain no independent noise not accoutned for by the $X_{i}$ ]

Causal states

Consider the equivalence relation on tuples $(x_{1}, . . ., x_{n})$ given $(x_{1}, . . ., x_{n}) \sim (x_{1}^{'}, . . ., x_{n}^{'})$ if for all $i = 1, . . ., n$ $p (X_{i} = x_{i} | x_{1}, . . .,_{i}, . . ., x_{n}) = p (X_{i} = x_{i} | x_{1}^{'}, . . ., {_{i}}^{'}, . . ., x_{n}^{'})$

We call the set of equivalence relation $Ω / \sim$ the set of causal states.

By pushing forward the distribution $p$ on $Ω$ along the quotient map $Ω ↠ Ω / \sim$

This gives a noiseless (strong?) natural latent $Λ$ .

Remark. Note that Wentworth's natural latents are generalizations of Crutchfield causal states (and epsilon machines).

Minimality and maximality

Let $X_{1}, . . ., X_{n}$ be random variables as before and let $Λ$ be a weak latent.

Minimality Theorem for Natural Latents. Given any other variable $N$ such that the $X_{i}$ are independent conditional on $N$ we have the following DAG

$Λ \to N \to {X_{i}}_{i}$

i.e. $p (X_{1}, . . ., X_{n} | N) = p (X_{1}, . . ., X_{n} | N, Λ)$

[OR IS IT for all $i$ ?]

Maximality Theorem for Natural Latents. Given any other variable $M$ such that the reconstrutability property holds with regard to $X_{i}$ we have

$M \to Λ \to {X_{i}}_{i}$

Some other things:

Weak latents are defined up to isomorphism?
noiseless weak (strong?) latents are unique
The causal states as defined above will give the noiseless weak latents
Not all systems are easily abstractable. Consider a multivariable gaussian distribution where the covariance matrix doesn't have a low-rank part. The covariance matrix is symmetric positive - after diagonalization the eigenvalues should be roughly equal.
Consider a sequence of buckets $B_{i}, i = 1, . . ., n$ and you put messages $m_{j}$ in two buckets $m_{j} \to B_{2 j}, B_{2 j + 1}$ . In this case the minimal latent has to remember all the messages - so the latent is large. On the other hand, we can quotient $B_{2 i}, B_{2 i + 1} \mapsto B_{i}^{'}$ : all variables become independent.

EDIT: Sam Eisenstat pointed out to me that this doesn't work. The construction actually won't satisfy the 'stability criterion'.

The noiseless natural latent might not always exist. Indeed consider a generic distribution $p$ on $2^{N}$ . In this case, the causal state cosntruction will just yield a copy of $2^{N}$ . In this case the reconstructavility/stability criterion is not satisfied.