Is the sum individual informativeness of two independent variables no more than their joint informativeness?

For a visualization, see information diagrams, and note that the central cell I(S; X; Y) must be non-positive (because I(S; X; Y) + I(X; Y | S) = I(X; Y) = 0).

Miss_Figg

Jul 08, 2019

We want to prove:

\int \int p_{x s} (x, s) log \frac{p_{x s} (x, s)}{p_{s} (s) p_{x} (x)} d x d s + \int \int p_{y s} (y, s) log \frac{p_{y s} (y, s)}{p_{s} (s) p_{y} (y)} d y d s \leq \int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s)}{p_{s} (s) p_{x y} (x, y)} d x d y d s

This can be rewritten as:

\int \int \int p_{x y s} (x, y, s) log \frac{p_{s x} (s, x)}{p_{s} (s) p_{x} (x)} d x d y d s + \int \int \int p_{x y s} (x, y, s) log \frac{p_{s y} (s, y)}{p_{s} (s) p_{y} (y)} d x d y d s \leq \int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s)}{p_{s} (s) p_{x} (x) p_{y} (y)} d x d y d s

After moving everything to the right hand side and simplifying, we get:

\int \int \int p_{x y s} (x, y, s) log \frac{p_{x y s} (x, y, s) p_{s} (s)}{p_{x s} (x, s) p_{y s} (y, s)} d x d y d s \geq 0

Now if we just prove that $q (x, y, s) = \frac{p_{x s} (x, s) p_{y s} (y, s)}{p_{s} (s)}$ is a probability distribution, then the left hand side is $K L (p_{x y s} (x, y, s), q)$ , and Kullback-Leibler divergence is always nonnegative.

Ok, q is obviously nonnegative, and its integral equals 1:

\int \int \int \frac{p_{x s} (x, s) p_{y s} (y, s)}{p_{s} (s)} d x d y d s = \int \frac{p_{s} (s) p_{s} (s)}{p_{s} (s)} d s = \int p_{s} (s) d s = 1

Q.e.d.

1 comment, sorted by

top scoring

Click to highlight new comments since: Today at 3:06 AM

[-]Jalex Stark5y10

Just for amusement, I think this theorem can fail when s, x, y represent subsystems of an entangled quantum state. (The most natural generalization of mutual information to this domain is sometimes negative.)

Moderation Log

[ Question ]

Is the sum individual informativeness of two independent variables no more than their joint informativeness?

10

New to LessWrong?

2 Answers sorted by top scoring

Jul 08, 2019

Jul 08, 2019

2 Answers sorted by
top scoring