What’s special about information at a distance? Some examples of what I mean:

What information about a little patch of pixels in a photo is relevant to another little patch of pixels far away in the same photo?
Relationship between exact voltages in two transistors which are far apart on the same CPU
When planning a road trip from San Francisco to LA, which details of the route within SF are relevant to the details of the route within LA?

The unifying feature in all these examples is that we imagine high-level observables which are “far apart” within some low-level model, in the sense that interactions between any two observables are mediated by many layers of hidden variables. In an undirected causal model, we could draw the picture like this:

There’s a handful of observed variables (black nodes), whose interactions are mediated by many layers (dotted groups) of unobserved variables (white nodes). Somewhere in the middle, everything connects up. We’re interested in the limit where the number of layers of hidden variables between any two observables is large.

In the limit of many hidden variables intermediating interactions, I suspect that there’s some kind of universal behavior going on - i.e. all models (within some large class) converge to behavior with some common features.

This post will give some hand-wavy, semi-mathematical arguments about what that behavior looks like. The main goal is to show some intuitive pictures; later posts can more carefully build out the details and boundaries of the phenomena sketched here.

Outline:

Background intuition on information distributed across variables, which is not contained in any one variable. Intuitively, it seems like this information should be quickly wiped out in large, noisy systems.
How to quantify distributed information.
Characterization of systems with zero distributed information.
Quick test: in a large system of normal variables, do we actually see the expected behavior? We do, and it looks a lot like a prototypical phase change in complex systems theory.

Intuition on Distributed Information

Fun fact: with three random variables $X$ , $Y$ , $Z$ , it’s possible for $X$ to contain zero information about $Z$ , and $Y$ to contain zero information about $Z$ , but for the pair $(X, Y)$ together to contain nonzero information about $Z$ .

Classic example: $X$ and $Y$ are u...