This seems like an interesting problem! I've been thinking about it a little bit but wanted to make sure I understood before diving in too deep. Can I see if I understand this by going through the biased coin example?
Suppose I have 2^5 coins and each one is given a unique 5-bit string label covering all binary strings from 00000 to 11111. Call the string on the label Λ.
The label given to the coin indicates its 'true' bias. The string 00000 indicates that the coin with that label has p(heads)=0. The coin labelled 11111 has p(heads)=1. The ‘true’ p(heads) increases in equal steps going up from 00000 to 00001 to 00010 etc. Suppose I randomly pick a coin from this collection, toss it 200 times and call the number of heads X_1. Then I toss it another 200 times and call the number of heads X_2.
Now, if I tell you what the label on the coin was (which tells us the true bias of the coin), telling you X_1 would not give you any more information to help you guess X_2 (and vice versa). This is the first Natural Latent condition (Λ induces independence between X_1 and X_2). Alternatively, if I didn’t tell you the label, you could estimate it from either X_1 or X_2 equally well. This is the other two diagrams.
I think that the full label Λ will be an approximate stochastic natural latent. But if we consider only the first bit[1] of the label (which roughly tells us whether the bias is above or below 50% heads) then this bit will be a deterministic natural latent because with reasonably high certainty, you can guess the first bit of Λ from X_1 or X_2. This is because the conditional entropy H(first bit of Λ|X_1) is low. On the other hand H(Λ | X_1) will be high. If I get only 23 heads out of 200 tosses, I can be reasonably certain that the first bit of Λ is a 0 (ie the coin has a less than 50% of coming up heads) but can't be as certain what the last bit of Λ is. Just because Λ satisfies the Natural Latent conditions within ϵ, this doesn’t imply that Λ satisfies H(Λ|X1)<ϵ. We can use X_1 to find a 5-bit estimate of Λ, but most of the useful information in that estimate is contained in the first bit. The second bit might be somewhat useful, but its less certain than the first. The last bit of the estimate will largely be noise. This means that going from using Λ to using ‘first bit of Λ’ doesn’t decrease the usefulness of the latent very much, since the stuff we are throwing out is largely random. As a result, the ‘first bit of Λ’ will still satisfy the natural latent conditions almost as well as Λ. By throwing out the later bits, we threw away the most 'stochastic' bits, while keeping the most 'latenty' bits.
So in this case, we have started from a stochastic natural latent and used it to construct a deterministic natural latent which is almost as good. I haven’t done the calculation, but hopefully we could say something like ‘if Λ satisfies the natural latent conditions within ϵ then the first bit of Λ satisfies the natural latent conditions within 2ϵ (or 3ϵ or something else)’. Would an explicit proof of a statement like this for this case be a special case of the general problem?
The problem question could be framed as something like: “Is there some standard process we can do for every stochastic natural latent, in order to obtain a deterministic natural latent which is almost as good (in terms of \epsilon)”. This process will be analogous to the ‘throwing away the less useful/more random bits of \lambda’ which we did in the example above. Does this sound right?
Also, can all stochastic natural latents can be thought of as 'more approximate' deterministic latents? If a latent satisfies the the three natural latents conditions within ϵ1, we can always find a (potentially much bigger) ϵ2 such that this latent also satisfies the deterministic latent condition, right? This is why you need to specify that the problem is showing that a deterministic natural latent exists with 'almost the same' ϵ. Does this sound right?
I'm going to talk about the 'first bit' but an equivalent argument might also hold for the 'first two bits' or something. I haven't actually checked the maths.
I'd have to run the numbers to check that 200 flips is enough to give a high-confidence estimate of Λ
It isn't enough. See plot. Also, 200 not being enough flips is part of what makes this interesting. With a million flips, this would pretty much just be the exact case. The fact that it's only 200 flips gives you a tradeoff in how many label_bits to include.
[
EDIT Aug 22 2025: This bounty is now closed! We havesolved it ourselvesand will accept a 500$ bonus for doing so.Nope, nevermind.]Our posts on natural latents have involved two distinct definitions, which we call "stochastic" and "deterministic" natural latents. We conjecture that, whenever there exists a stochastic natural latent (to within some approximation), there also exists a deterministic natural latent (to within a comparable approximation). We are offering a $500 bounty to prove this conjecture.
Some Intuition From The Exact Case
In the exact case, in order for a natural latent to exist over random variables X1,X2, the distribution has to look roughly like this:
Each value of X1 and each value of X2 occurs in only one "block", and within the "blocks", X1 and X2 are independent. In that case, we can take the (exact) natural latent to be a block label.
Notably, that block label is a deterministic function of X.
However, we can also construct other natural latents for this system: we simply append some independent random noise to the block label. That natural latent is not a deterministic function of X; it's a "stochastic" natural latent.
In the exact case, if a stochastic natural latent exists, then the distribution must have the form pictured above, and therefore the block label is a deterministic natural latent. In other words: in the exact case, if a stochastic natural latent exists, then a deterministic natural latent also exists. The goal of the $500 bounty is to prove that this still holds in the approximate case.
Approximation Adds Qualitatively New Behavior
If you want to tackle the bounty problem, you should probably consider a distribution like this:
Distributions like this can have approximate natural latents, while being qualitatively different from the exact picture. A concrete example is the biased die: X1 and X2 are each 500 rolls of a biased die of unknown bias (with some reasonable prior on the bias). The bias itself will typically be an approximate stochastic natural latent, but the lower-order bits of the bias are not approximately deterministic given X (i.e. they have high entropy given X).
The Problem
"Stochastic" Natural Latents
Stochastic natural latents were introduced in the original Natural Latents post. Any latent Λ over random variables X1,X2 is defined to be a stochastic natural latent when it satisfies these diagrams:
... and Λ is an approximate stochastic natural latent (with error ϵ) when it satisfies the approximate versions of those diagrams to within ϵ, i.e.
ϵ≥DKL(P[X,Λ]||P[Λ]P[X1|Λ]P[X2|Λ])
ϵ≥DKL(P[X,Λ]||P[X2]P[X1|X2]P[Λ|X1])
ϵ≥DKL(P[X,Λ]||P[X1]P[X2|X1]P[Λ|X2])
Key thing to note: if Λ satisfies these conditions, then we can create another stochastic natural latent Λ′ by simply appending some random noise to Λ, independent of X. This shows that Λ can, in general, contain arbitrary amounts of irrelevant noise while still satisfying the stochastic natural latent conditions.
"Deterministic" Natural Latents
Deterministic natural latents were introduced in a post by the same name. Any latent Λ over random variables X1,X2 is defined to be a deterministic natural latent when it satisfies these diagrams:
... and Λ is an approximate deterministic natural latent (with error ϵ) when it satisfies the approximate versions of those diagrams to within ϵ, i.e.
ϵ≥DKL(P[X,Λ]||P[Λ]P[X1|Λ]P[X2|Λ])
ϵ≥H(Λ|X1)
ϵ≥H(Λ|X2)
See the linked post for explanation of a variable appearing multiple times in a diagram, and how the approximation conditions for those diagrams simplify to entropy bounds.
Note that the deterministic natural latent conditions, either with or without approximation, imply the stochastic natural latent conditions; a deterministic natural latent is also a stochastic natural latent.
Also note that one can instead define an approximate deterministic natural latent via just one diagram, and this is also a fine starting point for purposes of this bounty:
What We Want For The Bounty
We'd like a proof that, if a stochastic natural latent exists over two variables X1,X2 to within approximation ϵ, then a deterministic natural latent exists over those two variables to within approximation roughly ϵ. When we say "roughly", e.g. 2ϵ or 3ϵ would be fine; it may be a judgement call on our part if the bound is much larger than that.
We're probably not interested in bounds which don't scale to zero as ϵ goes to zero, though we could maybe make an exception if e.g. there's some way of amortizing costs across many systems such that costs go to zero-per-system in aggregate (though we don't expect the problem to require those sorts of tricks).
Bounds should be global, i.e. apply even when ϵ is large. We're not interested in e.g. first or second order approximations for small ϵ unless they provably apply globally.
We might also award some fraction of the bounty for a counterexample. That would be much more of a judgement call, depending on how thoroughly the counterexample kills hope of any conjecture vaguely along these lines.
In terms of rigor and allowable assumptions, roughly the level of rigor and assumptions in the posts linked above is fine.
Why We Want This
Deterministic natural latents are a lot cleaner both conceptually and mathematically than stochastic natural latents. Alas, they're less general... unless this conjecture turns out to be true, in which case they're not less general. That sure would be nice.