Why Jensen-Shannon?

2mo

Background: The Ising Model

The Ising Model is a classic toy model of magnets. We imagine a big 2D or 3D grid, representing a crystal lattice. At each grid vertex $i$ , there’s a little magnetic atom with state $σ_{i}$ , which can point either up ( $σ_{i} = + 1$ ) or down ( $σ_{i} = - 1$ ). When two adjacent atoms point the same direction, their joint energy is lower than when they point different directions; atoms further apart don’t directly interact. So we write the energy function (aka Hamiltonian) as $H (σ) = - J \sum_{i adjacent to j} σ_{i} σ_{j}$ for some constant $J$ ; two adjacent atoms with the same direction contribute $- J$ energy, while two adjacent atoms with different directions contribute $+ J$ energy.

Read: Blue -> -1 or "cold" or "down" , Red -> +1 or "hot" or

... (read 1554 more words →)

Replying toEliezer's Unteachable Methods of Sanity

Eliezer's Unteachable Methods of Sanity

(Can you give one or more examples of what doing more fun things in your relationship looks like as opposed to worrying about preserving it?)

Replying toYou Are Much More Salient To Yourself Than To Everyone Else

You Are Much More Salient To Yourself Than To Everyone Else

Who are you, exactly?

Replying toCourtship Confusions Post-Slutcon

Courtship Confusions Post-Slutcon

I suspect mine isn't the majority view but the banter->sex "pipeline" is a misnomer. The point is the banter! For the usual reasons of making friends, having fun, breaking ice, as you say in another comment. And then as a side effect, if there's sexual compatibility, I might realize that Im not just-attracted to this person but actually want to go be physically intimate with them. (As I know you know since you've written about it elsewhere: there's a big difference between attraction and the prediction that hooking up will be fun.)

Again this may be idiosyncratic but IMO thinking of it as a pipeline at all is a major thing that screws... (read more)

Replying toNot A Love Letter, But A Thank You Letter

Not A Love Letter, But A Thank You Letter

I find myself suddenly less sure that you don't/can't experience the same companionate love that I do, which may or may not be the same as everyone else. Being on a team in the way you describe here, with the expectations and predictions of what that means as you describe here, "those little moment to moment coordinations," etc, seems like a typical sample from the distribution of what I write when someone asks me to try to describe what the value proposition of relationships/love is. It reads to me like a pretty fair description of what I mean when I say (an idealized form of) "companionship." I think my version might have... (read more)

Replying toEasy vs Hard Emotional Vulnerability

David Lorell3mo

Easy vs Hard Emotional Vulnerability

Might "uninhibited" or "unfiltered" be closer to the concept you're going for here?

An Analogue Of Set Relationships For Distributions

"But You'd Like To Feel Companionate Love, Right? ... Right?"

3mo

Here’s a conceptual problem David and I have been lightly tossing around the past couple days.

“A is a subset of B” we might visualize like this:

If we want a fuzzy/probabilistic version of the same diagram, we might draw something like this:

And we can easily come up with some ad-hoc operationalization of that “fuzzy subset” visual. But we’d like a principled operationalization.

Here’s one that I kinda like, based on maxent machinery.

Background Concept 1: $E [- l o g P [X]] \leq H_{P} (X)$ Encodes The Same Information About $X$ As $P$ Itself

First, a background concept. Consider this maxent problem:

${max}_{P^{'}} - \sum_{X} P^{'} [X] l o g P^{'} [X] s.t. - \sum_{X} P^{'} [X] l o g P [X] \leq - \sum_{X} P [X] l o g P [X]$

Or, more compactly

$maxent [X] s.t. E [- l o g P [X]] \leq H_{P} (X)$

In English: what is the maximum entropy distribution $P^{'}$ for which (the average number of bits used to encode a sample from $P^{'}$ using a code optimized for distribution $P$ ) is at most (the average... (read 816 more words →)

Replying to"But You'd Like To Feel Companionate Love, Right? ... Right?"

David Lorell3mo

(Written quickly, and therefore not compactly, in the shower. Apologies.)

I have suspected for a while that there is instrumental value in oxytocin/companionate-love/etc, beyond the terminal 'niceness' of it.

Consider alcohol (or other inhibition-lowering drugs):

Bars are, apparently, the standard place to meet up after work and relax in lots of cultures
Ensuring there is sufficient wine/alcohol at a dinner party (or parties of any sort) is IME considered a top priority
At parties, it is a meme that high schoolers are taught to fight that folks (especially the host) will frequently at least lightly suggest that you have a drink/other similar drug

Intuitively to me, "~everyone will have a better time here / this way compared to... (read 398 more words →)

•••

Toward Statistical Mechanics Of Interfaces Under Selection Pressure

The Zen Of Maxent As A Generalization Of Bayes Updates

3mo

Imagine using an ML-like training process to design two simple electronic components, in series. The parameters $θ^{1}$ control the function performed by the first component, and the parameters $θ^{2}$ control the function performed by the second component. The whole thing is trained so that the end-to-end behavior is that of a digital identity function: voltages close to logical 1 are sent close to logical 1, voltages close to logical 0 are sent close to logical 0.

Background: Signal Buffering

We’re imagining electronic components here because, for those with some electronics background, I want to summon to mind something like this:

This electronic component is called a signal buffer. Logically, it’s an identity function: it maps 0 to 0 and... (read 847 more words →)

Resampling Conserves Redundancy & Mediation (Approximately) Under the Jensen-Shannon Divergence

3mo

Jaynes’ Widget Problem^[1]: How Do We Update On An Expected Value?

Mr A manages a widget factory. The factory produces widgets of three colors - red, yellow, green - and part of Mr A’s job is to decide how many widgets to paint each color. He wants to match today’s color mix to the mix of orders the factory will receive today, so he needs to make predictions about how many of today’s orders will be for red vs yellow vs green widgets.

The factory will receive some unknown number of orders for each color throughout the day - $N_{r}$ red, $N_{y}$ yellow, and $N_{g}$ green orders. For simplicity, we will assume that Mr A starts out with a prior... (read 1977 more words →)

Replying toResampling Conserves Redundancy & Mediation (Approximately) Under the Jensen-Shannon Divergence

David Lorell3mo

Thanks for the check! I've added a collab containing my quick numerical stress tests, linked at the bottom of the post.

Resampling Conserves Redundancy & Mediation (Approximately) Under the Jensen-Shannon Divergence

David Lorell

3mo

Around two months ago, John and I published Resampling Conserves Redundancy (Approximately). Fortunately, about two weeks ago, Jeremy Gillen and Alfred Harwood showed us that we were wrong.

This proof achieves, using the Jensen-Shannon divergence ("JS"), what the previous one failed to show using KL divergence (" $D_{K L}$ "). In fact, while the previous attempt tried to show only that redundancy is conserved (in terms of $D_{K L}$ ) upon resampling latents, this proof shows that the redundancy and mediation conditions are conserved (in terms of JS).

Why Jensen-Shannon?

In just about all of our previous work, we have used $D_{K L}$ as our factorization error. (The error meant to capture the extent to which a given distribution fails to factor according to... (read 1032 more words →)

Replying toResampling Conserves Redundancy (Approximately)

David Lorell4mo

(Update 7)

After some back and forth last night with an LLM^[1], we now have a proof of "chainability" for the redundancy diagrams in particular. (And have some hope that this will be most of what we need to rescue the stochastic->deterministic nat lat proof.)

(Theorem) Chainability of Redunds

Let P be a distribution over $X_{1}$ , $X_{2}$ , and $Λ$ .
Define:
$Q [X, Λ] := P [X] P [Λ | X_{1}]$
$S [X, Λ] := P [X] P [Λ | X_{2}] = P [X] \sum_{X_{1}} P [X_{1} | X_{2}] P [Λ | X]$ $R [X, Λ] := P [X] Q [Λ | X_{2}] = P [X] \sum_{X_{1}} P [X_{1} | X_{2}] P [Λ | X_{1}]$

Where you can think of Q as 'forcing' P into factorizing per one redundancy pattern: $X_{2} \to X_{1} \to Λ$ , S as forcing the other pattern: $X_{1} \to X_{2} \to Λ$ , and R as forcing one after the other: first $X_{2} \to X_{1} \to Λ$ , and then $X_{1} \to X_{2} \to Λ$ .

The theorem states,
$D_{K L} (P | | R) \leq D_{K L} (P | | Q) + D_{K L} (P | | S)$ ,

Or in words: The error (in $D_{K L}$ from $P$ ) accrued by applying both factorizations to P, is bounded by the the sum of the errors accrued by applying each of... (read more)

Replying toResampling Conserves Redundancy (Approximately)

David Lorell4mo

(Update 5)

A conjecture we are working on which we expect to be generally useful beyond possibly rescuing the stoch->det proof that used to rely on the work in this post:

Chainability (Conjecture):
with $ϵ_{1} := D_{K L} (X_{1} \to X_{2} \to Λ)$ , $ϵ_{2} := D_{K L} (X_{1} \leftarrow Λ \to X_{2})$ ,
Define $Q [X, Λ] := P [X] P [Λ | X_{2}]$ , and $Q^{'} [X, Λ] := Q [X_{1} | Λ] Q [X_{2} | Λ] Q [Λ]$ .
Then, $D_{K L} (P | | Q) < n (ϵ_{1} + ϵ_{2})$

Here is a collab with a quick numerical test that suggests the bound holds (and that n=1, in this case).

(Note: The above as written is just one step of chaining, and ultimately we are hoping to show it holds for arbitrarily many steps, accumulating an associated number of epsilons as error.)

Replying toResampling Conserves Redundancy (Approximately)

David Lorell4mo

Natural Latents: Latent Variables Stable Across Ontologies

Do you have sympy code for the example noted at the bottom of the collab that claims a ratio of > 9.77 including the mediation ? I tried with the parameters you mention and am getting a ratio of ~3.4 (which is still a violation of previous expectations, tbc.)

Do-Divergence: A Bound for Maxwell's Demon

5mo

Background on where this post/paper came from

About a year ago, we wrote up a paper on natural latents for the ILLIAD proceedings. It was mediocre. The main shortcoming stemmed from using stochastic rather than deterministic natural latents, which give much less conceptually satisfying ontological stability guarantees; there was this ugly and confusing caveat on everything throughout the paper. We knew at the time that deterministic natural latents gave a much cleaner story, but we were more concerned about maintaining enough generality for our results to bind to reality than about getting the cleanest story.

Recently, we proved that existence of a stochastic natural latent implies existence of a deterministic natural latent (specifically under

... (read 5867 more words →)

120

(∃ Stochastic Natural Latent) Implies (∃ Deterministic Natural Latent)

6mo

Let’s start with the classic Maxwell’s Demon setup.

We have a container of gas, i.e. a bunch of molecules bouncing around. Down the middle of the container is a wall with a tiny door in it, which can be opened or closed by a little demon who likes to mess with thermodynamics researchers. Maxwell^[1] imagined that the little demon could, in principle, open the door whenever a molecule flew toward it from the left, and close the door whenever a molecule flew toward it from the right, so that eventually all the molecules would be gathered on the right side. That would compress the gas, and someone could then extract energy by allowing the... (read 755 more words →)

6mo

Our posts on natural latents have involved two distinct definitions, which we call "stochastic" and "deterministic" natural latents. We conjectured that, whenever there exists a stochastic natural latent (to within some approximation), there also exists a deterministic natural latent (to within a comparable approximation). Four months ago, we put up a bounty to prove this conjecture.

We've been bottlenecked pretty hard on this problem, and spent most of the last four months attacking it. At long last, we have a proof. As hoped, the proof comes with some qualitative new insights about natural latents, and we expect it will unbottleneck a bunch of future work. The main purpose of this post is to... (read 2608 more words →)

126

•••