For translatability guarantees, we also want an answer for why agents have distinct concepts for different things, and the criteria for carving up the world model into different concepts. My sketch of an answer is that different hypotheses/agents will make use of different pieces of information under different scenarios, and having distinct reference handles to different types of information allows the hypotheses/agents to access the minimal amount of information they need.

For environment structure, we'd like an answer for what it means for there to be an object that persists through time, or for there to be two instances of the same object. One way this could work is to look at probabilistic predictions of an object over its Markov blanket, and require some sort of similarity in probabilistic predictions when we "transport" the object over spacetime

I'm less optimistic about the mind structure foundation because the interfaces that are the most natural to look at might not correspond to what we call "human concepts", especially when the latter requires a level of flexibility not supported by the former. For instance, human concepts have different modularity structures with each other depending on context (also known as shifting structures), which basically rules out any simple correspondence with interfaces that have fixed computational structure over time. How we want to decompose a world model is an additional degree of freedom to the world model itself, and that has to come from other ontological foundations.

Seems like the main additional source of complexity is that each interface has its own local constraint, and the local constraints are coupled with each other (but lower-dimensional than parameters themselves); whereas regular statmech usually have subsystems sharing the same global constraints (different parts of a room of ideal gas are independent given the same pressure/temperature etc)

To recover the regular statmech picture, suppose that the local constraints have some shared/redundant information with each other: Ideally we'd like to isolate that redundant/shared information into a global constraint that all interfaces has access to, and we'd want the interfaces to be independent given the global constraint. For that we need something like relational completeness, where indexical information is encoded within the interfaces themselves, while the global constraint is shared across interfaces.

IIUC there are two scenarios to be distinguished: 

One is that the die has bias p unknown to you (you have some prior over p) and you use i.i.d flips to estimate bias as usual & get maxent distribution for a new draw. The draws are independent given p but not independent given your priors, so everything works out. 

The other is that the die is literally i.i.d over your priors. In this case everything from your argument routes through: Whatever bias\constraint you happen to estimate from your outcome sequence doesn't say anything about a new i.i.d draw because they're uncorrelated, the new draw is just another sample from your prior

I think steering is basically learning, backwards, and maybe flipped sideways. In learning, you build up mutual information between yourself and the world; in steering, you spend that mutual information. You can have learning without steering---but not the other way around---because of the way time works.

Alternatively, for learning your brain can start out in any given configuration, and it will end up in the same (small set of) final configuration (one that reflects the world); for steering the world can start out in any given configuration, and it will end up in the same set of target configurations

It seems like some amount of steering without learning is possible (open-loop control), you can reduce entropy in a subsystem while increasing entropy elsewhere to maintain information conservation

Nice, some connections with why are maximum entropy distributions so ubiquitous:

• If your system is ergodic, time average=ensemble average. Hence expected constraints can be estimated via following your dynamical system over time
• If your system follows the second law, then entropy increases subject to the constraints

So the system converges to the maxent invariant distribution subject to constraint, which is why langevin dynamics converges to the Boltzmann distribution, and you can estimate equilibrium energy by following the particle around

In particular, we often use maxent to derive the prior itself (=invariant measure), and when our system is out of equilibrium, we can then maximize relative entropy w.r.t our maxent prior to update our distribution

Congratulations!

I would guess the issue with KL relates to the fact that a bound on $D_{KL}(P\parallel Q)$ permits situations where $P(X=x)$ is small but $Q(X=x)$ is large (as we take the expectation under $P$), whereas JS penalizes both ways.

In particular, in the original theorem on resampling using KL divergence, the assumption bounds KL w.r.t the joint distribution $P(X,\Lambda )$, so there may be situation where the resampled probability$Q(X=x,\Lambda =\lambda )=P(X=x)P(\Lambda =\lambda |X_2=x_2)$ is large but $P(X=x,\Lambda =\lambda )$ is small. But the intended conclusion bounds the KL under the resampled distribution $Q$, so the error on the values $(X=x,\Lambda =\lambda )$ would be weighted much more under $Q$ than under $P$. Since we're taking expectation under $Q$ for the conclusion, the bound on the other resampling error under $P$ becomes insufficient.

Would this still give us guarantees on the conditional distribution $P\left(X|\Lambda \right)$?

E.g. Mediation: $D_{KL}\left(P\right(X_1,X_2,\Lambda )\parallel P(X_1|\Lambda \left)P\right(X_2|\Lambda \left)P\right(\Lambda \left)\right)$$=D_{KL}\left(P\right(X_1,X_2|\Lambda \left)P\right(\Lambda )\parallel P(X_1|\Lambda \left)P\right(X_2|\Lambda \left)P\right(\Lambda \left)\right)$$=D_{KL}\left(P\right(X_1,X_2|\Lambda )\parallel P(X_1|\Lambda \left)P\right(X_2|\Lambda \left)\right)$

is really about the expected error conditional on individual values of $\Lambda$, & it seems like there are distributions with high mediation error but low error when the latent is marginalized inside $D_{KL}$, which could be load-bearing when the agents cast out predictions on observables after updating on $\Lambda$

The current theory is based on classical hamiltonian mechanics, but I think the theorems apply whenever you have a markovian coarse-graining. Fermion doubling is a problem for spacetime discretization in the quantum case, so the coarse-graining might need to be different. (E.g. coarse-grain the entire hilbert space, which might have locality issues but probably not load-bearing for algorithmic thermodynamics)

On outside view, quantum reduces to classical (which admits markovian coarse-graining) in the correspondence limit, so there must be some coarse-graining that works

I also talked to Aram recently & he's optimistic that there's an algorithmic version of the generalized heat engine where the hot vs cold pool correspond to high vs low k-complexity strings. I'm quite interested in doing follow-up work on that

The continuous state-space is coarse-grained into discrete cells where the dynamics are approximately markovian (the theory is currently classical) & the "laws of physics" probably refers to the stochastic matrix that specifies the transition probabilities of the discrete cells (otherwise we could probably deal with infinite precision through limit computability)