I think dust theory is wrong in the most permissive sense: there are physical constraints on what computations (and abstractions) can be like. The most obvious one is "things that are not in each others lightcone can't interact" and interaction is necessary for computation (setting aside acausal trades and stuff, which I think are still causal in a relevant sense, but don't want to get into rn). But there are also things like: information degrades over distance (roughly the number of interactions, i.e., telephone theorem) and so you'd expect "large" computations to take a certain shape, i.e., to have a structure which supports this long-range communication such as e.g., wires. 

More than that, though, I think if you disrespect the natural ordering of the environment you end up paying thermodynamic costs. Like, if you take the spectrum of visible light, ordered from ~400 to 800 nm and you just randomly pick wavelengths and assign them to colors arbitrarily (e.g., "red" is wavelengths 505, 780, 402, etc.), then you have to pay more cost to encode the color. Because, imo, the whole point of abstractions is that they're strategically imprecise. I don't have to model the exact wavelengths of the color red, it's whatever is in the range ~600-800, and I can rely on averages to encode that well enough. But if red is wavelengths 505, 780, 402, etc., now averages won't help, and I need to make more precise measurements. Precision is costly: it uses more bits, and bits have physical cost (e.g., Landauer's limit). 

I guess you could argue that someone else might go and see the light spectrum differently, i.e., what looks like wavelengths 505 vs 780 to us looks like wavelengths 505 vs 506 to them? But without a particular reason to think so it seems like a general purpose counterargument to me. You could always say that someone would see it differently—but why would they?     

I'd be very interested in a concrete construction of a (mathematical) universe in which, in some reasonable sense that remains to be made precise, two 'orthogonal pattern-universes' (preferably each containing 'agents' or 'sophisticated computational systems') live on 'the same fundamental substrate'. One of the many reasons I'm struggling to make this precise is that I want there to be some condition which meaningfully rules out trivial constructions in which the low-level specification of such a universe can be decomposed into a pair $(s_1,s_2)$ such that $s_1$ and $s_2$ are 'independent', everything in the first pattern-universe is a function only of $s_1$, and everything in the second pattern-universe is a function only of $s_2$. (Of course, I'd also be happy with an explanation why this is a bad question :).)

Stepping back from the physical questions, we can also wonder why generalization works in general without reference to a particular physical model of the world. And we find, much like you have, that generalization works because it is contingent on the person doing the generalizing.

In philosophy we talk about this through the problem of induction, which arises because the three standard options for justifying its validity are unsatisfactory: assuming it is valid as a matter of dogma, proving it is a valid method of finding the truth (which bumps into the problem of the criterion), or proving its validity recursively (i.e. induction works because it's worked in the past).

One of the standard approaches is to start from what would be the recursive justification and ground out the recursion by making additional claims, and a commonly needed claim is known as the uniformity principal, which says roughly that we should expect future evidence to resemble past evidence (in Bayesian terms we might phrase this as future and past evidence drawing from the same distribution). But the challenge then becomes to justify the uniformity principal, and it leads down the same path you've explored here in your post, finding that ultimately we can't really justify it except if we privilege our personal experiences of finding that each new moment seems to resemble the past moments we can recall.

This ends up being the practical means by which we are able to justify induction (i.e. it seems to work when we've tried it), but also does nothing to guarantee it would work in another universe or even outside our Hubble volume.

This sounds a lot like dust theory, although it’s an angle on it I hadn’t seen sketched: You, as a specific self-preserving macrostate partition, pick out similar macrostate partitions (that share information with you). And this is always possible, no matter which macrostate partition you are (even if the resulting partitions look, to our partition-laden eyes, more chaotic).

This sounds a bit like Wolfram's grander 'ruliad' multiverse, although I'm not sure if he claims that every ruliad must have observers, no matter how random-seeming... It seems like a stretch to say that. After all, while you could always create a mapping between random sequences of states and an observer, by a giant lookup table if nothing else, this mapping might exceed the size of the universe and be impossible, or something like that. I wonder if one could make a Ramsey-theory-style argument for more modest claims?

Is it not possible that, inside that randomly-seeming universe, an infinitude of macroscopic patterns exist (analogous to us), complex enough to track the similar macroscopic patterns they care about (analogous to our pressure), but written in a partitional language we don't discern?

Is homomorphic encryption an example of the thing you are talking about?

Eliezer has a defense of there being "objectively correct" macrostates / categories. See Mutual Information, and Density in Thingspace. He concludes:

And the way to carve reality at its joints, is to draw your boundaries around concentrations of unusually high probability density in Thingspace.

The open problems with this approach seem to be that it requires some objective notion of probability, and that there is an objectively preferred way of defining "thingspace". (Regarding the latter, I guess the dimensions of thingspace should fulfill some statistical properties, like being as probabilistically independent as possible, or something like that.) Otherwise everyone could have their own subjective probability and their own subjective thingspace, and "carving reality at its joints" wouldn't be possible.

But it seems to me that techniques from unsupervised / self-supervised learning do suggest that there are indeed some statistical features that allow for some objectively superior clustering of data.

In the ML example, generalization won't work when approximating a function which is a completely random jumble of points.

Nice article, minor question. You seem to be treating random functions as qualitatively different from regular/some-flavor-of-deterministic ones (please correct if not the case). Other than in mathematical settings, I'm not sure how that works, since you would expect some random noise in (or on top of) the data you are recording (and feeding your model), and that same noise would contaminate that determinism. 

Also, when approximating a completely random jumble of points, can't you build models to infer the distribution from where those points are taken? I get it wont be as accurate when predicting but I fail to see why that's not an issue of degrees.