Unfortunately, the claim that evolutionarily fine-tuned priors do all the work to prevent internal reward-hacking seems lacking to me, because in practice we are uncertain about our own feelings and preferences. We don't actually have locked-in, invariant preferences, and it's unclear to me how active inference explains this; preferences are usually encoded as priors over observations, but ironically these are never updated.

I don't think actinf proposes that all preferences are locked-in and invariant, just that the "deepest" priors---related to body temp., humidity, etc.---are (see Section 9). Imo if you're talking about a deep (hierarchical) actinf agent, then all the forward predictions are kinda-preferences, to varying degrees; the slower to update, deeper layers are more preference-y and the faster to update layers closer to sensory input are more belief-y. There's some interesting discussion of this here.

Active inference thus implicitly assumes agents to be consistently, definitively settled on their preferences.

So with that in mind, I think I'd disagree with this and agree with something more like "active inference assumes agents are settled in the preferences that are necessary to keep them alive, but not in the preferences that are necessary to bring those states about."

That said, though, I think your overall idea is interesting. If you're thinking about subagents and superagents in terms of actinf, you might want to check out this paper.

Thanks for writing this post! I am glad someone doing alignment is paying attention to Markov blankets.

Not everything that is statistically couched from the outside world by a boundary is sensibly described as an agent. For instance, a rock is a self-organised entity with reasonably clear boundaries. Moreover, these boundaries are generally more robust than those of living beings, lasting considerably longer.

I don’t think a rock’s boundary is particularly robust. If I smash a rock with a hammer, bye-bye rock-blanket. If I smash you with a hammer, you will be injured but still exist. (You are in general better at keeping your internal states within a certain set—i.e. maintaining your non-equilibrium steady states—such that your blanket persists through time, and you do this by acting on the world.)

If you are interested in distinguishing people from rocks, I think https://arxiv.org/pdf/2210.12761 is worth a look. See in particular figure 2 on page 14.

I think this undersells the interest of seeing agency as a fractal-like phenomenon that doesn't fit a clear, discrete separation between agents and their environments.

https://www.pure.ed.ac.uk/ws/files/39856902/How_to_Knit_Your_Own_Markov_Blanket.pdf and https://arxiv.org/pdf/2502.21217 may be of interest.

And, not exactly what you are talking about, but somewhat related: I have recently been thinking about why Markov blankets change. Feel free to DM if you want to talk.