This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
This probably generalizes beyond ~agent boundaries to general partitioning of "raw information blobs" into concepts/abstractions. You choose a way of making sense of the world that minimizes the expected future amount of [runtime correction by re-computing the abstraction from "raw sense data"] (minimally leaky abstractions).
(Well, this actually feels like a restatement of Markov/causal blankets, but the connection to your point stands, and it seems like an interesting restatement to me.)