Thank you for sharing this. I've long used the image (or metaphor) of the swirl of cream in a cup of coffee for this purpose, but have never expressed the idea as well.
This is a tangent but:
willing to metaphorically drop anvils on our heads
That might be a mistake people make (if not in that particular way*) but, I don't think we have any reason to believe AIXI would do such a thing.
AIXI doesn't do things 'just to find out what would happen'. It's an attempt at a utility maximizer. (Without a reason to believe there's a benefit in doing so, why would it do so? Wireheading might make sense - but anvils? Really? Why?)
*See below.
**Fail to prevent it from falling on it - maybe.
This post is mostly to share this quote that presents a metaphor describing the dance between map and territory (and the key insight of embeddedness) that was new to me, so let me share it first:
Consider a whirlpool. We can observe it, we can admire it, and we can talk about it. But we don't readily appreciate that what we glibly call "the whirlpool" is not a particular thing. The whirlpool, shifting in the lake or on the river, doesn't hold its form. It becomes shallower; the molecules of water are in continuous change. In no two moments do we find the same configuration of water molecules defining the surface or shape of the whirlpool. Yes we view "the whirlpool" as a thing[.]
We imagine enduring things, even as we think of "them" as fleeting. We hold to this notion at the expense of realizing that there's no actual thing there at all.
[...]
If you attend very carefully, it becomes apparent that everything is like the whirlpool—not a thing at all, but change itself. If we could speed up our perspective enough, even a mass of stone would appear, over the course of millennia, to change and disappear, just like a whirlpool. (And if we alter our perspective in the opposite way, we'd see the stone's constant molecular and atomic changes.)
But there's a far more subtle point here. It's only out of convention that I used the terms stone and whirlpool. Because, in fact, there is only change[.] There's no abiding thing called "stone" or "whirlpool" that changes. There's really no particular thing that's stone or whirlpool or you or me at all.
This is from chapter 40, "Ice Forming in Fire", of Buddhism Is Not What You Think: Finding Freedom Beyond Beliefs by Steve Hagen. The title of the chapter is from a perhaps less easily relatable metaphor from Dogen that Hagen liberally translates as "What is Reality? An icicle forming in fire".
I like the whirlpool metaphor because it makes clear how it is our minds bringing things into existence by finding useful patterns in the soup of reality and reifying those patterns into more persistent-seeming forms. Our minds readily trick us into thinking our maps, models, and ideas of what reality is like are more solid than they actually are. It's a quick and easy jump from observation to reification and then from reification to detachment of the map from the territory, paying more attention to our beliefs about what reality is than reality itself. And then we're quickly off behaving not much better than AIXI, willing to metaphorically drop anvils on our heads because we fail to realize how we're embedded in reality.