There is another sense in which I would not want to say that there is any particular hierarchy between natural/unnatural/rational constraints.

I think there's a lot to unpack here. I'm going to give it a preliminary go, anticipating that it's likely to be to a bit all over the place. The main thread I want to pull is what it means to impose a particular hierarchy between the constraints, and then see how this leads to many possible hierarchies in such a way that it feels like no particular hierarchy is privileged. 

From a "natural" point of view, which privileges physical time, individuation is something that must be explained - a puzzle which is at the heart of the mystery of the origin of life. From this point of view, "rationality" or "coherence" is also something that must be explained (which is what Richard Ngo is gesturing out in his comment / post). 

From a "rational" point of view, we can posit abstract criteria which we want our model of agency to fulfil. For instance Logical Induction (Garrabrant et al. 2016), takes a formalisation of the following desideratum (named "Approximate Inexploitability", or "The Logical Induction Criterion"): "It should not be possible to run a Dutch book against a good reasoner in practice." (ibid, p.8, p.14), and then constructs an agent entirely within logic from this. Something like "rationality" or "coherence" is assumed (for well argued reasons), and the structure of agency is deduced from there. This kind of move is also what underpins selection theorems. In my view, individuation also needs to be explained here, but it's often simply assumed (much like it is in most of theoretical biology). 

The "unnatural" point of view is much more mysterious to us. When I use the term, I want to suggest that individuation can be assumed, but physical time becomes something that must be explained. This is a puzzle which is touched on in esoteric areas of physics (e.g. "A smooth exit from eternal inflation?" - Hawking and Hertog 2018), and consciousness science (e.g. "A measure for intrinsic information" - Barbosa et al. 2020), and discussed in "religious" or "spiritual" contexts, but in reality very poorly understood. I think you gesture at a really interesting perspective on this by relating it to "thinghood" in active inference - but to me this misses a lot of what makes this truly weird - the reasons I decided to label it "unnatural" in the first place. 

It's deeply confusing to me at this stage how the "unnatural" point of view relates to the "rational" one, I'd be curious to hear any thoughts on this, however speculative. I do, however, think that there is a sense in which none of the three hierarchies I'm gesturing at in this comment are "the real thing" - they more feel like prisms through which we can diffract the truth in an attempt to break it down into manageable components. 

"Social" is slightly too coarse-grained a tag. The thing we're actually interested in is "whether successfully predicting the behaviour of other members of its own species is a strong selection pressure". Social collaboration is one way this happens - another seems to be "deception" arms races (such as corvids stealing and hiding things from each other), or specific kinds of mating rituals. It also depends on the relative strength of other selection pressures - in most cases highly intelligent creatures also seem to have developed a "slack" in resources they can devote to intelligence (e.g. humans cooking food). 

This does seem to hold for cephalopods - a strong datapoint for which being their highly sophisticated forms of communication (e.g. video below).