If you haven't already seen it, you might find this paper I wrote (preprint, in review) relevant to your interest in this topic, since I suspect the issue runs even deeper than technical assumptions down to philosophical assumptions; or more precisely, hinge assumptions one must adopt to deal with unknowability (or if you don't believe in unknowability, then just pragmatic assumptions made to deal with epistemic uncertainty).

I'm curious about your thoughts around the problems of embedded agency. My view is that most of what's meaningful about understanding embedded agency is that it exposes the problems of machine learning models that make strong assumptions about the world that don't hold up, such that the problems of embedded agency are the problems of making overly strong assumptions (cf. AIXI and the anvil problem). Is this what you are pointing to or were your trying to say something different; your words are short enough that there's some ambiguity to me about where you suspect the problem-causing assumptions lie.