okay so I'm reading https://intelligence.org/2018/10/29/embedded-agents/.

it seems like this problem can't have existed? why does miri think this is a problem? it seems like it's only a problem if you ever thought infinite aixi was a valid model. it ... was never valid, for anything. it's not a good theoretical model, it's a fake theoretical model that we used as approximately valid even though we know it's catastrophically nonsensical; finite aixi begins to work, of course, but at no point could we actually treat alexei as an independent agent; we're all just physical circuits, alexei as much as emmy. to the degree that a physical system can exhibit intelligence, it's by budgeting compute. I feel like this article is the kind of thing one writes if your theoretical models don't even allow neural networks to exist, and I continue to think that the main reason the miri folks come up with nonsense is because they think you can make exact proofs about physics, when like, you can only ever even in principle make margin proofs, and those proofs are only valid to the degree you can trust that your verifier didn't have a random failure.

like, why do we need to have a model of intelligence as separate from physical systems? can't we just make our assertions directly about physical systems?

in terms of logical counterfactuals, it seems to me that a counterfactual is when a set of particles that have a representation for a larger set of particles assign their representation to values that the larger set of particles didn't take. so, being logical counterfactuals isn't special - all counterfactuals are to some degree logical.

if your decision theory of counterfactuals is a functional counterfactual, then it seems to me that that just means your counterfactual is a statement like "the universe, but any observations that match <x> get overwritten for consideration by attributes <y>". where <x> is a matcher on a series of physical states; if a system passes through physical states that match, then <x>'s output is <y> instead of the physical output. that doesn't seem complicated to me.

but maybe it only seems obvious because I've read discovering agents and in 2018 nobody had done that. maybe zac kenton solved this and I'm just echoing the solution and thinking that in retrospect the idea of being confused is confusing.

but I still feel this deep frustration with the question, like, why are you even asking that? it's clearly a necessarily useless question to ask "but aixi", because, everything was always going to be embedded, what else would it mean to be an agent besides to be a hunk of matter reacting to impulses? this feels like the kind of nonsense written by someone who doesn't have a mindset of searching for the most powerful algorithm and trying to build it, but instead of someone who has epistemic learned helplessness about the idea that it's possible to characterize what algorithms are worth running.

have you noticed you're not one algorithm, but a network of distributed microcomputers we call neurons, none of whom can ever entirely trust that they heard each other correctly?