I’ve tried several times to summarize the core question my research is trying to tackle (and, indeed, I often think of research progress as a process of asking increasingly good core questions).^[1] This post gives the deepest version of that question I’ve found thus far: how should you relate to the parts of the world you can’t directly model or control?

Let me explain further in terms of a distinction between two perspectives. From the third person perspective you think of yourself as “outside” the world, looking in. You’re a good Bayesian, in that you have a set of mutually exclusive collectively exhaustive hypotheses. You choose actions by multiplying your credences by your utilities over those hypotheses, and you treat those actions as the only way you influence the world.

Some problems with the third person perspective (aka Cartesian or dualistic agency) were described in Scott and Abram’s sequence on embedded agency. One crucial issue is that most realistic environments contain other agents which are modeling you back, which means that your thoughts might affect the world via channels that aren’t just your actions. Game theory somewhat mitigates this problem, but only in the very specific case where all agents know (that all agents know, that all agents know…) that they’re in an equilibrium. A more subtle problem is that, even in the absence of other agents, modeling yourself as part of your environment can raise contradictions when you consider taking different possible actions.

Another key problem is that any realistic agent is too “small” to model most of the wider world. If we take this problem seriously enough, we end up at what I’ll call the first person perspective. From this perspective, the world is primarily a stream of sensory data—you don’t have a set of hypotheses that reliably carve it into mutually exclusive (let alone collectively exhaustive) possible worlds. Your main job is to learn (partial, overlapping) concepts that help to predict and explain that data. Predictive processing is one framework which implicitly operates from the first-person perspective (though the full active inference framework is a bit more complicated); so does perceptual control theory.

You can think of babies as operating from the first-person perspective; I also find it helpful to think about it as the perspective of individual cells. Each cell has some boundary w...

Steps towards a theory of coalitional agency

Steps towards a theory of coalitional agency