The separation principle: where do beliefs and desires come from?

Fernando Rosas

TLDR: Psychology, economics, and other disciplines describe agents as systems driven by beliefs and desires. This post argues that the belief-desire view can be derived from classic theorems from optimal control and reinforcement learning. This suggests seeing beliefs and desires as properties of optimal policies rather than as assumptions from folk psychology.

Introduction

One way to think about agents is as "systems that act for reasons".^[1] This compact statement can be interpreted as encapsulating two key implications:

The notion of action assumes a boundary between agent and environment, so that the former can act on the latter.
The term reason captures two kinds of internal activity: motivations associated with how to achieve specific goals or outcomes, and beliefs regarding what the agent infers to be the current state of affairs.

In other words, an agent is a well-differentiated system that acts based on beliefs and desires. This view is compatible with perspectives that have been developed by various disciplines:

Behavioural science, which sees agency as goal-directed behaviour.
Economics, which treats agency as the ability to select policies to achieve an objective.
Cybernetics, which conceptualises agency as the ability to regulate the environment and keep it within a desired subset of states.

Instead of taking the working definition^[2]

agency = beliefs + desires

at face value, in this post I will discuss conceptual tools that can allow us to derive this construction from formal considerations. In particular, I will discuss the separation principle, which I believe has potential to provide a rigorous foundation for this conceptualisation. These ideas take inspiration from cybernetics and computational mechanics, but distinctly leverage classic results in optimal control theory and reinforcement learning.

Motivation — cybernetics.

A related line of thinking pertains to the (in)famous Good Regulator Theorem. This result, put forward by cyberneticists Conant and Ashby in 1970, suggests that an effective controller requires an internal model of what is being controlled. This makes intuitive sense: it is hard to predict a system that follows an intricate internal mechanism, and perhaps the only way to effectively control it is by understanding it.

Unfortunately, the original paper has been both a source of inspiration and confusion. Lots have been written trying to clarify it; to read more, see this post and this post, and also this manuscript. This post will follow these ideas in spirit, but not the formalisation as proposed there.

Motivation — computational mechanics.

Another related line of work is computational mechanics, which combines principles of information theory, theoretical computer science, and some sprinkles of statistical mechanics to describe the dynamics of prediction. Computational mechanics derives minimal structures required for optimal prediction, which take form in the -machine and -transducer. These ideas offer a rigorous explanation to (i) why optimal prediction requires specific computations, and (ii) what properties those computations need to satisfy. To read more, see this thesis or this paper.

Screenshot 2026-08-30 at 20.18.07.png — Figure adapted from Crutchfield & Feldman, *Regularities unseen, randomness observed: levels of entropy convergence*, arXiv:cond-mat/0102181

More recently, computational mechanics is providing effective tools to study the internal representations in deep learning. Moreover, while originally formulated to study prediction, computational mechanics also provides a promising foundation for understanding perception-action loops. The ideas developed here are heavily influenced by both the concepts and formalism of computational mechanics.

What is a separation principle?

Let's start by discussing what separation principles are, in general.

As a first approximation, consider a situation where we want to minimise the function . A direct calculation shows that

Hence, solving the problem for and separately does not yield the solution for . Put simply, one does not solve the problem by solving its parts separately.

A separation principle takes place when the divide-and-conquer strategy actually works, so one can solve a complicated optimisation problem by optimising various subcomponents separately. Concretely, a separation theorem guarantees that optimal performance can be achieved by solving sub-tasks in isolation.

Let me further illustrate this idea by presenting an important separation theorem at the core of information theory.

The communication problem studied by information theory is about how to send bits from source to destination without error. The tools that a communication engineer employs to face this problem are two types of coding techniques:

Source coding, which compresses the original signal by removing redundancy in order to transmit as few bits as necessary.^[3]
Channel coding corresponds to error-correcting codes, which adds redundancy to protect the information bits from noise.^[4]

Most people have heard of the following results by Claude Shannon:

The source coding theorem, which states that the shortest achievable average code length is given by the entropy of the source.
The noisy-channel coding theorem, which states that the maximal transmission rate that allows arbitrarily low decoding error is given by the mutual information between input and output symbols (maximised over input distributions).

However, these results would not be as useful in the absence of the third, less known result known as the source-channel separation theorem.^[5] This result states that, under relatively mild conditions,^[6] an optimal solution for the overall communication problem can be achieved by breaking the problem into two separate parts: compression and error correction. This means that an optimal communication strategy can be designed by a team separately working on compression and another working on error-correction, without requiring coordination between them. Needless to say, this makes the problem much more tractable!

The inference-control separation principle

Let us now review results from optimal control theory and reinforcement learning that provide conditions for separating inference and control. In doing this, I will use the non-standard term inference-control separation principle to group together results that are often presented independently of each other, despite having a lot in common.

Separation principle in optimal control theory

Let us consider the classic instantiation of separation in the context of optimal control theory. For this, consider a control setting made by three components:

a system to be controlled, whose state is described by ,
the measurements an agent has access to, described by , and
the actions .

For simplicity, we assume that the system evolves dynamically over discrete time via linear dynamics and noisy observations that can be described as

where and are matrices, is the system's initial condition, and and are independent white Gaussian noise terms.

The action is determined according to a policy that has no direct access to the state , but only to the sequence of measurements . Thus, we take to be a deterministic function of the sequence of past measurements and actions . The controller aims to minimise the quadratic cost

where , , and are matrices and is the length of the time window over which the optimisation takes place. This is known as the discrete-time LQG problem, as it involves Linear dynamics, a Quadratic loss function, and Gaussian variables — being the one setting where everything is exactly solvable.

Separation principle — To understand the solution of this problem, let us first consider the case of full observability, where . In this setting, the optimal controller can be shown to only depend on the current state of the system, and the optimal solution can be expressed as

where is the "optimal gain" matrix (whose formula can be found in standard textbooks). So, if the controller can directly see the state of the system, this directly tells what the optimal action should be.

What happens with partial observability? In this setting, the optimal controller can be written as

where is the same as above and is an estimation of the state of the system given by

In fact, is the result of Kalman filtering, which is how recursive Bayesian filtering looks like under Gaussian dynamics.^[7]

In summary, the optimal policy can be described as "estimate first, and then act as if your estimate were the actual state of the system". Thus, a separation theorem is taking place separating inference and control.

Separation principle in reinforcement learning

To understand what the separation principle looks like in reinforcement learning, let us slightly modify the nomenclature and notation:

the state of the system to be controlled is described by a discrete variable ,
the measurements that the agent has access to are described by , and
the actions are denoted by .

Let's assume that the variables follow a partially observed Markov decision process (POMDP), so that the environment dynamics are Markovian conditioned on the agent's action via a Markov kernel ,^[8] and the current observation depends on the current state of environment via a Markov kernel . The agent takes actions following a policy that maps sequences of previous actions and observations into distribution of next actions.

In this setting, the probability of observing a sequence of observations and actions is

where is a prior distribution over the environment's initial state. This is roughly the same as the control scenario studied above, but considering general discrete variables instead of Gaussian ones.

Instead of a loss function, consider a reward that the agent tries to maximise. More specifically, let us define the discounted future reward

where is the so-called discount factor. Then, the goal of the agent is to find a policy that maximises .

Separation principle — To solve this POMDP, let us first consider the Bayesian beliefs about the latent state . There are two ways to understand Bayesian beliefs:

As random variables given by the conditional expectation , where the trajectory is regarded as a random variable.
As probability distributions given by , which correspond to the above expectation when conditioned on a specific realisation of .

Bayesian beliefs have two key properties. First, as random variables, they contain all the information in that is useful for predicting (i.e., they are sufficient statistics). Second, as distributions, they can be efficiently updated via

where is a deterministic function, and hence the dynamics of are Markovian conditioned on (as they only depend on ). These facts can be used to build a new Markov decision process (MDP), known as belief MDP, defined by the following variables:

the states are the beliefs of the latent state of the POMDP,^[9]
actions are (same as in the POMDP),
the reward function is .

In contrast with the original problem, this MDP is fully observable (as the agent knows its own beliefs), and therefore can be solved using standard RL techniques. The crucial point is that combining the estimation of Bayesian beliefs with the optimal solution for the belief MDP gives rise to a solution of the full POMDP.^[10] Thus, a separation principle is at play separating inference (calculating beliefs) and control (solving the belief MDP).

Interim summary

Both results in LQG control and POMDPs give rise to the same solution: they show that the optimal controller for a partially observed system can be built by

first performing optimal prediction over the latent state of the system, and then
solving the control problem using that prediction as a state.

To further understand how the LQG and POMDP settings relate, let's discuss the shape of beliefs and the effect of actions (not essential for the rest of the post).

Beliefs are distributions, not point estimates.

In the LQG setting, the optimal controller does not need the full distribution of the prediction — the prediction is a Gaussian posterior, but the controller only uses the mean and not the variance. This property is known as certainty equivalence, which suggests to act as if a point estimate were the true state.

It would be tempting to describe beliefs as "best guesses" (e.g. maximum a posteriori), but certainty equivalence only holds because of the friendly properties of the LQG setup — a more general setting would not allow for this simplification.

In the more general POMDP case, the agent must carry the full belief distribution, not a point estimate. Nonetheless, separation survives in the Bayesian filter + belief-MDP form. Thus, "beliefs" in the agency sense are not mere best guesses but the full posterior, and only under special conditions do they collapse to a point estimate. More generally, the separation principle suggests to think of an agent's epistemic state not as what it thinks is true, but how its uncertainty is shaped.

The dual effect.

Actions do two things at once: they change the state of the system (control effect) and also change what the agent will subsequently be able to observe and thus learn (informational effect). Think of a robot with a noisy range sensor deciding whether to drive straight toward a goal or take a detour past a landmark: the detour is suboptimal for the control objective in isolation, but it sharpens the robot's position estimate, which improves every subsequent decision. This second channel — action shaping future belief — is what control theorists call dual effect.

Part of why LQG separation is so clean is that while the evolution of the Kalman mean estimate depends on past actions and observations, its covariance does not. This means that the agent's uncertainty about the state at any future time is fixed the moment the problem is specified, as no action-observation sequence can make it larger or smaller. This implies that actions have no informational effect: there is no possible epistemic bonus that can reduce uncertainty, no matter what the agent does. This also explains why certainty equivalence holds: given that the covariance of the posterior evolves in a predetermined manner, it carries no useful information for the controller.

This simplification does not hold in more general POMDP settings, where actions can influence future information as well as the environment. In the belief MDP construction, the "state" is the belief , and the transition kernel captures both effects at once: how the action moves the underlying environment and how it reshapes the posterior.

Implications

Let us now return to the question we started with: can the working definition

agency = beliefs + desires

be derived, rather than assumed? The separation theorems reviewed above suggest that, at least in a specific sense, the answer is yes.

Beliefs and desires as properties of solutions

Let's recapitulate what the separation principle delivers. We posed a single, monolithic optimisation problem: find a policy mapping histories into actions that minimise/maximise loss/reward. Nothing in this problem statement mentions beliefs, inference, or internal representations — the problem is stated in terms of pure behaviour. Yet, the solution naturally factorises into sub-components. Indeed, the above results state that the optimal policy can always be realised as the composition of two blocks:

An inference module, which compresses the history into a Bayesian belief about the latent state that can be recursively updated. This module knows about dynamics but is oblivious about loss/rewards.
A control module, which selects actions as a function of that belief alone. This module knows about loss/rewards but not about raw observations.

The interface between them is exactly the belief state, which is the output of the inference module and the input of the control module. Thus, the inference module can be interpreted as "where beliefs are made"; similarly, the control module can be interpreted as "where desires take place", as it is the only part of the policy that is directed by the reward/loss function. In this way, beliefs and desires are recovered not as assumptions about agents but as properties of solutions.

This is worth pausing on. The folk-psychological notions of belief and desire have been with us at least since Aristotle, and they are usually treated as primitives — as the ingredients one starts with when theorising about minds. What the separation principle offers is something different: not a definition of these terms but a derivation of them. Beliefs and desires appear here not because we built them in, but because the optimal solution to a problem stated in purely behavioural terms turns out to have that shape. Whether or not one takes folk psychology seriously as a theory of mind, this is at least a reason to think its central distinction is not arbitrary.

It is remarkable that this modularity arises from solving a single, monolithic optimisation problem. We have seen this shape before: it is precisely the payoff that made the source-channel separation theorem so valuable! There, an optimal communication system could be designed by a compression team and an error-correction team working independently, without coordination — the source coder needed to know nothing about the channel, and the channel coder needed to know nothing about the source.

The inference-control separation delivers the exact analogue for agency, conferring a form of compositional generalisation. Indeed, if the reward changes but the environment does not, only the control module needs to be updated; if the environment shifts but the goals remain, only the inference module needs revising. Thus, the two components can be developed, improved, and swapped independently.

Agents as cognitive light-cones

These results may also provide some formal ground to conceiving of agents as cognitive light-cones.^[11] The idea goes as follows: an agent is a system whose behaviour is both sensitive to what it can remember about the past (the past component of the cone) and what it can foresee (the future component of the cone).

Figure from Levin (2019), Frontiers in psychology, 10, 2688.

This idea suggests an operational way to assess the capabilities of an agent: measure the size of its light-cone. This could be done using tools from computational mechanics — for example, studying how deep into the past causal states go, and doing the same for retrodictive causal states (which build sufficient statistics from future to past).^[12]

The separation principle is normative, not descriptive

Please note that the separation principle says that an optimal agent can be built as inference + control, but it does not say that any given agent is built this way. In particular, a policy trained end-to-end by gradient descent is under no obligation to organise itself into a filter feeding a planner.

But the separation principle gives us something subtler: it tells us that the belief state is a sufficient statistic for optimal behaviour. This suggests that any agent approaching optimality must be computing something as informative as the Bayesian posterior (whether or not it is legible as such). In this way, the separation principle turns into a lens for interpretability: we should expect belief-like structure inside competent agents, and we can go looking for it. Recent work from colleagues at Simplex in finding belief-state geometry in the residual streams of transformers is, I think, an early vindication of this expectation. At XOR Labs we are currently investigating to what degree these results transfer to deep RL agents trained to do control tasks.

At this point, it is useful to distinguish two separate concerns:

Control problems / POMDPs can have multiple optimal policies. In these settings, the separation principle states that there exists one optimal policy that can be factored, but doesn't guarantee all do.^[13]
When only one optimal solution exists, the separation principle says this unique behaviour admits a realisation as filter + belief-controller — but it does not say that's the only realisation. Indeed, the same mapping from histories to next action can be implemented in different ways; for instance, in a finite setting the mapping can always be implemented as a large look-up table.

Regarding the second concern, there are various pieces of evidence suggesting that deep learning architectures have an implicit bias for simplicity, which makes them find simpler implementations of algorithms whenever they exist.^[14] Therefore, while it is often possible to implement a policy via a gigantic look-up table, there are reasons to believe that policies obtained via deep learning techniques will tend towards factorised ones.^[15]

Where does separation fail? As Shannon's source-channel separation theorem, the inference-control separation theorems hold under fairly general conditions — not relying on finite alphabets or stationarity. That said, for it to be useful one needs to assume that the agents under consideration have enough compute to build (quasi-)optimal policies. Under bounded rationality, model misspecification, or multi-agent interaction, the clean factorisation can break: what to represent starts depending on what you want, and inference becomes goal-driven.^[16] Many of the most interesting aspects of agency may come from the ways in which real agents deviate from the separation principle — but that is a topic for a future post.

Coda

Before concluding, let me ask: does the separation principle derive beliefs and desires from first principles?

Concretely, the separation principle offers the following:

It reveals that optimal policies can be factored into two separate modules.
The signal shared between the modules can be interpreted as beliefs, so that the first (inference) module can be interpreted as "where beliefs are made".
The second (control) module can be interpreted as "where desires take place", as it is the only part of the policy that is directed by the reward/loss function.

Thus, the separation principle provides formal ground for articulating the view of agents as "systems that act for reasons", displaying behaviour that is sensitive to both the past (what it remembers as part of its beliefs) and the future (due to foreseeing and planning to achieve goals).

However, one has to acknowledge that the separation principle assumes an optimisation problem which already contains a reward or cost function. Following this reasoning, one could argue that the desire or preference has not been really derived, as it has been baked in externally. More precisely, one can interpret the separation principle not as deriving why desires exist in the first place, but as explaining why they can be treated as separate entities from beliefs.^[17]

^{^}
There is a huge literature discussing various views on what agency is; if you want to read more, perhaps take a look into this and this paper and references therein.
^{^}
This post will take the boundary between agent and environment as given to focus on their internal activity. To read about boundaries, see these posts.
^{^}
Examples include Huffman and arithmetic coding.
^{^}
Examples include Hamming, Reed-Solomon, or convolutional codes.
^{^}
In contrast with the other two, this result doesn't have a wikipedia page...
^{^}
For technical details, see Cover & Thomas Chapter 7.13.
^{^}
For more details about the separation principle in LQR, see this paper. For a good introduction to Kalman and Bayesian filtering, see this book.
^{^}
denotes the space of probability distributions over the set .
^{^}
The transition probabilities can be computed directly; for formulas see this paper or wikipedia.
^{^}
For proofs, see this and this paper.
^{^}
I first saw this idea in this inspiring paper of Michael Levin.
^{^}
I'll expand on this in a future post.
^{^}
This is a real issue, but can be disregarded in some settings. For instance, "almost every" finite POMDP has a single optimal solution. Multiple optimal solutions in POMDPs are caused, for example, by ties in the optimal Q-function, which are broken by arbitrarily small perturbations on the reward.
^{^}
There have been numerous discussions about the simplicity biases of deep learning, see for example this post.
^{^}
For example, we have evidence that transformers find factored representations whenever they are available — see this paper.
^{^}
This is arguably where phenomena like motivated reasoning and wishful thinking become rational responses to bounded resources rather than mere biases. To read more, see this and this.
^{^}
For answering the why question, one may need to consider arguments based on natural selection, or perhaps approaches to agency based on the enactive tradition — as developed, for example, in this paper. See related discussions questioning utility functions in this post.

Under bounded rationality, model misspecification, or multi-agent interaction, the clean factorisation can break: what to represent starts depending on what you want, and inference becomes goal-driven.^[16]

In general, having beliefs about aspects of the world that you will never have desires about, and that are also causally disconnected from anything that you may ever have desires about, is a waste of scarce computational resources. Biological organism are generally evolved to primarily model the aspects of their environment needed to let them solve problems that matter to them (such as finding food and a mate, and avoiding predators) and not to think about things that don't affect them (such as, say, doing minerology on rocks). Humans are unusually generalist: our niche is so wide that it's really hard to predict what facts might later turn out to be useful, which is whey we figured out things like minerology, mining, smelting, and metalwork. Nevertheless, even as generalists, we're still particularly well adapted with mental capabilities useful for locating ripe edible fruit, avoiding large predators and venomous snakes and spiders, and gossiping about other members of our social group. For example, children are very interested in animals (and plants), especially large predators.

This is absolutely right! The framing presented here states that one way to express the optimal policy is via a factorisation whose first module computes an estimate of the full state of the environment. But, as you well say, this can be quite wasteful. A more efficient inference step would be to only estimate the task-relevant dimensions of the environment. I'll write a follow up soon-ish expanding this on this point.

The lightcones in the diagram from Levin 2019 look wrong - shouldn't the agent be a point with a lightcone extending backwords for the past states that can affect it and a lightcone extending forwards for the future states it can affect, as in the later diagram you show? Is Levin just wrong or was it intentional?

In this context, the lightcones represent what the agent remembers and can predict/intentionally affect, instead of the traditional space-time causal influence lightcones. His largest circle is the planet, so he's probably just focusing on de facto cognitive bounds instead of in the limit relativistic ones.

The caption from Levin 2019:

Arbitrary cognitive “Individuals” can be classified according to their computational boundary. (A) Each living system has a delimited “area of concern” – a region of space-time, with the organism at its center, within which its cognitive apparatus functions to take measurements and act. The borders of its cognition are schematized on a semi-quantitative state space defined as follows. The vertical axis is time. Values below the individual’s Now are past events, of which it may have a memory extending some duration in the past; values above the Now are future events, which it may be able to predict or anticipate to some distance in the future. The horizontal axis represents three dimensions of space. Each individual, based on its sensory and effector apparatus, and the complexity and organization of information-processing unit layers between them, can measure and attempt to modify conditions within some distance of itself. (B) The size and shape of this cognitive boundary defines the sophistication of the agent and determines the scale of its goal directedness. This scheme enables multiple agents, regardless of their composition/structure or origin (evolved, engineered) to be directly plotted on the same space. The shape of boundary defines each agent’s “cognitive light cone” – anything outside this region is mentally inaccessible to that system. Here are illustrated a few representative life forms. Primitive agents such as ticks may only have a very small area within which they can sense signals and operate – immediately next to them, and without much memory or ability to anticipate future events. Dogs have significant memory, but very limited ability to plan for the future and can only really care about events in their local vicinity (it is not possible to get a dog to care about what will happen several miles away, or in 2 weeks). Humans exhibit a great diversity of cognitive boundary shapes but on average have a memory that lasts ~10² years, can anticipate decades into the future, and often plan and act to attempt to modify events on quite distant spatial scales (sometimes planetary or even beyond). A variety of as-yet unknown alien, engineered, and bio-synthetic life forms could occupy every conceivable corner of this option space. (C) In this scheme, Individuals can overlap – the same biophysical system can support a number of coexisting, coupled Selves with different cognitive borders. A coordinated swarm of animals, the individual animals themselves, their organs, their cells, and even the metabolic and transcriptional networks inside the cells each have their own cognitive horizon. They cooperate or compete based on specific circumstances and each can be addressed semi-independently because of the differential goals they pursue (and thus, the different positive and negative reinforcements that can be brought to bear to modify events at a given level). All panels courtesy of Jeremy Guay of Peregrine Creative.

I'm still not getting it? Something with a larger life area should have a light-frustrum (that is, a cone with a cut so instead of a point it's a circle). What we remember (functionally: the variables we can vary our actions based on, a la cartesian framea) and what we intentionally can affect are causal influence bounds, even if they don't travel at the speed of light or have chunks taken from them due to our forgetfulness/inability to act.

By the diagram, it's as if as we look into the future, we can only affect a smaller region, which is very false! Time lets my causes propagate outward further. It also is as if the area that we have memories of diminishes as we go back through time - which is kinda true (we don't have archaeology from places where humans weren't) but mostly false, as we learned things about what stuff away from us was like (like if you tell me you ate pasta yesterday, then your memory of pasta-eating can influence both of us even if you did it further away).

Cognitive and control bounds are different from causal influence bounds. I know a lot more about what happened yesterday than 10 years ago. Similarly, I'm capable of planning what I'm going to do tomorrow, while my control over what happens 100 years from now is much more limited, despite my actions having much bigger ripple effects 100 years down the line (see the #1 objection to longtermism). Maybe think of it as a simplified descriptive model of typical bounds of cognitive concern?

The timescale isn't the point here, it's the spread through space over time. Why would my memories/other knowledge of the past lead to me having less spatial breadth the further back I consider? As an example, I don't know what my friend's apartment looked like yesterday, but I do know what it looked like about a few days ago when I last saw it - and this is a direct consequence of more things having influenced me.

For the future: When it comes to things I can control in the next five seconds, they boil down to stuff I can immediately reach (like my wallet). In a few hours of driving however, I could control the stuff in my house. And so the stuff I can control has spread out in space as time has passed, because I could use slower opportunities.

These are closely related to causality, it's just that the relevant nodes in my mind need to be correspondable by me to past-states (and so be distinguishable to me and have a correspondence I can compute) and the nodes in the future need to be alterable in structured ways (which probably means caring about higher level latents as nodes) via plans I can come up with and effects I understand.

Oh, I see. Yes, perhaps something like a cylinder with a fading gradient on both ends might have been a better visual representation of my understanding of what Levin was trying to illustrate.

I really like this direction of thinking. And I would love to see more of it. I have gone I to depth on control theory myself too and I would love to see something like this in current Models trained for Agency

Thank you for this insightful post. I am trying to understand your ideas in full detail: please can you tell me if the following description is 100% aligned? I believe the formal signatures matter!

DOMAIN:

environmental states, as in: the states of the environment
actions, as in: the interventions that the observer can apply to the environment
observations, as in: the measurements that the observer can take from the environment.

FUNCTIONS:

reward function: observation history + applied action + following observation --> real number (the reward)
inference function: new observation + observation history --> expected new environmental state
- optimal inference function: the inference function that optimizes some latent compression of the environmental states (mostly via unsupervised learning; also see information bottleneck). It does not depend on rewards
policy function: current environmental state + environmental state history --> probability distribution of choosable actions
- optimal policy function: the policy function that optimizes the reward's expectation over time, under given constraints. It does not depend on observations.

THE SEPARATION PRINCIPLE: given a reward function and given an optimization problem of the type: current observation + observation history --> probability distribution of choosable actions, you can provably split it into two separated and independent optimization problems: one for the inference function, and the other for the policy function. Even the domain splits: the inference function compresses all observations into some latent environmental state, while the policy function controls such latent environmental states (in the original problem, you don't have environmental states: you only have direct observations).

What I wrote above is wrong, as it doesn't comply with the POMDP framework. What's missing is the following additional domain:

personal belief, as in: the probability distribution over environmental states, as currently believed by the observer.

Then the functions become like this:

reward function: current environmental state + applied action + new environmental state --> real number, i.e. the reward assigned to the intervention (the reward function is hidden: you only get access to the reward scalar, and only after you apply some action)
inference function: previous beliefs + applied action + new observation --> new beliefs
- optimal inference function: the inference function that optimizes some latent compression of the environmental states (via unsupervised learning; also see information bottlenecks). Importantly: it does not depend on rewards
policy function: current belief state --> probability distribution of choosable interventions
- optimal policy function: the policy function that optimizes the reward's expectation over time, under given constraints. Importantly: it does not depend on observations.

It seems that the POMDP framework requires the concept of (bayesian) beliefs; I am not sure that is a requirement for your separation principle though - I may be wrong, but beliefs could be skipped entirely, as long as you run inferences over the observation history directly.

Indeed, the optimal policy is a mapping from histories (sequences of previous observations and actions) to the next action. This mapping can be implemented in multiple ways, including ways that do not compute beliefs. However, the separation principle says that under general conditions there is one way to implement this mapping that involves a factorisation into two modules, and the interface between the two modules are Bayesian beliefs. So Bayesian beliefs are not presupposed, but result as a natural piece of the factorisation. I hope this helps!

Thank you for confirming my second interpretation: Bayesian beliefs are at the core of the separation principle, according to the framework. That generates further questions though :)

You are using the separation principle to naturally justify the introduction of "beliefs", and to show that such beliefs can be optimized independently - e.g. via Bayes.
You convinced me so far, the introduction seems indeed natural and it feels aligned with common sense: if you want to achieve a goal in some environment, a valid approach is to (1) understand the environment's behaviour in general, and (2) leverage such understanding for specific interventions.

I can tell you more: this paper https://yuxinchen2020.github.io/publications/Reward-free-exploration.pdf proves, mathematically, that reward-based optimization can also be achieved by reward-free exploration first, and deterministic goal-planning later. That seems to confirm a stronger version of your thesis! It means that the separation principle provides an additional benefit: theoretically, your policy function does not need any training at all, it only needs deterministic planning!

Let's look at the differences as well: exploration can form beliefs, but they are not necessarily Bayesian beliefs. It looks closer to world-modelling, and you can do that with many other unsupervised techniques.

Moreover, to be compatible with the "folk" distinction beliefs/desires (or true/good, epistemic/telic, cognitive/conative...) you implicitly framed the reward as some proxy of your "desires". That doesn't quite capture my intuition behind the pair beliefs/desires! Only the "beliefs" part looks correctly justified in this analogy. And people, in many cases, can change their reward functions, depending on their taste. I think half of the equation is still missing here.

Just food for thoughts, the post is very interesting the way it is.