TLDR: Psychology, economics, and other disciplines describe agents as systems driven by beliefs and desires. This post argues that the belief-desire view can be derived from classic theorems from optimal control and reinforcement learning. This suggests seeing beliefs and desires as properties of optimal policies rather than as assumptions from folk psychology.

Introduction

One way to think about agents is as "systems that act for reasons".^[1] This compact statement can be interpreted as encapsulating two key implications:

• The notion of action assumes a boundary between agent and environment, so that the former can act on the latter.
• The term reason captures two kinds of internal activity: motivations associated with how to achieve specific goals or outcomes, and beliefs regarding what the agent infers to be the current state of affairs.

In other words, an agent is a well-differentiated system that acts based on beliefs and desires. This view is compatible with perspectives that have been developed by various disciplines:

1. Behavioural science, which sees agency as goal-directed behaviour.
2. Economics, which treats agency as the ability to select policies to achieve an objective.
3. Cybernetics, which conceptualises agency as the ability to regulate the environment and keep it within a desired subset of states.

Instead of taking the working definition^[2]

agency = beliefs + desires

at face value, in this post I will discuss conceptual tools that can allow us to derive this construction from formal considerations. In particular, I will discuss the separation principle, which I believe has potential to provide a rigorous foundation for this conceptualisation. These ideas take inspiration from cybernetics and computational mechanics, but distinctly leverage classic results in optimal control theory and reinforcement learning.

Motivation — cybernetics.

A related line of thinking pertains to the (in)famous Good Regulator Theorem. This result, put forward by cyberneticists Conant and Ashby in 1970, suggests that an effective controller requires an internal model of what is being controlled. This makes intuitive sense: it is hard to predict a system that follows an intricate internal mechanism, and perhaps the only way to effectively control it is by understanding it.

Unfortunately, the original paper has been both a source of inspiration and confusion. Lots have been written trying to clarify it; to read more, see this post and this post, and also this manuscript. This post will follow these ideas in spirit, but not the formalisation as proposed there.

Motivation — computational mechanics.

Another related line of work is computational mechanics, which combines principles of information theory, theoretical computer science, and some sprinkles of statistical mechanics to describe the dynamics of prediction. Computational mechanics derives minimal structures required for optimal prediction, which take form in the -machine and -transducer. These ideas offer a rigorous explanation to (i) why optimal prediction requires specific computations, and (ii) what properties those computations need to satisfy. To read more, see this thesis or this paper.

More recently, computational mechanics is providing effective tools to study the internal representations in deep learning. Moreover, while originally formulated to study prediction, computational mechanics also provides a promising foundation for understanding perception-action loops. The ideas developed here are heavily influenced by both the concepts and formalism of computational mechanics.

What is a separation principle?

Let's start by discussing what separation principles are, in general.

As a first approximation, consider a situation where we want to minimise the function . A direct calculation shows that

.

Hence, solving the problem for and separately does not yield the solution for . Put simply, one does not solve the problem by solving its parts separately.

A separation principle takes place when the divide-and-conquer strategy actually works, so one can solve a complicated optimisation problem by optimising various subcomponents separately. Concretely, a separation theorem guarantees that optimal performance can be achieved by solving sub-tasks in isolation.

Let me further illustrate this idea by presenting an important separation theorem at the core of information theory.

The communication problem studied by information theory is about how to send bits from source to destination without error. The tools that a communication engineer employs to face this problem are two types of coding techniques:

• Source coding, which compresses the original signal by removing redundancy in order to transmit as few bits as necessary.^[3]
• Channel coding corresponds to error-correcting codes, which adds redundancy to protect the information bits from noise.^[4]

Most people have heard of the following results by Claude Shannon:

1. The source coding theorem, which states that the shortest achievable average code length is given by the entropy of the source.
2. The noisy-channel coding theorem, which states that the maximal transmission rate that allows arbitrarily low decoding error is given by the mutual information between input and output symbols (maximised over input distributions).

However, these results would not be as useful in the absence of the third, less known result known as the source-channel separation theorem.^[5] This result states that, under relatively mild conditions,^[6] an optimal solution for the overall communication problem can be achieved by breaking the problem into two separate parts: compression and error correction. This means that an optimal communication strategy can be designed by a team separately working on compression and another working on error-correction, without requiring coordination between them. Needless to say, this makes the problem much more tractable!

The inference-control separation principle

Let us now review results from optimal control theory and reinforcement learning that provide conditions for separating inference and control. In doing this, I will use the non-standard term inference-control separation principle to group together results that are often presented independently of each other, despite having a lot in common.

Separation principle in optimal control theory

Let us consider the classic instantiation of separation in the context of optimal control theory. For this, consider a control setting made by three components:

• a system to be controlled, whose state is described by ,
• the measurements an agent has access to, described by , and
• the actions .

For simplicity, we assume that the system evolves dynamically over discrete time via linear dynamics and noisy observations that can be described as

,

,

where and are matrices, is the system's initial condition, and and are independent white Gaussian noise terms.

The action is determined according to a policy that has no direct access to the state , but only to the sequence of measurements . Thus, we take to be a deterministic function of the sequence of past measurements and actions . The controller aims to minimise the quadratic cost

,

where , , and are matrices and is the length of the time window over which the optimisation takes place. This is known as the discrete-time LQG problem, as it involves Linear dynamics, a Quadratic loss function, and Gaussian variables — being the one setting where everything is exactly solvable.

Separation principle — To understand the solution of this problem, let us first consider the case of full observability, where . In this setting, the optimal controller can be shown to only depend on the current state of the system, and the optimal solution can be expressed as

,

where is the "optimal gain" matrix (whose formula can be found in standard textbooks). So, if the controller can directly see the state of the system, this directly tells what the optimal action should be.

What happens with partial observability? In this setting, the optimal controller can be written as

,

where is the same as above and is an estimation of the state of the system given by

.

In fact, is the result of Kalman filtering, which is how recursive Bayesian filtering looks like under Gaussian dynamics.^[7]

In summary, the optimal policy can be described as "estimate first, and then act as if your estimate were the actual state of the system". Thus, a separation theorem is taking place separating inference and control.

Separation principle in reinforcement learning

To understand what the separation principle looks like in reinforcement learning, let us slightly modify the nomenclature and notation:

• the state of the system to be controlled is described by a discrete variable ,
• the measurements that the agent has access to are described by , and
• the actions are denoted by .

Let's assume that the variables follow a partially observed Markov decision process (POMDP), so that the environment dynamics are Markovian conditioned on the agent's action via a Markov kernel ,^[8] and the current observation depends on the current state of environment via a Markov kernel . The agent takes actions following a policy that maps sequences of previous actions and observations into distribution of next actions.

In this setting, the probability of observing a sequence of observations and actions is

,

where is a prior distribution over the environment's initial state. This is roughly the same as the control scenario studied above, but considering general discrete variables instead of Gaussian ones.

Instead of a loss function, consider a reward that the agent tries to maximise. More specifically, let us define the discounted future reward

,

where is the so-called discount factor. Then, the goal of the agent is to find a policy that maximises .

Separation principle — To solve this POMDP, let us first consider the Bayesian beliefs about the latent state . There are two ways to understand Bayesian beliefs:

• As random variables given by the conditional expectation , where the trajectory is regarded as a random variable.
• As probability distributions given by , which correspond to the above expectation when conditioned on a specific realisation of .

Bayesian beliefs have two key properties. First, as random variables, they contain all the information in that is useful for predicting (i.e., they are sufficient statistics). Second, as distributions, they can be efficiently updated via

,

where is a deterministic function, and hence the dynamics of are Markovian conditioned on (as they only depend on ). These facts can be used to build a new Markov decision process (MDP), known as belief MDP, defined by the following variables:

• the states are the beliefs of the latent state of the POMDP,^[9]
• actions are (same as in the POMDP),
• the reward function is .

In contrast with the original problem, this MDP is fully observable (as the agent knows its own beliefs), and therefore can be solved using standard RL techniques. The crucial point is that combining the estimation of Bayesian beliefs with the optimal solution for the belief MDP gives rise to a solution of the full POMDP.^[10] Thus, a separation principle is at play separating inference (calculating beliefs) and control (solving the belief MDP).

Interim summary

Both results in LQG control and POMDPs give rise to the same solution: they show that the optimal controller for a partially observed system can be built by

1. first performing optimal prediction over the latent state of the system, and then
2. solving the control problem using that prediction as a state.

To further understand how the LQG and POMDP settings relate, let's discuss the shape of beliefs and the effect of actions (not essential for the rest of the post).

Beliefs are distributions, not point estimates.

In the LQG setting, the optimal controller does not need the full distribution of the prediction — the prediction is a Gaussian posterior, but the controller only uses the mean and not the variance. This property is known as certainty equivalence, which suggests to act as if a point estimate were the true state.

It would be tempting to describe beliefs as "best guesses" (e.g. maximum a posteriori), but certainty equivalence only holds because of the friendly properties of the LQG setup — a more general setting would not allow for this simplification.

In the more general POMDP case, the agent must carry the full belief distribution, not a point estimate. Nonetheless, separation survives in the Bayesian filter + belief-MDP form. Thus, "beliefs" in the agency sense are not mere best guesses but the full posterior, and only under special conditions do they collapse to a point estimate. More generally, the separation principle suggests to think of an agent's epistemic state not as what it thinks is true, but how its uncertainty is shaped.

Implications

Let us now return to the question we started with: can the working definition

agency = beliefs + desires

be derived, rather than assumed? The separation theorems reviewed above suggest that, at least in a specific sense, the answer is yes.

Beliefs and desires as properties of solutions

Let's recapitulate what the separation principle delivers. We posed a single, monolithic optimisation problem: find a policy mapping histories into actions that minimise/maximise loss/reward. Nothing in this problem statement mentions beliefs, inference, or internal representations — the problem is stated in terms of pure behaviour. Yet, the solution naturally factorises into sub-components. Indeed, the above results state that the optimal policy can always be realised as the composition of two blocks:

• An inference module, which compresses the history into a Bayesian belief about the latent state that can be recursively updated. This module knows about dynamics but is oblivious about loss/rewards.
• A control module, which selects actions as a function of that belief alone. This module knows about loss/rewards but not about raw observations.

The interface between them is exactly the belief state, which is the output of the inference module and the input of the control module. Thus, the inference module can be interpreted as "where beliefs are made"; similarly, the control module can be interpreted as "where desires take place", as it is the only part of the policy that is directed by the reward/loss function. In this way, beliefs and desires are recovered not as assumptions about agents but as properties of solutions.

This is worth pausing on. The folk-psychological notions of belief and desire have been with us at least since Aristotle, and they are usually treated as primitives — as the ingredients one starts with when theorising about minds. What the separation principle offers is something different: not a definition of these terms but a derivation of them. Beliefs and desires appear here not because we built them in, but because the optimal solution to a problem stated in purely behavioural terms turns out to have that shape. Whether or not one takes folk psychology seriously as a theory of mind, this is at least a reason to think its central distinction is not arbitrary.

It is remarkable that this modularity arises from solving a single, monolithic optimisation problem. We have seen this shape before: it is precisely the payoff that made the source-channel separation theorem so valuable! There, an optimal communication system could be designed by a compression team and an error-correction team working independently, without coordination — the source coder needed to know nothing about the channel, and the channel coder needed to know nothing about the source.

The inference-control separation delivers the exact analogue for agency, conferring a form of compositional generalisation. Indeed, if the reward changes but the environment does not, only the control module needs to be updated; if the environment shifts but the goals remain, only the inference module needs revising. Thus, the two components can be developed, improved, and swapped independently.

Agents as cognitive light-cones

These results may also provide some formal ground to conceiving of agents as cognitive light-cones.^[11] The idea goes as follows: an agent is a system whose behaviour is both sensitive to what it can remember about the past (the past component of the cone) and what it can foresee (the future component of the cone).

This idea suggests an operational way to assess the capabilities of an agent: measure the size of its light-cone. This could be done using tools from computational mechanics — for example, studying how deep into the past causal states go, and doing the same for retrodictive causal states (which build sufficient statistics from future to past).^[12]

The separation principle is normative, not descriptive

Please note that the separation principle says that an optimal agent can be built as inference + control, but it does not say that any given agent is built this way. In particular, a policy trained end-to-end by gradient descent is under no obligation to organise itself into a filter feeding a planner.

But the separation principle gives us something subtler: it tells us that the belief state is a sufficient statistic for optimal behaviour. This suggests that any agent approaching optimality must be computing something as informative as the Bayesian posterior (whether or not it is legible as such). In this way, the separation principle turns into a lens for interpretability: we should expect belief-like structure inside competent agents, and we can go looking for it. Recent work from colleagues at Simplex in finding belief-state geometry in the residual streams of transformers is, I think, an early vindication of this expectation. At XOR Labs we are currently investigating to what degree these results transfer to deep RL agents trained to do control tasks.

At this point, it is useful to distinguish two separate concerns:

• Control problems / POMDPs can have multiple optimal policies. In these settings, the separation principle states that there exists one optimal policy that can be factored, but doesn't guarantee all do.^[13]
• When only one optimal solution exists, the separation principle says this unique behaviour admits a realisation as filter + belief-controller — but it does not say that's the only realisation. Indeed, the same mapping from histories to next action can be implemented in different ways; for instance, in a finite setting the mapping can always be implemented as a large look-up table.

Regarding the second concern, there are various pieces of evidence suggesting that deep learning architectures have an implicit bias for simplicity, which makes them find simpler implementations of algorithms whenever they exist.^[14] Therefore, while it is often possible to implement a policy via a gigantic look-up table, there are reasons to believe that policies obtained via deep learning techniques will tend towards factorised ones.^[15]

Where does separation fail? As Shannon's source-channel separation theorem, the inference-control separation theorems hold under fairly general conditions — not relying on finite alphabets or stationarity. That said, for it to be useful one needs to assume that the agents under consideration have enough compute to build (quasi-)optimal policies. Under bounded rationality, model misspecification, or multi-agent interaction, the clean factorisation can break: what to represent starts depending on what you want, and inference becomes goal-driven.^[16] Many of the most interesting aspects of agency may come from the ways in which real agents deviate from the separation principle — but that is a topic for a future post.

Coda

Before concluding, let me ask: does the separation principle derive beliefs and desires from first principles?

Concretely, the separation principle offers the following:

1. It reveals that optimal policies can be factored into two separate modules.
2. The signal shared between the modules can be interpreted as beliefs, so that the first (inference) module can be interpreted as "where beliefs are made".
3. The second (control) module can be interpreted as "where desires take place", as it is the only part of the policy that is directed by the reward/loss function.

Thus, the separation principle provides formal ground for articulating the view of agents as "systems that act for reasons", displaying behaviour that is sensitive to both the past (what it remembers as part of its beliefs) and the future (due to foreseeing and planning to achieve goals).

However, one has to acknowledge that the separation principle assumes an optimisation problem which already contains a reward or cost function. Following this reasoning, one could argue that the desire or preference has not been really derived, as it has been baked in externally. More precisely, one can interpret the separation principle not as deriving why desires exist in the first place, but as explaining why they can be treated as separate entities from beliefs.^[17]

1. ^{^}

   There is a huge literature discussing various views on what agency is; if you want to read more, perhaps take a look into this and this paper and references therein.

2. ^{^}

   This post will take the boundary between agent and environment as given to focus on their internal activity. To read about boundaries, see these posts.

3. ^{^}

   Examples include Huffman and arithmetic coding.

4. ^{^}

   Examples include Hamming, Reed-Solomon, or convolutional codes.

5. ^{^}

   In contrast with the other two, this result doesn't have a wikipedia page...

6. ^{^}

   For technical details, see Cover & Thomas Chapter 7.13.

7. ^{^}

   For more details about the separation principle in LQR, see this paper. For a good introduction to Kalman and Bayesian filtering, see this book.

8. ^{^}

   denotes the space of probability distributions over the set .

9. ^{^}

   The transition probabilities can be computed directly; for formulas see this paper or wikipedia.

10. ^{^}

    For proofs, see this and this paper.

11. ^{^}

    I first saw this idea in this inspiring paper of Michael Levin.

12. ^{^}

    I'll expand on this in a future post.

13. ^{^}

    This is a real issue, but can be disregarded in some settings. For instance, "almost every" finite POMDP has a single optimal solution. Multiple optimal solutions in POMDPs are caused, for example, by ties in the optimal Q-function, which are broken by arbitrarily small perturbations on the reward.

14. ^{^}

    There have been numerous discussions about the simplicity biases of deep learning, see for example this post.

15. ^{^}

    For example, we have evidence that transformers find factored representations whenever they are available — see this paper.

16. ^{^}

    This is arguably where phenomena like motivated reasoning and wishful thinking become rational responses to bounded resources rather than mere biases. To read more, see this and this.

17. ^{^}

    For answering the why question, one may need to consider arguments based on natural selection, or perhaps approaches to agency based on the enactive tradition — as developed, for example, in this paper. See related discussions questioning utility functions in this post.