1060

LESSWRONG
LW

1059
Free Energy PrinciplePredictive ProcessingEmbedded AgencyHyperstitionsNeurosciencePlanning & Decision-MakingPsychologySelf Fulfilling/Refuting PropheciesWorld Modeling
Frontpage

10

The Way You Go Depends A Good Deal On Where You Want To Get: FEP minimizes surprise about actions using preferences about the future as *evidence*

by Christopher King
27th Apr 2025
7 min read
5

10

10

The Way You Go Depends A Good Deal On Where You Want To Get: FEP minimizes surprise about actions using preferences about the future as *evidence*
2J Bostock
8Yldedly
8Ariel Cheng
1Yldedly
1Christopher King
New Comment
5 comments, sorted by
top scoring
Click to highlight new comments since: Today at 5:35 PM
[-]J Bostock5mo20

Under this formulation, FEP is very similar to RL-as-inference. But RL-as-inference is a generalization of a huge number of RL algorithms from Q-learning to LLM fine-tuning. This does kind of make sense if we think of FEP as a just a different way of looking at things, but it doesn't really help us narrow down the algorithms that the brain is actually using. Perhaps that's actually all FEP is trying to do though, and Friston has IIRC said things to that effect---that FEP is just a reframing/generalization and not an actual model of the underlying algorithms being employed.

Reply
[-]Yldedly5mo80

There are some conceptual differences. In RL, you define a value function for all possible states. In active inference, you make desirable sense data a priori likely. Sensory space is not only lower-dimensional than (unobserved) state space, but you only need to define a single point in it, rather than a function on the whole space. It's often a much more natural way of defining goals and is more similar to control theory than RL. You're directly optimizing for a desired (and known) outcome rather than having to figure out what to optimize for by reinforcement. For example, if you want a robot to walk to some goal point, RL would have to make the robot walk around a bit, figure out that the goal point gives high reward, and then do it (in another rollout). In active inference (and control theory), the robot already knows where the goal point is (or rather, what the world looks like when standing at that point), and merely figures out a sequence of actions that get it there. 
Another difference is that active inference automatically balances exploration and exploitation, while in RL it's usually a hyperparameter. In RL, it tends to look like doing many random actions early on, to figure out what gives reward, and later on do actions that keep the agent in high-reward states. In control theory, exploration is more bespoke, and built specifically for system identification (learning a model) or adaptive control (adjusting known parameters based on observations). In active inference, there's no aimless flailing about, but the agent can do any kind of experiment that minimizes future uncertainty - testing what beliefs and actions are likely to achieve the desired sense data. Here's a nice demo of that: 

Reply
[-]Ariel Cheng5mo80

See https://arxiv.org/abs/2006.12964:

CAI augments the ‘natural’ probabilistic graphical model with exogenous optimality variables. 4 . In contrast, AIF leaves the structure of the graphical model unaltered and instead encodes value into the generative model directly. These two approaches lead to significant differences between their respective functionals. AIF, by contaminating the veridical generative model with value-imbuing biases, loses a degree of freedom compared to CAI which maintains a strict separation between the veridical generative model of the environment and its goals. In POMDPs, this approach results in CAI being sensitive to an ‘observation-ambiguity’ term which is absent in the AIF formulation. Secondly, the different methods for encoding the probability of goals – likelihoods in CAI and priors in AIF – lead to different exploratory terms in the objective functionals. Specifically, AIF is endowed with an expected information gain that CAI lacks. AIF approaches thus lend themselves naturally to goal-directed exploration whereas CAI mandates only random, entropy-maximizing exploration.

These different ways of encoding goals into probabilistic models also lend themselves to more philosophical interpretations. CAI, by viewing goals as an additional exogenous factor in an otherwise unbiased inference process, maintains a clean separation between veridical perception and control, thus maintaining the modularity thesis of separate perception and action modules (Baltieri & Buckley, 2018). This makes CAI approaches consonant with mainstream views in machine learning that see the goal of perception as recovering veridical representations of the world, and control as using this world-model to plan actions. In contrast, AIF elides these clean boundaries between unbiased perception and action by instead positing that biased perception (Tschantz, Seth, & Buckley, 2020) is crucial to adaptive action. Rather than maintaining an unbiased world model that predicts likely consequences, AIF instead maintains a biased generative model which preferentially predicts our preferences being fulfilled. Active-inference thus aligns closely with enactive and embodied approaches (Baltieri & Buckley, 2019; Clark, 2015) to cognition, which view the action-perception loop as a continual flow rather than a sequence of distinct stages.

Reply
[-]Yldedly5mo10

Nice, CAI is another similar approach, kind of in between the three already mentioned. I think "losing a degree of freedom" is very much a good thing, both computationally and functionally. 

Reply
[-]Christopher King5mo10

Yeah my understanding is that FEP is meant to be quite general, the P and Q are doing a lot of the theory's work for it.

Chapter 5 describes how you might apply it to the human brain in particular.

Reply
Moderation Log
More from Christopher King
View more
Curated and popular this week
5Comments
Free Energy PrinciplePredictive ProcessingEmbedded AgencyHyperstitionsNeurosciencePlanning & Decision-MakingPsychologySelf Fulfilling/Refuting PropheciesWorld Modeling
Frontpage

The motivation for this post came when I was reading various Scott Alexander posts on the Free Energy Principle, in which he seemed very confused about it (the title God Help Us, Let’s Try To Understand Friston On Free Energy might have hinted at that). I was intrigued but also very confused, so I fell down a rabbit hole, so to speak, trying to figure it out. I am very grateful to Active Inference: The Free Energy Principle in Mind, Brain, and Behavior, which was published after Scott's post. This open access book has this amusing section in the preface:

A Note from Karl Friston
I have a confession to make. I did not write much of this book. Or, more precisely, I was not allowed to. This book’s agenda calls for a crisp and clear writing style that is beyond me. Although I was allowed to slip in a few of my favorite words, what follows is a testament to Thomas and Giovanni, their deep understanding of the issues at hand, and, importantly, their theory of mind—in all senses.

Free Energy and the Dark Room problem

The Free Energy principle asserts that biological systems (such as trees and the human brain) have beliefs and take actions that minimize a quantity known as Variational Free Energy. Variational Free Energy is a function of beliefs (Q) and sense data (y), where sense data comes from the external world (x) and from actions.

F(Q,y)=DKL(Q(x)∥P(x|y))−lnP(y)

Q(x) is meant to be the system's attempt at approximating P(x|y). For simplicity, for most of this post I'll assume that the system can do perfect Bayesian inference, so that Q(x)=P(x|y), implying that F(Q,y)=−lnP(y). In other words, the system is trying to minimize how surprising the sense data is.

An apparent paradox is called the Dark Room problem. A dark room has very little suprise by virtue of having very little information. So why don't all organisms hang out in them? According to Translating Predictive Coding Into Perceptual Control

The main proposed solution is to claim you have some built-in predictions (of eg light, social interaction, activity levels), and the dark room will violate those.

Friston's explanation is similar, if a bit wordy.

The free-energy principle says that we harvest sensory signals that we can predict (cf., emulation theory; Grush, 2004); ensuring we keep to well-trodden paths in the space of all the physical and physiological variables that underwrite our existence.

I am pretty sure these resolutions are don't actually resolve the problem. While technically true, they miss the core problem.

The actual resolution is this: deciding to be in a dark room minimizes future free energy, but FEP actually says that actions are minimizing the present free energy.

In other words, there has been substantial confusion about what exactly is being optimized with respect to what variables.

You can sense your actions now, and that is the only sense you can change

Usually when we think of senses, we mean sensing the external world, such as by using sight and sound and what not. However, in the Free Energy literature, this definition is typically somewhat expanded: actions also count as sense data. In this way, we can factor P(y) as P(actions|external senses)P(external senses).

This is why minimizing the present free energy is even possible. No action you can take (such as entering a dark room) will immediately register in your vision or other external senses. Only your actions can immediately enter the sense data while solving this optimization problem.

You choose the least surprising action

‘The rule is, jam to-morrow and jam yesterday—but never jam to-day.’

— Lewis Carroll, "Through the Looking-Glass"

Consider a choice between the dark room on the left and a room with delicious food on the right. The problem is, you don't know what food! The door is currently closed. But you do know the food is good. In particular, you know it is more likely that you will go into the food room than the dark room (because obviously you like food better than darkness).

Consider these priors:

P(Go left into the dark room)=13P(Go right into the food room and it has steak)=16P(Go right into the food room and it has cotton candy)=16P(Go right into the food room and it has pizza)=16P(Go right into the food room and it has jam)=16

Steak, Cotton Candy, Pizza, and Jam
Four equally delicious possibilities, but we don't know which one we'll get.

So, sadly, if you go into the food room, you will have more free energy than if you went to into the dark room. Do we need a fudge factor? (Would adding fudge into the mix make things worse?)

Nope. You choose the action that is currently least surprising according to your own model and preferences. Minimizing surprise now means choosing the action a that maximizes P(a|external senses), or equivalently, minimizes −lnP(a|external senses). Since P(Go right)=2/3>P(Go left)=1/3, "Go right" is the less surprising action choice right now. The potential future sensory surprise inside the food room is not the deciding factor for the current action.

We can even add random sensory information (like random numbers) to the right room. This will increase its future free energy, but not the present free energy of choosing right.

Likewise, if you end up in the dark room, the least surprising action is to leave immediately. The fact that you will be surprised once you leave is irrelevant.

This is also why the "humans don't like the dark" explanation is technically correct. If humans did really like darkness, P(Go left) might be higher than P(Go right). The reason it is a bad explanation is that is missed the point about the predictability of the dark room being irrelevant for the current action choice. Random sensory info might contribute billions of bits to the free energy! I am sure that Friston understands this, but I think he didn't realize that others missed that point!

P is well-calibrated

Here is another apparent problem:

So for example, suppose it’s freezing cold out, and this makes you unhappy, and so you try to go inside to get warm. FE/PC would describe this as “You naturally predict that you will be a comfortable temperature, so the cold registers as strong prediction error, so in order to minimize prediction error you go inside and get warm.” PCT would say “Your temperature set point is fixed at ‘comfortable’, the cold marks a wide deviation from your temperature set point, so in order to get closer to your set point, you go inside”.

The PCT version makes more sense to me here because the phrase “you naturally predict that you will be a comfortable temperature” doesn’t match any reasonable meaning of “predict”.

This again mixes up the present and the future, but in the other direction. When you predict you will be a comfortable temperature, that is a belief about the future. All predictions are beliefs about the future.

But didn't I just say that the Free Energy Principle is about the present? The trick is that learning about the future requires you to do a Bayesian update on your predicted actions in the present. This is known as planning-as-inference. In fact, by dropping the first term in free energy, this entire post has secretly been about planning-as-inference.

So why do people say that free energy is minimizing prediction errors? Because your actions now affect your predictions about the future.

P(comfy later|cold now,start moving inside)>P(comfy later|cold now,do nothing)

Thus, our prior preference/prediction to be comfortable soon makes the action "start moving inside now" the most probable (least surprising) action to infer in the present. Starting to move inside now reduces our prediction error about the future, and Bayes favors hypotheses that reduce prediction errors.

And in fact, if we decide to go inside, all our beliefs turn out correct! We are cold now, we decided to go inside, and we will be comfy later. So P is a well-calibrated self-fulfilling prophecy.

Same thing for the motor cortex. How ought your muscles to move? To quote the cheshire cat, "that depends a good deal on where you want to get to." It is not telling you where your muscles are now, it is fulfilling predictions about where your muscles will be a couple fractions of a second from now (which just so happens to usually be close to where they are now).

Since P is making sensible predictions, I think “you naturally predict that you will be a comfortable temperature” is completely reasonable!

Note that P is occasionally wrong tho. For example, yesterday (true story) when I was chewing food, I thought I had partially swallowed it, when in fact I had swallowed all of it. In other words, I had a false belief that I had food in my mouth! Did that cause me to resolve the error by putting food in my mouth? No, the most likely action was to try chewing the food, which I did, causing my teeth to clack when no food was there (and presumably causing many prediction errors in my motor cortex). When I discovered the error, was my response to put food in my mouth? No, because that prediction error was now a few moments in the past. My beliefs about the past updated, but my decisions in the present would only influence the future probability, not the past probability, of having food in my mouth.