(The original title for this was "Markets are equivalent to stuff", because I also demonstrate market-MDP/Bellman and market-backprop analogies. While those scratched an itch I've long had, the more important result is the headline one. Part of my work at MATS in Richard Ngo's stream.)
A bunch of people have vaguely noticed intuitive analogies between economies/markets and machine learning algorithms: John Wentworth, Aanjaneya Kumar and Anders Sandberg at the AI Objectives institute, the ancient Neural Bucket Brigade (Davis, Schmidhuber). Several papers discuss RL algorithms based on sequentially auctioning a unit "right to act" among agents---most recently Caspar Oesterheld's BRIA but also Baum's Hayek Machine, its predecessors the classifier systems (Holland 1985, Schmidhuber 1987, Schmidhuber 1989) and its successors Hayek-for-POMDPs (Kwee et al 2001) and Hayek with Vickrey auctions (Chang et al 2020). In "Market-based architectures in RL and beyond" I suggested that a better market model for RL might come out of factoring the "right to act" into "goods" axes, just like in real markets.
The analogy is attractive and suggestive:
Economies are intelligent agents built out of sub-agents much less capable than themselves (see e.g. "I, Pencil") suggesting that markets are an example of intelligence being an "emergent phenomenon".
Bounded rationality is very analogous to the Efficient Market Hypothesis. How can you be rational when computing the rational decision itself has a cost---how do you optimize over that? You are rational/markets are efficient "conditional on the information (including computational information) present in the system", whatever that means.
Markets have analogs of inconsistency (arbitrage) and Godel's incompleteness theorem: a prediction market is limited in its ability to efficiently price the question "This question will resolve YES if this market gives less than 50% probability to it". (More specifically, a market must give zero subsidy to such a question: its subjective "value for information" on that question must be zero.)
Related to 3: "Market dynamics" is famously under-studied because again it is fundamentally computational and boundedly-rational rather than "there exists an equilibrium, DONE".
The dynamical nature of markets feels very similar to backpropagation (as well as e.g. Bellman recursion): prices are gradients (specifically: gradients on activations, not weights---they do not zero out at equilibrium), the gradient
In this article we will formalize this intuition, and explain that in fact, markets are equivalent to a lot of things: backprop, RL, and continuous Bayesian inference (aka "replicators").
In the first section we will describe the general economic model and how "prices" (gradients) arise naturally. In the subsequent sections we will start describing the analogies.
1. Setup
Here is how we will define an "economy".
Definition 1. There is a vector space in which quantity vectors live (so its dimensionalty is the number of unique goods). There is a directed graph, where every node is a "factory" labelled with production function and every edge is labelled by a quantity vector . The graph is subject to physical constraints:
Where and . Furthermore some nodes, called "consumers" have a utility function .
We may imagine a global welfare objective (WLOG all the weights are 1).
U.S.O.W. We will usually think of as a DAG and are continuous and differentiable; utilities have no explicit dependence on flows or shares. Cyclic graphs are certainly interesting, but they are hard in general.
(But what about factories choosing what to produce? Shouldn't be a set and be chosen from this set? For now we will treat that as a set of every possible , and the choice of what to produce amounts to routing to the right . Maybe this is bad from a computational POV or whatever; alternatively we may consider the 's themselves as trainable sub-agents with some internal parameters.)
1.1. General price recursion
Introduce Lagrange multipliers for the production constraints:
Differentiate with respect to quantity vectors , we get the recursion:
i.e. the price of output from equals the direct marginal utility/cost at , plus the direct marginal utility at , plus the downstream production value obtained by sending the good through . The term is is the backpropagation operation.
However, this does not directly let us perform backpropagation on the market graph, since are not free variables: they must satisfy the constraint . So instead parameterize quantity flows using routing shares:
The forward computation is (where indicates the elementwise product, and is an exogenous resource endowment if any):
Defining adjoints and , and differentiating:
Now, these shadow price propagation equations themselves are just true: not dependent on any particular dynamics/optimization algorithm. They are highly suggestive of a backprop-like dynamics, and the fact that these prices actually are so important in real-world economic decision-making suggests that this suggestiveness is right.
You can transform the shares into logit space to do normal backpropagation without simplex constraints, or you can directly do optimization on a simplex, e.g. via entropic mirror ascent which gives the update rule (with geometric learning rate ) as in Section 4. Regardless, the optimization rule is always: "move outputs towards buyers with higher marginal value " At equilibrium, all active destinations of the same good from the same source have equal marginal value, i.e. for active edges (, ):
2. Markets are MDPs; Advantage = Rivalrous goods
Here is a nice interpretation of our economic model. Imagine there is only one unit good, i.e. a particle in the whole market. If the production functions are identity, then its movement through the market is governed by policy , collecting rewards along the way.
Production makes the particle "branch": at node a unit of good becomes a bundle . We may regard the Jacobian as a "matrix-valued discount factor". We also have a Bellman recursion:
RL
market
state
"one unit of good sits at node "
action
"ship it to "
policy
routing share
reward
marginal utility collected along the way
action value
destination price
state value
average received price
advantage
price premium
visitation measure
quantity present
Optimizing this market (computing the s) is then equivalent to solving this MDP. In fact, we have a "Policy Gradient Theorem" (the equation (1.4) from earlier):
Or parameterizing with logits :
Proof. Chain rule through the softmax Jacobian applied to (1.4).
In fact even the interpretation of as a "matrix-valued discount factor" has a natural interpretation. Generalizing , we have in a cyclic economy the Hawkins-Simon condition where is the full matrix of 's (for a particular good) and is the spectral radius (the largest absolute eigenvalue), which tells us that a cyclic economy cannot a perpetual motion machine.
(Well actually, it kinda can be a perpetual motion machine, if the dynamics of the cyclic economy represent actual time rather than just convergence to an equilibrium. But tbh I would rather represent that unrolled, as an infinite graph with nodes stretching out into the future.)
(In the "wide market RL" I introduced in my paper Market-based RL, I don't think of the MDP itself as a market, but the transition probabilities are determined by an additional Garrabrant-like program market of traders and the network is more "fundamentally dynamic" rather than graph-like. Something in-between---dynamic like program markets but efficient like graphs---like a graph that forms new connections or performs search of some sort when shadow prices exceed some friction-induced background transaction cost, seems like a good model for describing real markets, Knightian uncertainty etc.)
3. Chain markets are MLPs
Theorem 1 (chain = MLP). Let the economy be a directed chain with resource injected at node , production functions , and a single consumer with utility at node . Then all shares are trivially equal to 1, and:
The forward pass is the feedforward computation , , with welfare : goods bundles are activation vectors (one neuron per good; a node/factory is a whole layer).
The market's shadow prices are the backprop gradients: .
Internal parameterization. If has parameters , then by Lemma 1, gradient ascent by each firm on its own profit at posted prices is identical to backprop/SGD on the network's weights: .
Conversely, every MLP whose activations are nonnegative (i.e. with ReLU or similar activation functions) arises this way, with bias vectors = exogenous endowments injected at layer and the objective function as the terminal consumer's utility.
Proof. 1–2 are the forward/backward recursions of Section 1 with a single out-edge; 3 is Lemma 4.1; 4 is construction.
Theorem 1 completely formalizes the "economies = backprop/neural networks" intuition that people have been groping at. Note that economies are more general than classical MLPs: they are neural networks with routing (which is kind of "transposed static attention": in markets, the goods are scarce while in transformers, the receiver-side attention budget is scarce).
4. Continuous Bayesian inference as a dynamics for markets
In my post Reward as hyperstitional information, I conveyed that entropic mirror ascent (aka continuous-time Bayesian inference) is the natural dynamics for probabilities. Rewards and such are just an "evidence rate".
It seems very suggestive that our routing shares are literally just probabilities (of "a particle of good being sent to some factory"). Indeed, we can let the dynamics of our market be "entropic mirror ascent on total welfare" i.e. update the shares each time step as (indices are implicit):
From 1.4 we know that the share gradient equals the revenue rate thus the solution to this is (read the "Reward as hyperstitional information post" if this isn't immediately clear):
Differentiate with respect to time to get the usual replicator flow/continuous-time Bayesian inference:
So: the market (under entropic mirror ascent) is a Bayesian learner, with each agent doing "hyperstitious" learning. I.e. each agent has a belief distribution on the question "where will I send an infinitesimal quantity to?" and the infinitesimal evidence it receives for each hypothesis is exponentiated revenue ratio .
Revenues are rewards are information (log-likelihoods).
You can use this equivalence to transfer any result.
domain
mass
fitness / evidence
equilibrium
evolution
population shares
reproductive fitness
fitness equalized (Fisher)
markets
supply shares / wealth
prices, revenue
law of one price, no arbitrage
inference
posterior probability
log-likelihood
likelihoods equalized / posterior support
Much like Fisher's fundamental theorem of natural selection we have the "fundamental theorem of welfare dynamics": fix the technologies and utilities and let all shares evolve simultaneously by (4.2). Then
Proof. (since the partials are total derivatives). Substituting (1.4) and (4.2), each block contributes .
5. Markets rationalize reward and identity
I started my MATS work with the following observation:
Utility functions and hyperstitional "identities" (in the sense of predictive processing---i.e. you do what you predict you'll do) are the same thing.
Both are "arbitrary": the choice of utility function, nor the co-ordinated choice of equilibrium (or whatever) are determined by existing rules of rationality.
Purely predictive agents (i.e. non-embedded agents, not Predict-o-matic) do not have identities; their beliefs are completely specified by Bayes (in the limit of all information being supplied). [I mean sure you could say that some priors may not allow convergence to the same thing, or debate what it means to take the limit of "all information being supplied", or talk about boundedly rational agents having "globally inconsistent" beliefs, but this seems like a qualitatively different thing.]
So ideally, we would like to have a theory where internal sub-agent co-ordination is equivalent to a choice of utility function. More precisely: the theory should connect this to reward (which ultimately "selects" or trains the identity), and build a satsifying picture that explains how that exogenous reward itself fits into a broader predictive-processing-only picture.
Or more simply: rational behaviour for a purely predictive/non-embedded agent is completely specified by Bayes's theorem, but for any agent that performs actions in the world/whose beliefs affect the world, there is an arbitrary thing called the "utility function". The utility function is shaped by reward, but then reward is arbitrary.
I continued:
The picture I expect right now is vaguely something like:
"the universe as a whole" (or closed systems more generally) do not have identities/utility functions, but individual sub-agents do
Identities are the result of the universe hierarchically enforcing its will on its sub-agents via reward/selection. [But the question of exactly how actions i.e. things causally determined by agent beliefs, emerge, should be clarified].
Reward has to do with constraints in statistical mechanics
This intuition is largely based on Point 2, and also the suggestive connection between evolutionary fitness functions and Bayesian evidence (Harper 2009, The Replicator Equation as an Inference Dynamic), which supports the idea that evolutionary selection is somehow the universe "inserting its information" into an agent.
Now observe how markets have two processes going on simultaneously:
Agents learn the shares , e.g. through continuous-time Bayesian inference, with prices (or rather revenue) as the "information" or reward supplied.
Price recursion i.e. equations (1.1)-(1.4) or equivalently (2.1)-(2.2) which follow a Bellman/backprop-like rule.
FAQ: Wait, how can price recursion be equivalent to both Bellman and backprop?
Backprop/reverse-mode autodiff on any computation graph produces a "Bellman-shaped" recursion, i.e. value of a node = sum of its consumer nodes' values transformed by .
What makes the market price recursion actual Bellman (with matrix-valued discount factors ) rather than merely "Bellman-shaped" is the simplex constraint which lets the routing shares be interpreted as probabilities.
In other words:
The economy is exactly the "super-agent" that generates reward to its sub-agents, and price recursion is the "rational theory of reward".
In Reward as hyperstitional information, reward was entirely an underdetermined hyperstition. Markets (equations (2.1)-(2.2)) tell you how to determine it.
appendix: Caveats and questions
... well, with two caveats:
I. Our market model still has utility functions at the end of the tunnel. Sure you can probably replace these with probabilities active inference style, but the point is they are still under-determined. So markets tell you how a super-agent gives rewards to its sub-agents, and maybe that super-agent gets rewards from its super-agent etc. but what is there "all the way up"?
E.g. factories' rewards are derived from human consumers' utility functions, humans' rewards are determined by evolution's fitness functions, evolution's fitness functions are themselves hyperstitional but maybe there is a grand cosmic-scale selection process that selects good evolutionary priors over bad evolutionary priors, and maybe everything somehow boils down to "the universe wants to maximize its entropy" or something like that.
Thermodynamics (free entropies) seems relevant to this.
II. Markets are just a model, right? It's not the case that all reward is computed based on price recursion (in fact nothing does perfectly so); rather we generally use heuristics. But maybe this is like how nobody actually follows Bayesian reasoning, but Bayes is still the ideal or optimal mechanism in some sense.
This would need a Dutch book type result.
And a third question, not crucial to the "price recursion rationalizes reward" philosophy but the continuous Bayesian inference dynamic which I still find important because .
III. Wealths and perfect competition. We kinda just postulated continuous Bayesian inference as the dynamics of markets because it feels right. A proper justification of this would require a game-theoretic consideration of agents maximizing their local objectives (presumably with an Aumann-style agent continuum) and having wealths/budgets.
AI tells me continuous Bayesian inference has a natural interpretation in terms of wealths (which kinda makes sense; accumulated evidence is wealth after all right).
Another questions I am interested in, that I feel is related but am not sure:
IV. "Dynamic rationality".
It seems to me that dynamics is fundamental to understanding "bounded rationality": the fact that you cannot instantly argmax a utility function or compute a market equilibrium. The fact that you can have systems that are "dynamically complete" (there is eventually an Arrow-Debreu security for every event in the world) and "dynamically consistent" (at most a finite amount of arbitrage can be extracted from the market in infinite time) but not complete and consistent (Gödel's theorem) also seems suggestive.
Questions:
Continuous Bayes feels like a natural model of dynamics: only an infinitesimal amount of information gets processed in infinitesimal time. How does this relate to all the other things about bounded rationality, like logical non-omniscience and Knightian uncertainty?
But Continuous Bayes only models inserting information, what about relaxing constraints/information (e.g. Newton's law of cooling)? Is there a model of the latter, and is it important? Feels like it might be relevant to Question I, where I spoke of thermodynamics.
It seems as though dynamics too may be "arbtitrary" like reward, and like reward, "selected by the super-agent"---i.e. markets prefer agents who learn, they prefer agents who learn faster, respond faster to shocks, etc.
(The original title for this was "Markets are equivalent to stuff", because I also demonstrate market-MDP/Bellman and market-backprop analogies. While those scratched an itch I've long had, the more important result is the headline one. Part of my work at MATS in Richard Ngo's stream.)
A bunch of people have vaguely noticed intuitive analogies between economies/markets and machine learning algorithms: John Wentworth, Aanjaneya Kumar and Anders Sandberg at the AI Objectives institute, the ancient Neural Bucket Brigade (Davis, Schmidhuber). Several papers discuss RL algorithms based on sequentially auctioning a unit "right to act" among agents---most recently Caspar Oesterheld's BRIA but also Baum's Hayek Machine, its predecessors the classifier systems (Holland 1985, Schmidhuber 1987, Schmidhuber 1989) and its successors Hayek-for-POMDPs (Kwee et al 2001) and Hayek with Vickrey auctions (Chang et al 2020). In "Market-based architectures in RL and beyond" I suggested that a better market model for RL might come out of factoring the "right to act" into "goods" axes, just like in real markets.
The analogy is attractive and suggestive:
In this article we will formalize this intuition, and explain that in fact, markets are equivalent to a lot of things: backprop, RL, and continuous Bayesian inference (aka "replicators").
In the first section we will describe the general economic model and how "prices" (gradients) arise naturally. In the subsequent sections we will start describing the analogies.
1. Setup
Here is how we will define an "economy".
Definition 1. There is a vector space
Where
We may imagine a global welfare objective
U.S.O.W. We will usually think of
(But what about factories choosing what to produce? Shouldn't
1.1. General price recursion
Introduce Lagrange multipliers
i.e. the price of output from
However, this does not directly let us perform backpropagation on the market graph, since
The forward computation is (where
Defining adjoints
You can transform the shares into logit space to do normal backpropagation without simplex constraints, or you can directly do optimization on a simplex, e.g. via entropic mirror ascent which gives the update rule (with geometric learning rate
2. Markets are MDPs; Advantage = Rivalrous goods
Here is a nice interpretation of our economic model. Imagine there is only one unit good, i.e. a particle in the whole market. If the production functions are identity, then its movement through the market is governed by policy
Production makes the particle "branch": at node
Optimizing this market (computing the
Or parameterizing with logits
Proof. Chain rule through the softmax Jacobian
In fact even the interpretation of
(Well actually, it kinda can be a perpetual motion machine, if the dynamics of the cyclic economy represent actual time rather than just convergence to an equilibrium. But tbh I would rather represent that unrolled, as an infinite graph with nodes stretching out into the future.)
(In the "wide market RL" I introduced in my paper Market-based RL, I don't think of the MDP itself as a market, but the transition probabilities are determined by an additional Garrabrant-like program market of traders and the network is more "fundamentally dynamic" rather than graph-like. Something in-between---dynamic like program markets but efficient like graphs---like a graph that forms new connections or performs search of some sort when shadow prices exceed some friction-induced background transaction cost, seems like a good model for describing real markets, Knightian uncertainty etc.)
3. Chain markets are MLPs
Theorem 1 (chain = MLP). Let the economy be a directed chain
Proof. 1–2 are the forward/backward recursions of Section 1 with a single out-edge; 3 is Lemma 4.1; 4 is construction.
Theorem 1 completely formalizes the "economies = backprop/neural networks" intuition that people have been groping at. Note that economies are more general than classical MLPs: they are neural networks with routing (which is kind of "transposed static attention": in markets, the goods are scarce while in transformers, the receiver-side attention budget is scarce).
4. Continuous Bayesian inference as a dynamics for markets
In my post Reward as hyperstitional information, I conveyed that entropic mirror ascent (aka continuous-time Bayesian inference) is the natural dynamics for probabilities. Rewards and such are just an "evidence rate".
It seems very suggestive that our routing shares are literally just probabilities (of "a particle of good being sent to some factory"). Indeed, we can let the dynamics of our market be "entropic mirror ascent on total welfare" i.e. update the shares each time step as (indices
Differentiate with respect to time to get the usual replicator flow/continuous-time Bayesian inference:
So: the market (under entropic mirror ascent) is a Bayesian learner, with each agent doing "hyperstitious" learning. I.e. each agent
Revenues are rewards are information (log-likelihoods).
You can use this equivalence to transfer any result.
Much like Fisher's fundamental theorem of natural selection we have the "fundamental theorem of welfare dynamics": fix the technologies
Proof.
5. Markets rationalize reward and identity
I started my MATS work with the following observation:
Or more simply: rational behaviour for a purely predictive/non-embedded agent is completely specified by Bayes's theorem, but for any agent that performs actions in the world/whose beliefs affect the world, there is an arbitrary thing called the "utility function". The utility function is shaped by reward, but then reward is arbitrary.
I continued:
Now observe how markets have two processes going on simultaneously:
In other words:
The economy is exactly the "super-agent" that generates reward to its sub-agents, and price recursion is the "rational theory of reward".
In Reward as hyperstitional information, reward was entirely an underdetermined hyperstition. Markets (equations (2.1)-(2.2)) tell you how to determine it.
appendix: Caveats and questions
... well, with two caveats:
I. Our market model still has utility functions at the end of the tunnel. Sure you can probably replace these with probabilities active inference style, but the point is they are still under-determined. So markets tell you how a super-agent gives rewards to its sub-agents, and maybe that super-agent gets rewards from its super-agent etc. but what is there "all the way up"?
E.g. factories' rewards are derived from human consumers' utility functions, humans' rewards are determined by evolution's fitness functions, evolution's fitness functions are themselves hyperstitional but maybe there is a grand cosmic-scale selection process that selects good evolutionary priors over bad evolutionary priors, and maybe everything somehow boils down to "the universe wants to maximize its entropy" or something like that.
Thermodynamics (free entropies) seems relevant to this.
II. Markets are just a model, right? It's not the case that all reward is computed based on price recursion (in fact nothing does perfectly so); rather we generally use heuristics. But maybe this is like how nobody actually follows Bayesian reasoning, but Bayes is still the ideal or optimal mechanism in some sense.
This would need a Dutch book type result.
And a third question, not crucial to the "price recursion rationalizes reward" philosophy but the continuous Bayesian inference dynamic which I still find important because .
III. Wealths and perfect competition. We kinda just postulated continuous Bayesian inference as the dynamics of markets because it feels right. A proper justification of this would require a game-theoretic consideration of agents maximizing their local objectives (presumably with an Aumann-style agent continuum) and having wealths/budgets.
AI tells me continuous Bayesian inference has a natural interpretation in terms of wealths (which kinda makes sense; accumulated evidence is wealth after all right).
Another questions I am interested in, that I feel is related but am not sure:
IV. "Dynamic rationality".
It seems to me that dynamics is fundamental to understanding "bounded rationality": the fact that you cannot instantly argmax a utility function or compute a market equilibrium. The fact that you can have systems that are "dynamically complete" (there is eventually an Arrow-Debreu security for every event in the world) and "dynamically consistent" (at most a finite amount of arbitrage can be extracted from the market in infinite time) but not complete and consistent (Gödel's theorem) also seems suggestive.
Questions: