(This article broadly explains mirror ascent, continuous Bayesian inference and information geometry in full. Title refers to the result in section 3.)

The logarithmic scoring rule exactly puts a price on information by the usual measure: the expected score an agent with some belief distribution gets is exactly where is the true distribution (or, more accurately: the score we expect that agent to get is , from our perspective/belief ).

Consequently, the value of some information is precisely its mutual information with the event of interest, etc.

Here's a question: does every proper scoring rule correspond to an information measure? For instance, does the quadratic scoring rule incentivize obtaining information in the sense of "quadratic negentropy" much like logarithmic scoring incentivizes obtaining information in the sense of "logarithmic negentropy"?

generalized entropies and Bregman divergence

A scoring rule is some function that tells you how much an agent gets for belief when an outcome is realized---equivalently that is how someone with belief would score belief . This also tells you how someone with belief would score belief :

Properness means that maximizes it, i.e.

is the score we give to our own beliefs, thus there is no bias penalty, only an uncertainty penalty: it measures our "certainty" about our own beliefs.

(So and are the generalized "negentropy" and "cross-negentropy" respectively, and their negatives are the generalized entropy and cross-entropy.)

Now observe: fix and look at the maps and . The latter is linear in , and they touch at the point (everywhere else lies below ).

Thus for every , is the tangent plane under the graph of at the point . In other words, any (neg-)entropy function naturally induces a cross (neg-)entropy function by its tangent planes:

When is convex, i.e. it sits above all its tangent planes (in fact it is exactly the max of all its tangent planes), the resulting scoring rule is proper i.e. . Thus every convex function generates an information theory.

So the Generalized Scoring , the Generalized Cross-Entropy and the Generalized Entropy are all equally fundamental; you can derive the rest from any one of them.

We also define the Generalized KL-divergence ("Bregman Divergence") as the vertical gap between and the tangent plane , i.e. "how much does believe itself to be better than ":

For log score , we have usual information theory. For the quadratic score we have cross-negentropy , negentropy (the negative of which, the quadratic entropy, is called the "Gini impurity") and Bregman divergence .

An entropy function inducing a cross-entropy function may remind you of how norms naturally induce inner products (polarization identity). So is cross-entropy an inner product? Not really. If you take a quadratic form (differing from the above quadratic entropy by constant term and factor) then , and

Which is not the inner product, but it does suggest a generalized "inner product" of distributions: . But it is not in general symmetric, etc.

mirror ascent

What exactly is gradient ascent?

"Climbing in the direction of steepest ascent"

ok, what is "the direction of steepest ascent?"

Among all -steps in each direction, we choose the -step that gives the highest rise in the objective (as measured by a first-order Taylor approximation of the objective, i.e. the gradient)

I.e. given a current value , we want to find the that maximizes . The first-order Taylor approximation for this is ---but we cannot trust the first-order Taylor approximation if we move too far, so we restrict the search to an -neighboruhood around . Thus:

This is a constrained optimization problem, so define a Lagrangian:

Differentiating we get:

For the usual Euclidean metric, so we can solve and get usual gradient ascent:

Instead suppose is an arbitrary Bregman divergence, i.e. for convex . The gradient of this with respect to its first parameter is simply

So plugging this back into we get the update rule:

For convex , there is a one-to-one map between the values of and the values of ; i.e. the gradients live in a "-dual space" or mirror space and this amounts exactly to doing gradient ascent in the mirror space, i.e. "mirror ascent".

For (the usual quadratic metric), the map is the identity (scaled by ), and---

reward is hyperstitional information

for we have thus the update rule is multiplicative (entropic mirror ascent):

(we are allowed to normalize because for probabilities the original problem is itself a constrained optimization problem with a simplex constraint)

In other words: optimizing some objective over probability space with entropic mirror ascent, is continuous-time Bayesian inference with likelihood function .

Unlike usual Bayesian inference:

1. this likelihood function depends on the current
2. this likelihood function can be ... anything.

From a predictive processing/active infernece perspective, all rational behaviour must be Bayesian inference, and utility functions are fake. What you see as "utility functions"/goals are actually the agent's prior: and inference arbitrages between the agent's beliefs and the environment (when the environment's information updates the agent that is "learning"; when the agent's information influences the environment that is "action").

And similarly, now we have that reward is also, information. What is it information about? Well, an RL agent produces probabilities for its actions and samples from them (so the policy also functions as the agent's beliefs about its own actions), and reward updates these probabilities---so it is simply information about "what I will do", and is definitionally rational, because "what I will do" is actually updated and the agent's beliefs about it are updated in-sync. You can specify any objective whatsoever, and optimizing it (at least through entropic mirror ascent) will still be rational Bayesian inference. In other words, reward is hyperstitional information.

(And if you multiply all the exponentiated rewards the agent gets through its training, you get its final prior, which expresses its exponentiated utility function. Kinda, maybe, oversimplified, whateever.)

In bandits, the objective is just the reward; in more general RL you must use Q-values. PPO methods used in modern RL also are approximately entropic mirror ascent (TRPO is exactly it; other stuff is kinda). You may also recall that evolution (replicator dynamics) is Bayesian inference (Harper 2009, explained by John Baez here). Indeed, it is exactly equivalent to this dynamic, with the fitness function as an exponentiated reward.

continuous-time mirror ascent and information geometry

It is interesting to take the continuous limit of this dynamics. We have:

(i.e. the velocity in the mirror space is the gradient of your objective function). What does this look like in the original primal space? Well we can apply the chain rule to rewrite the left side as (where is the Hessian) thus:

This gives us a very natural interpretation of mirror ascent: if you let the Hessian defines a Riemannian metric on your space, then mirror ascent is simply gradient ascent of the objective on that geometry. Physicists are of course quite used to transforming between primal and dual spaces by multiplying by the metric and its inverse respectively.

Pitfall prevention: this is not gravity, nor is it any Newtonian force (as that would drive the acceleration not the velocity).

When is entropy, its Hessian is called the "Fisher information matrix" or "Fisher-Rao metric" and the geometry defined by this Riemannian metric is called information geometry.

what I'm confused about: dynamics

Finally, the thing I am actually confused about:

You know, a big problem in life is that people like expressing everything in terms of equilibrium (in economics, game theory, entropostatics aka thermodynamics) with little regard for the dynamics that leads to the equilibrium.

(This is closely related to how we just model agents as rational (i.e. imagine that the utility function can just be arg-maxed) instead of modeling bounded rationality which has computational costs (and you can't just minus them from the objective either, because you don't know what those costs are) and is fundamentally dynamic because you can't instantly optimize something but that computational time is itself a cost...)

Entropic mirror ascent seems like a very promising model of dynamics. Because evolution (the most fundamental learning algorithm) actually follows it, and it is naturally a dynamical Bayesian inference. And I have some work suggesting that economies can naturally be thought of as doing it too.

But on the other hand ... I can't imagine how dynamical thermodynamics could have anything to do with this? The simplest actually dynamical example in thermodynamics is Newton's law of cooling, and that actually happens to be gradient ascent on entropy, not continuous Bayesian inference/entropic mirror ascent.

Modeling thermodynamic dynamics fundamentally seems to have something to do with thinking like "Ok, I am not completely uncertain about this because I know where the system was 1 second ago; that is somehow a soft constraint", i.e. a generalized notion of "fraction". And I don't even know how gradient ascent on entropy comes out of this, maybe Newton's law of cooling is just a coincidence?