We will continue where we left off. In the last post, I introduced a few important observables along with the setup. We are interested in their asymptotics, and the behaviors of these random variables. But before we dive into them, we first see some motivation for their definitions, and how they are all connected to each other (the WAIC definition was pretty non-intuitive to me the first time I saw it). Somewhere along the middle, I will slightly detach from Watanabe's notation to do this and approach this topic differently, before converging back to his notation. This is primarily done to make the relationships between the observables clearer, and to give a more detailed introduction. Once this is done, we will start with the asymptotics.
How do we see the relationships between those observables though, and even try to begin their asymptotics? To help you get settled with the mindset, I will give you the spoiler beforehand: Generating Functions.
For an arbitrary triple of a true distribution, a statistical model, and a prior, the behaviors of the free energy, the losses, and WAIC are derived by the following procedure:
Firstly, we define the formal relation between a true distribution and a statistical model.
Secondly, definitions of Bayesian observables and their normalized versions are introduced.
Thirdly, the cumulant generating functions of the Bayesian prediction are defined.
The Basic Theory of Bayesian Statistics is proved using the cumulant generating functions.
Refer to this site for the notebook version of this, examples and exercises with solutions.
All feedback is appreciated.
The Formal Relation
We make 4 main definitions right now.
Definition 1 (Realizability): Let be the set of all parameters. If there exists such that almost surely, then is said to be realizable by . For a given pair of true distribution and statistical model, the set of true parameters is defined by
Thus, realizability holds iff the set of true parameters is not an empty set (one can see this by using the fact that the integral of both the functions is 1 and then using the almost sure condition).
We can alternatively write
The average log loss function is defined by
It follows that
where is the entropy of the true distribution and denotes the KL divergence. Hence, if is realizable by a statistical model, then the average log loss function is minimized iff , and the minimum value is the entropy of the true distribution.
Some motivation: As we continue the learning process, our statistical model gets closer to the true distribution. The realizability assumption makes it simple for us and tells us that our statistical model can represent the true distribution, which need not be the case in general.
Definition 2 (Regularity): For a given pair of and , let , which is called the set of optimal parameters for the minimum average log loss. is said to be regular for if the following three conditions hold:
1) is a singleton set.
2) is in the interior of ( is equipped with the subspace topology from ).
3) The Hessian matrix of the average log loss function at is positive definite.
Let us see a simple consequence of these definitions. If is a compact set and if is a continuous function, then has a minimum point, hence is not an empty set. In this case, is realizable by iff (reason this by yourself).
Some motivation: The regularity condition tells us that we can approximate the loss landscape near its minimum by a parabola. This follows by doing a Taylor approximation, doing an eigendecomposition of the Hessian, and then using the fact that all the eigenvalues are positive (do the exact calculations yourself). One of the main claims is that this is a terrible approximation in many cases and does not hold for the loss landscapes of general neural networks.
Definition 3 (Essential Uniqueness): Assume that is not an empty set. If there exists a unique probability density function such that for arbitrary , , then it is said that the optimal probability density function is essentially unique.
If is realizable by , then the optimal probability density function is essentially unique, because . Thus, realizability essential uniqueness.
Definition 4 (Relatively Finite Variance of Log Density Ratio Function): For a given pair and , the log density ratio function is defined by
If there exists such that for an arbitrary pair ,
then the function has a relatively finite variance.
Remark: The function has a relatively finite variance iff
Some motivation: What is the purpose of the theory that we are developing? We want to study the model's distance from the truth, and how that develops over time. We want to abstract out the truth's randomness from our process. Let us see what this condition is really about.
Let us look at the average log loss function (do you prefer to look at the asymptotics of a sum or a product? Take log.) whose definition is derived naturally from the average loss function.
Removing the entropy considerations from , we get the "excess log loss over the truth per data point", which is what the definition is in the realizable case (but look at the next lemma to understand the point in the general case). Thus the importance. This remark also says the mathematical statement to note, that the expectation of the log density ratio is the KL divergence between the true distribution and the model in the realizable case.
Lemma: Assume that for arbitrary and . If has a relatively finite variance, then the optimal probability density is essentially unique.
Assume that and are arbitrary elements of . By the definition of ,
and as the integrand is non-negative, we get almost surely with respect to , hence almost surely with respect to . Do observe that this is not a clever proof, rather just making use of all the given information.
Thus, we may denote . It thus follows that
Thus, relatively finite variance essential uniqueness. However, the converse need not be true. It does hold with a few added conditions.
Lemma: Assume that is a compact set and that is realizable by and that the log density ratio function is a continuous function of . If there exist such that for an arbitrary ,
then has a relatively finite variance.
I refer the reader to the book for the proof. It is not important to the rest of the theory, just some technical details.
Lemma: Assume that is a compact set and that for an arbitrary pair and , the second derivatives of and are continuous functions. If is regular for , then the log density ratio function has a relatively finite variance.
The proof of this lemma is alongside the proof of the previous lemma.
Assume that is compact. We thus have the following relations:
1) RegularRelatively Finite Variance.
2) RealizableRelatively Finite Variance.
3) Relatively Finite VarianceEssentially Unique.
Normalized Variables
We have the average and empirical log loss functions:
Assume now that has a relatively finite variance. Then the optimal probability density is essentially unique, so we may denote the log density ratio by .
See that
We define the normalized average log loss function as . A key thing to note in all the definitions is that we will be rewriting them in terms of . The formal advantage is explained in a short while, but the understanding is that we want to remove the entropy of the optimal probability density and consider only the KL divergence between and the model , as that is what we want to analyse (how that changes over time and what are the asymptotics).
(For this to make sense, note that , which equals in the realizable case.)
Similarly, we have the normalized empirical log loss function
Furthermore, a key property of our normalized function is that , and .
The partition function is defined as
Thus,
Thus,
We define this as the normalized marginal likelihood .
Thus,
Let us recall that the posterior is
Free energy is the negative log of the partition function. The normalized free energy is defined similarly.
Here is the table for the generalization, cross validation, and training losses and the WAIC, and also their normalized versions.
Observable
Definition
Normalization
Generalization loss
Training Loss
Cross Validation Loss
WAIC
Let us connect all these observables.
Definition: For , we define the -predictive at as
Note that this is the -power mean of with respect to the posterior distribution. Essentially, the observables come from this family. The three main predictives from the family are the Bayes Posterior (), the Gibbs posterior (, one would need to prove that the limit exists. But this case is not important for now), and the term for the cross validation loss ().
Recall that the moment generating function of a random variable with PDF is given by and the cumulant generating function is
We will look at the cumulant generating function of the log likelihood at (with a little notational change) with respect to the posterior, which is exactly the log of the .
We can now write the observables in terms of the CGF.
Thus they are all derived from the CGF. We will discuss WAIC shortly, and also note how it comes about.
Let us discuss the normalization first.
We define
in the normalized coordinates.
The corresponding definitions in terms of give the normalized variables.
Note that
The power of introducing the CGF is to be able to do a Taylor expansion here.
Doing a Taylor approximation near 0 (here ),
where
So,
Summing over the training points and writing for the accumulated remainder terms, we get
This gives rise to WAIC, and using the normalized CGF gives rise to the normalized WAIC (refer to the table for the definition one last time).
Caveat: Watanabe does an abuse of notation (harmless) and defines it the following way. We will continue with this.
Thus, his definition is just the mean of our definitions. However, do note that the two means are different, since and do not commute.
I will now state the result which explains why we need to look at the normalized observables.
All these asymptotics abstract away from the prior. In the realizable case, they abstract away from the statistical model as well.
We need to clarify the behavior of these random variables, look at higher order terms. The normalized versions of these all converge to 0 in probability, removing the effects of and and focusing on what is necessary.
We will now explain the asymptotics. We will be considering the normalized observables (and drop the (0) superscript):
Theorem: Let be random variables, and assume that
(That is, inside radius 1, the third derivatives of the function are bounded by the term).
Then,
Proof: By Taylor's theorem with the Lagrange remainder, given , there exists s.t. and
gives the theorem (similar method for dealing with ).
We thus get the theoretical behaviors of the free energy, generalization loss, and the rest by the following procedure.
Recipe for Bayesian Theory Construction
An arbitrary triple of is fixed. The set of parameters is denoted , and .
The empirical and average log loss functions are defined by
Find the optimal parameters minimizing .
Check that the log density ratio function has a relatively finite variance. Then we have essential uniqueness.
Define
The normalized partition function is given by
Then the free energy
The average by the posterior is equal to
Calculate and . From these we obtain
and the empirical variance term , which estimates .
Finally, based on the basic theorem,
and has the same expansion as (which also proves that ).
Further Steps
We have already set up the framework and the basic theory, and also discovered important results (which were not restricted to regular models). We are now ready to find the behavior of the regular posterior distribution, and then generalize that behavior to the general case.
Foreword
We will continue where we left off. In the last post, I introduced a few important observables along with the setup. We are interested in their asymptotics, and the behaviors of these random variables. But before we dive into them, we first see some motivation for their definitions, and how they are all connected to each other (the WAIC definition was pretty non-intuitive to me the first time I saw it). Somewhere along the middle, I will slightly detach from Watanabe's notation to do this and approach this topic differently, before converging back to his notation. This is primarily done to make the relationships between the observables clearer, and to give a more detailed introduction. Once this is done, we will start with the asymptotics.
How do we see the relationships between those observables though, and even try to begin their asymptotics? To help you get settled with the mindset, I will give you the spoiler beforehand: Generating Functions.
For an arbitrary triple of a true distribution, a statistical model, and a prior, the behaviors of the free energy, the losses, and WAIC are derived by the following procedure:
Refer to this site for the notebook version of this, examples and exercises with solutions.
All feedback is appreciated.
The Formal Relation
We make 4 main definitions right now.
Thus, realizability holds iff the set of true parameters is not an empty set (one can see this by using the fact that the integral of both the functions is 1 and then using the almost sure condition).
We can alternatively write
The average log loss function is defined by
It follows that
where
Some motivation: As we continue the learning process, our statistical model gets closer to the true distribution. The realizability assumption makes it simple for us and tells us that our statistical model can represent the true distribution, which need not be the case in general.
Let us see a simple consequence of these definitions. If
Some motivation: The regularity condition tells us that we can approximate the loss landscape near its minimum by a parabola. This follows by doing a Taylor approximation, doing an eigendecomposition of the Hessian, and then using the fact that all the eigenvalues are positive (do the exact calculations yourself). One of the main claims is that this is a terrible approximation in many cases and does not hold for the loss landscapes of general neural networks.
If
Remark: The function
Some motivation: What is the purpose of the theory that we are developing? We want to study the model's distance from the truth, and how that develops over time. We want to abstract out the truth's randomness from our process. Let us see what this condition is really about.
Let us look at the average log loss function (do you prefer to look at the asymptotics of a sum or a product? Take log.) whose definition is derived naturally from the average loss function.
Removing the entropy considerations from
Assume that
and as the integrand is non-negative, we get
Thus, we may denote
Thus, relatively finite variance
I refer the reader to the book for the proof. It is not important to the rest of the theory, just some technical details.
Lemma: Assume that
The proof of this lemma is alongside the proof of the previous lemma.
Assume that
1)
2)
3)
Normalized Variables
We have the average and empirical log loss functions:
Assume now that
See that
We define the normalized average log loss function as
(For this to make sense, note that
Similarly, we have the normalized empirical log loss function
Furthermore, a key property of our normalized function is that
The partition function is defined as
Thus,
Thus,
We define this as the normalized marginal likelihood
Thus,
Let us recall that the posterior is
Free energy is the negative log of the partition function. The normalized free energy is defined similarly.
Here is the table for the generalization, cross validation, and training losses and the WAIC, and also their normalized versions.
Observable
Definition
Normalization
Generalization loss
Training Loss
Cross Validation Loss
WAIC
Let us connect all these observables.
Definition: For
Note that this is the
Essentially, the observables come from this family. The three main predictives from the family are the Bayes Posterior (
Recall that the moment generating function of a random variable
We will look at the cumulant generating function of the log likelihood at
We can now write the observables in terms of the CGF.
Thus they are all derived from the CGF. We will discuss WAIC shortly, and also note how it comes about.
Let us discuss the normalization first.
We define
in the normalized coordinates.
The corresponding definitions in terms of
Note that
The power of introducing the CGF is to be able to do a Taylor expansion here.
Doing a Taylor approximation near 0 (here
where
So,
Summing over the training points and writing
This gives rise to WAIC, and using the normalized CGF gives rise to the normalized WAIC (refer to the table for the definition one last time).
Caveat: Watanabe does an abuse of notation (harmless) and defines it the following way. We will continue with this.
Thus, his definition is just the mean of our definitions. However, do note that the two means are different, since
I will now state the result which explains why we need to look at the normalized observables.
All these asymptotics abstract away from the prior. In the realizable case, they abstract away from the statistical model as well.
We need to clarify the behavior of these random variables, look at higher order terms. The normalized versions of these all converge to 0 in probability, removing the effects of
We will now explain the asymptotics. We will be considering the normalized observables (and drop the (0) superscript):
Proof: By Taylor's theorem with the Lagrange remainder, given
We thus get the theoretical behaviors of the free energy, generalization loss, and the rest by the following procedure.
Recipe for Bayesian Theory Construction
Find the optimal parameters minimizing
The normalized partition function is given by
Then the free energy
Calculate
and the empirical variance term
Finally, based on the basic theorem,
and
Further Steps
We have already set up the framework and the basic theory, and also discovered important results (which were not restricted to regular models). We are now ready to find the behavior of the regular posterior distribution, and then generalize that behavior to the general case.