I guess (but don't know) that most people who downvote Garrett's comment overupdated on intuitive explanations of singular learning theory, not realizing that entire books with novel and nontrivial mathematical theory have been written on it. 

I do all of these except 3, and implementing a system like 3 is among my deprioritized things in my ToDo-list. Maybe I should prioritize it.

Yes the first! Thanks for the link!

I really enjoyed reading this post! It's quite well-written. Thanks for writing it.

The only critique is that I would have appreciated more details on how the linear regression parameters are trained and what exactly the projection is doing. John's thread is a bit clarifying on this.

One question: If you optimize the representation in the residual stream such that it corresponds to a particular chosen belief state, does the transformer than predict the next token as if in that belief state? I.e., does the transformer use the belief state for making predictions?

MATS mentorships are often weekly, but only for limited time, unlike PhD programs that offer mentorship for several years. These years are probably often necessary to develop good research taste.

(Fwiw, I don’t remember problems with stipend payout at seri mats in the winter program. I was a winter scholar 2022/23.)

This is very helpful, thanks! Actually, the post includes several sections, including in the appendix, that might be more interesting to many readers than the grant recommendations themselves. Maybe it would be good to change the title a bit so that people also expect other updates.

Thanks for the reply!

As I show in the examples in DSLT1, having degenerate Fisher information (i.e. degenerate Hessian at zeroes) comes in two essential flavours: having rank-deficiency, and having vanishing second-derivative (i.e. $K\left(w\right)=w^4$). Precisely, suppose $d$ is the number of parameters, then you are in the regular case if $K\left(w\right)$ can be expressed as a full-rank quadratic form near each singularity, 

$$K\left(w\right)=\sum _{i=1}^d{w}_i^2.$$

Anything less than this is a strictly singular case. 

So if $K\left(w\right)=w^2$, then $w=0$ is a singularity but not a strict singularity, do you agree? It still feels like somewhat bad terminology to me, but maybe it's justified from the algebraic-geometry--perspective. 

Zeta Functions in Singular Learning Theory

In this shortform, I very briefly explain my understanding of how zeta functions play a role in the derivation of the free energy in singular learning theory. This is entirely based on slide 14 of the SLT low 4 talk of the recent summit on SLT and Alignment, so feel free to ignore this shortform and simply watch the video.

The story is this: we have a prior $\phi \left(w\right)$, a model $p(x\mid w)$, and there is an unknown true distribution $q\left(x\right)$. For model selection, we are interested in the evidence of our model for a data set $D_n=\{x_1,\dots ,x_n\}$, which is given by

where $K_n\left(w\right)=\frac{1}{n}{\sum }_{i=1}^n\log \frac{q\left(x_i\right)}{p(x_i\mid w)}$ is the empirical KL divergence. In fact, we are interested in selecting the model that maximizes the average of this quantity over all data sets. The average is then given by

where $K\left(w\right)={\int }_{x\sim q}q\left(x\right)\log \frac{q\left(x\right)}{p(x\mid w)}$ is the Kullback-Leibler divergence. 

But now we have a problem: how do we compute this integral? Computing this integral is what the free energy formula is about. 

The answer: by computing a different integral. So now, I'll explain the connection to different integrals we can draw. 

Let

which is called the state density function. Here, $\delta$ is the Dirac delta function.  For different $t$, it measures the density of states (= parameter vectors) that have $K\left(w\right)=t$. It is thus a measure for the "size" of different level sets. This state density function is connected to two different things. 

Laplace Transform to the Evidence

First of all, it is connected to the evidence above. Namely, let $L\left(v\right)$ be the Laplace transform of $v$. It is a function $L\left(v\right):N\to R$ given by

In first step, we changed the order of integration, and in the second step we used the defining property of the Dirac delta. Great, so this tells us that $L\left(v\right)=\overline{Z}$! So this means we essentially just need to understand $v$.

Mellin Transform to the Zeta Function

But how do we compute $v$? By using another transform. Let $M\left(v\right)$ be the Mellin transform of $v$. It is a function $M\left(v\right):C\to C$ (or maybe only defined on part of $C$?) given by

Again, we used a change in the order of integration and then the defining property of the Dirac delta. This is called a Zeta function. 

What's this useful for?

The Mellin transform has an inverse. Thus, if we can compute the zeta function, we can also compute the original evidence as

Thus, we essentially changed our problem to the problem of studying the zeta function $M\left(v\right).$ To compute the integral of the zeta function, it is then useful to perform blowups to resolve the singularities in the set of minima of $K\left(w\right)$, which is where algebraic geometry enters the picture. For more on all of this, I refer, again, to the excellent SLT low 4 talk of the recent summit on singular learning theory. 

Thanks for the answer! I think my first question was confused because I didn't realize you were talking about local free energies instead of the global one :) 

As discussed in the comment in your DSLT1 question, they are both singularities of $K\left(w\right)$ since they are both critical points (local minima).

Oh, I actually may have missed that aspect of your answer back then. I'm confused by that: in algebraic geometry, the zero's of a set of polynomials are not necessarily already singularities. E.g., in $f(x,y)=xy$, the zero set consists of the two axes, which form an algebraic variety, but only at $(0,0)$ is there a singularity because the derivative disappears.
Now, for the KL-divergence, the situation seems more extreme: The zero's are also, at the same time, the minima of $K$, and thus, the derivative disappears at every point in the set $W_0$. This suggests every point in $W_0$ is singular. Is this correct?

So far, I thought "being singular" means the effective number of parameters around the singularity is lower than the full number of parameters. Also, I thought that it's about the rank of the Hessian, not the vanishing of the derivative. Both perspectives contradict the interpretation in the preceding paragraph, which leaves me confused. 

The uninteresting answer is that SLT doesn't care about the prior (other than its regularity conditions) since it is irrelevant in the $n\to \infty$ limit.

I vaguely remember that there is a part in the MDL book by Grünwald where he explains how using a good prior such as Jeffrey's prior somewhat changes asymptotic behavior for $n\to \infty$, but I'm not certain of that.