Alice: A lot of people are talking about Singular Learning Theory. Do you know what it is?

Bob: I do. (pause) Kind of.

Alice: Well, I don't. Explanation time?

Bob: Uh, I'm not really an expert on it. You know, there's a lot of materials out there that--

Alice: that I realistically won't ever actually look at. Or, I've looked at them a little, but I still have basically no idea what's going on. Maybe if I watched a dozen hours of introductory lectures I'd start to understand it, but that's not currently happening.

What I really want is a short overview of what's going on. That's self-contained. And easy to follow. Aimed at a non-expert. And which perfectly answers any questions I might have. So, I thought I'd ask you!

Bob: Sorry, I'm actually really not--

Alice: Pleeeease?

[pause]

Bob: Ah, fine, I'll try.

So, you might have heard of ML models being hard to interpret. Singular Learning Theory (SLT) is an approach for understanding models better. Or, that's one motivation, at least.

Alice: And how's this different from a trillion other approaches to understanding AI?

Bob: A core perspective of SLT is studying how the model develops during training. Contrast this to, say, mechanistic interpretability, which mostly looks at the fully trained model. SLT is also more concerned about higher level properties.

As a half-baked analogue, you can imagine two approaches to studying how humans work: You could just open up a human and see what's inside. Or, you could notice that, hey, you have these babies, which grow up into children, go through puberty, et cetera, what's up with that? What are the different stages of development? Where do babies come from? And SLT is more like the second approach.

Alice: This makes sense as a strategy, but I strongly suspect you don't currently know what an LLM's puberty looks like.

Bob: (laughs) No, not yet.

Alice: So what do you actually have?

Bob: The SLT people have some quite solid theory, and some empirical work building on top of that. Maybe I'll start from the theory, and then cover some of the empirical work.

Alice: (nods)

I. Theoretical foundations

Bob: So, as you know, nowadays the big models are trained with gradient descent. As you also know, there's more to AI than gradient descent. And for a moment we'll be looking at the Bayesian setting, not gradient descent.

Alice: Elaborate on "Bayesian setting"?

Bob: Imagine a standard deep lear...

Dialogue (format)