Announcing Timaeus

Daniel Murfet; Alexander Gietelink Oldenziel; Stan van Wingerden

The strongest critique of developmental interpretability we know is the following: while it is established that phase transitions exist in neural network training, it is not yet clear how common they are, and whether they make a good target for alignment.

Is it established that phase transitions exist in the training of non-toy neural networks?

There are clearly s-shaped loss curves in many non-toy cases, but I'm not aware of any known cases which are clearly phase transitions as defined here (which is how the term is commonly used in e.g. physics and how I think this post wants to use the term).

For instance, while formation of induction-like attention heads^[1] probably results in s-shaped loss curves in at least some cases, my understanding is that this probably has nothing to do with changes in the minima of some notion of energy (as would be required for the definition linked above I think). I think the effect is probably the one described in Multi-Component Learning and S-Curves. Unless there is some notion of energy such that this multi-component case of s-shaped loss curves is well described as a phase transition and that's what's discussed in this post?

Some important disclaimers:

I'm not very familiar with the notion of phase transition I think this post is using.
It seems as though there are cases where real phase transitions occur during the training of toy models and there are also phase transitions in the final configuration of toy models as various hyperparameters change. (I'm pretty confident that Toy models of superposition has both phase transitions during training and in final configurations (for phase transitions in final configurations there seems to be one with respect to "does superposition occur").There is also this example of a two layer ReLU model; however, I haven't read or understood this example, I'm just trusting the authors claim that there is a phase transition here).

These attention heads probably do a bunch of stuff which isn't that well described as induction, so I'm reluctant to call them "induction heads". ↩︎

[-]Daniel Murfet2yΩ17329

Great question, thanks. tldr it depends what you mean by established, probably the obstacle to establishing such a thing is lower than you think.

To clarify the two types of phase transitions involved here, in the terminology of Chen et al:

Bayesian phase transition in number of samples: as discussed in the post you link to in Liam's sequence, where the concentration of the Bayesian posterior shifts suddenly from one region of parameter space to another, as the number of samples increased past some critical sample size . There are also Bayesian phase transitions with respect to hyperparameters (such as variations in the true distribution) but those are not what we're talking about here.
Dynamical phase transitions: the "backwards S-shaped loss curve". I don't believe there is an agreed-upon formal definition of what people mean by this kind of phase transition in the deep learning literature, but what we mean by it is that the SGD trajectory is for some time strongly influenced (e.g. in the neighbourhood of) a critical point $w_{α}^{*}$ and then strongly influenced by another critical point $w_{β}^{*}$ . In the clearest case there are two plateaus, the one with higher loss corresponding to the label $α$ and the one with the lower loss corresponding to $β$ . In larger systems there may not be a clear plateau (e.g. in the case of induction heads that you mention) but it may still reasonable to think of the trajectory as dominated by the critical points.

The former kind of phase transition is a first-order phase transition in the sense of statistical physics, once you relate the posterior to a Boltzmann distribution. The latter is a notion that belongs more to the theory of dynamical systems or potentially catastrophe theory. The link between these two notions is, as you say, not obvious.

However Singular Learning Theory (SLT) does provide a link, which we explore in Chen et al. SLT says that the phases of Bayesian learning are also dominated by critical points of the loss, and so you can ask whether a given dynamical phase transition $α \to β$ has "standing behind it" a Bayesian phase transition where at some critical sample size the posterior shifts from being concentrated near $w_{α}^{*}$ to being concentrated near $w_{β}^{*}$ .

It turns out that, at least for sufficiently large $n$ , the only real obstruction to this Bayesian phase transition existing is that the local learning coefficient near $w_{β}^{*}$ should be higher than near $w_{α}^{*}$ . This will be hard to prove theoretically in non-toy systems, but we can estimate the local learning coefficient, compare them, and thereby provide evidence that a Bayesian phase transition exists.

This has been done in the Toy Model of Superposition in Chen et al, and we're in the process of looking at a range of larger systems including induction heads. We're not ready to share those results yet, but I would point you to Nina Rimsky and Dmitry Vaintrob's nice post on modular addition which I would say provides evidence for a Bayesian phase transition in that setting.

There are some caveats and details, that I can go into if you're interested. I would say the existence of Bayesian phase transitions in non-toy neural networks is not established yet, but at this point I think we can be reasonably confident they exist.

[-]ryan_greenblatt2yΩ370

Thanks for the detailed response!

So, to check my understanding:

The toy cases discussed in Multi-Component Learning and S-Curves are clearly dynamical phase transitions. (It's easy to establish dynamical phase transitions based on just observation in general. And, in these cases we can verify this property holds for the corresponding differential equations (and step size is unimportant so differential equations are a good model).) Also, I speculate it's easy to prove the existence of a bayesian phase transition in the number of samples for these toy cases given how simple they are.

[-]Daniel Murfet2y60

Yes I think that's right. I haven't closely read the post you link to (but it's interesting and I'm glad to have it brought to my attention, thanks) but it seems related to the kind of dynamical transitions we talk briefly about in the Related Works section of Chen et al.

[-]ryan_greenblatt2yΩ122012

More generally, I wish that when people used the term "phase transition", they clarified whether they meant "s-shaped loss curves" or some more precise notion. Often, people are making a non-mechanistic claim when they say "phase transition" (we observed a loss curve with a s-shape), but there are also mechanistic claims which require additional evidence.

In particular, when citing other work somewhere, it would be nice to clarify what notion of phase transition the other work is discussing.

[-]Orpheus162y190

Congrats on launching!

Besides funding, are there things other people could provide to help your group succeed? Are there particular types of people you’d be excited to collaborate with, get mentorship/advice from, etc?

[-]Daniel Murfet2y204

Thanks Akash. Speaking for myself, I have plenty of experience supervising MSc and PhD students and running an academic research group, but scientific institution building is a next-level problem. I have spent time reading and thinking about it, but it would be great to be connected to people with first-hand experience or who have thought more deeply about it, e.g.

People with advice on running distributed scientific research groups
People who have thought about scientific institution building in general (e.g. those with experience starting FROs in biosciences or elsewhere)
People with experience balancing fundamental research with product development within an institution

I am seeking advice from people within my institution (University of Melbourne) but Timaeus is not a purely academic org and their experience does not cover all the hard parts.

[-]plex2y177

Congratulations on launching!

Added you to the map:

and your Discord to the list of communities, which is now a sub-page of aisafety.com.

One question: Given that interpretability might well lead to systems which are powerful enough to be an x-risk long before we have a strong enough understanding to direct a superintelligence, so publish-by-default seems risky, are you considering adopting a non-publish-by-default policy? I know you talk about capabilities risks in general terms, but is this specific policy on the table?

[-]Jesse Hoogland2y53

To be clear, our policy is not publish-by-default. Our current assessment is that the projects we're prioritizing do not pose a significant risk of capabilities externalities. We will continue to make these decisions on a per-project basis.

[-]Richard_Ngo2y60

FWIW for this sort of research I support a strong prior in favor of publishing.

[-]Algon2y80

phase transitions organize learning and that detecting, locating, and understanding these transitions

Only the third ability sounds like it could be useful for alignment. The others seem like they'd be firing all the time, and if you only had that and, for instance, the ability to understand the semantics of a single neuron, then I don't think that gets you very far. Or am I missing something obvious?

[-]Daniel Murfet2y205

I think it is too early to know how many phase transitions there are in e.g. the training of a large language model. If there are many, it seems likely to me that they fall along a spectrum of "scale" and that it will be easier to find the more significant ones than the less significant ones (e.g. we discover transitions like the onset of in-context learning first, because they dramatically change how the whole network computes).

As evidence for that view, I would put forward the fact that putting features into superposition is known to be a phase transition in toy models (based on the original post by Elhage et al and also our work in Chen et al) and therefore seems likely to be a phase transition in larger models as well. That gives an example of phase transitions at the "small" end of the scale. At the "big" end of the scale, the evidence in Olsson et al that induction heads and in-context learning appears in a phase transition seems convincing to me.

On general principles, understanding "small" phase transitions (where the scale is judged relative to the overall size of the system, e.g. number of parameters) is like probing a physical system at small length scales / high energy, and will require more sophisticated tools. So I expect that we'll start by gaining a good understanding of "big" phase transitions and then as the experimental methodology and theory improves, move down the spectrum towards smaller transitions.

On these grounds I don't expect us to be swamped by the smaller transitions, because they're just hard to see in the first place; the major open problem in my mind is how far we can get down the scale with reasonable amounts of compute. Maybe one way that SLT & developmental interpretability fails to be useful for alignment is if there is a large "gap" in the spectrum, where beyond the "big" phase transitions that are easy to see (and for which you may not need fancy new ideas) there is just a desert / lack of transitions, and all the transitions that matter for alignment are "small" enough that a lot of compute and/or very sophisticated ideas are necessary to study them. We'll see!

[-]Algon2y53

Thank you, that was helpful. If I'm getting this right, you think the "big" transitions plausibly correspond to important capability gains. So under that theory, "chain of thought" and "reflection" arised due to big phase transitions in GPT-3 and 4. I think it'd be great if researchers could, if not access training checkpoints of these models, then at least make bids for experiments to be performed on said models.

[-]Daniel Murfet2y72

That's what we're thinking, yeah.

[-]Review Bot2y*10

The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?