Introduction Note: These are results which have been in drafts for a year, see discussion about how we have moved on to thinking about these things. Our team at AI Safety Camp has been working on a project to model the trajectories of language model outputs. We're interested in predicting...
tl;dr: We compute the evolution of the local learning coefficient (LLC), a proxy for model complexity, for an algorithmic transformer. The LLC decreases as the model learns more structured solutions, such as head specialization. This post is structured in three main parts, (1) a summary, giving an overview of the...
Note: These are results which have been in drafts for a year, see discussion about how we have moved on to thinking about these things.
Our team at AI Safety Camp has been working on a project to model the trajectories of language model outputs. We're interested in predicting not just the next token, but the broader path an LLM's generation might take. This post summarizes our key experiments and findings so far.
TL;DR: We tried some simple probing experiments to identify "text type." Results seem promising, but this approach is likely not the best way forward for our goals.
tl;dr: We compute the evolution of the local learning coefficient (LLC), a proxy for model complexity, for an algorithmic transformer. The LLC decreases as the model learns more structured solutions, such as head specialization.
This post is structured in three main parts, (1) a summary, giving an overview of the main results, (2) the Fine Print, that delves into various cross-checks and details and (3) Discussion and Conclusions.
In this work we study the development of simple algorithmic transformers, which are transformers that learn to perform algorithmic tasks. A major advantage of this setup is that we can control several (hyper)parameters, such as the complexity of the training data and... (read 3035 more words →)
Thank you for your answer! You clarified my confusion!
I would be interested to know more about your concept of (inner) optimization in its full complexity and nuances. I would really appreciate it if you could point me to any previous writings, regarding this.
My previous reads on this topic include this post from Yudkowsky and this post from Flint where (to the best of my understanding) an optimizing system evolves according to some preference ordering that has a low probability of occurring spontaneously. I find their definitions to be a bit more general than the one you are referring to here (please correct me if I am wrong).
I am curious about the above because I am currently working on a project related to this topic. I am interested in formalizing some concepts regarding optimizers and their potential evolution towards agentic structure in some limit with rigorous math.
Hi, I enjoyed reading the post, as it clarified some thoughts that I was having regarding the topic. Could you please briefly elaborate on this part of your post:
..., then for the right operationalization properties about optimization are properties that belong to the complexity class of the system it’s internal to.
I understand the first part of that sentence and the rough idea of what you're getting at, but not exactly what you mean in this particular section.
This was a very useful and timely post for me. I was on the lookout for a tool to use to evaluate the quality of my predictions, and the Brier score was a concept I didn't know before now. I will try to incorporate this in my daily routine. Thank you!
One additional variable one could also keep track of is something like a "correction" factor. If I predict a given task will take me 1 week to complete, and it ends up taking 2 weeks instead, then next time I am faced with a task of a similar nature I should remember that last time I was wrong by a factor 2.... (read more)