The Plan - 2024 Update
29Erik Jenner
8johnswentworth
26Lucius Bushnaq
25johnswentworth
11Lucius Bushnaq
10johnswentworth
9Lucius Bushnaq
13johnswentworth
10Lucius Bushnaq
11johnswentworth
14Neel Nanda
12kave
5Neel Nanda
9johnswentworth
7kave
17Garrett Baker
36Neel Nanda
6Nathan Helm-Burger
3Aprillion
1metawrong
15Lucius Bushnaq
8Nathan Helm-Burger
2Daniel Tan
10Jonas Hallgren
5johnswentworth
5Jonas Hallgren
7Aprillion
3Mateusz Bagiński
New Comment
theoretical progress has been considerably faster than expected, while crossing the theory-practice gap has been mildly slower than expected. (Note that “theory progressing faster than expected, practice slower” is a potential red flag for theory coming decoupled from reality, though in this case the difference from expectations is small enough that I’m not too worried. Yet.)
I don't know how much the difficulty of crossing the theory-practice gap has deviated from your expectations since then. But I would indeed be worried that a lot of the difficulty is going to be in getting any good results for deep learning, and that finding additional theoretical/conceptual results in other settings doesn't constitute much progress on that. (But kudos for apparently working on image generator nets again!)
As a sidenote, your update from 2 years ago also mentioned:
I tried to calculate “local” natural abstractions (in a certain sense) in a generative image net, and that worked quite well.
I assume that was some other type of experiment involving image generators? (and the notion of "working well" there isn't directly comparable to what you tried now?)
I don't know how much the difficulty of crossing the theory-practice gap has deviated from your expectations since then.
It's been pretty on-par.
But I would indeed be worried that a lot of the difficulty is going to be in getting any good results for deep learning, and that finding additional theoretical/conceptual results in other settings doesn't constitute much progress on that.
Amusingly, I tend to worry more about the opposite failure mode: findings on today's nets won't generalize to tomorrow's nets (even without another transformers-level paradigm shift), and therefore leveraging evidence from other places is the only way to do work which will actually be relevant.
(More accurately, I worry that the relevance or use-cases of findings on today's nets won't generalize to tomorrow's nets. Central example: if we go from a GPT-style LLM to a much bigger o1/o3-style model which is effectively simulating a whole society talking to each other, then the relationship between the tokens and the real-world effects of the system changes a lot. So even if work on the GPT-style models tells us something about the o1/o3-style models, its relevance is potentially very different.)
I assume that was some other type of experiment involving image generators? (and the notion of "working well" there isn't directly comparable to what you tried now?)
Yeah, that was on a little MNIST net. And the degree of success I saw in that earlier experiment was actually about on par with what we saw in our more recent experiments, our bar was just quite a lot higher this time around. This time we were aiming for things like e.g. "move one person's head" rather than "move any stuff in any natural way at all".
And for that purpose, we want to start as “close to the metal” as possible. We definitely do not want our lowest-level data to be symbolic strings, which are themselves already high-level representations far removed from the environment we’re trying to understand.
I don't understand the intuition here. In what sense are patterns of pixels “closer to the metal” than patterns of tokens? Why can't our low-level data be bigrams, trigrams, questions, and Monty Python references instead of edges, curves, digits and dog fur? What's the difference?
The thing I ultimately care about is patterns in our physical world, like trees or humans or painted rocks. I am interested in patterns in speech/text (like e.g. bigram distributions) basically only insofar as they tell something useful about patterns in the physical world. I am also interested in patterns in pixels only insofar as they tell something useful about patterns in the physical world. But it's a lot easier to go from "pattern in pixels" to "pattern in physical world" than from "pattern in tokens" to "pattern in physical world". (Excluding the trivial sense in which tokens are embedded in the physical world and therefore any pattern in tokens is a pattern in the physical world; that's not what we're talking about here.)
That's the sense in which pixels are "closer to the metal", and why I care about that property.
Does that make sense?
Excluding the trivial sense in which tokens are embedded in the physical world and therefore any pattern in tokens is a pattern in the physical world; that's not what we're talking about here.
I suppose my confusion might be related to this part. Why are tokens embedded in the physical world only in a "trivial" sense? I don't really see how the laws and heuristics of predicting internet text are in a different category from the laws of creating images of cars for the purposes we care about when doing interpretability.
I guess you could tell a story where looking into a network that does internet next-token prediction might show you both the network's high-level concepts created over high-level statistical patterns of tokens, and human high-level concepts as the low-level tokens and words themselves, and an interpretability researcher who is not thinking carefully might risk confusing themselves by mixing the two up. But while that story may sound sort of plausible when described in the abstract like that, it doesn't actually ring true to me. For the kind of work most people are doing in interpretability right now, I can't come up with a concrete instantiation of this abstract failure mode class that I'd actually be concerned about. So, at the moment, I'm not paying this much mind compared to other constraints when picking what models to look at.
Does the above sound like I'm at least arguing with your thesis, or am I guessing wrong on what class of failure modes you are even worried about?
Interpretability on an LLM might, for example, tell me a great deal about the statistical relationships between the word "human" and the word "dog" in various contexts. And the "trivial" sense in which this tells me about physical stuff is that the texts in question are embedded in the world - they're characters on my screen, for instance.
The problem is that I don't care that much about the characters on my screen in-and-of themselves. I mostly care about the characters on my screen insofar as they tell me about other things, like e.g. physical humans and dogs.
So, say I'm doing interpretability work on an LLM, and I find some statistical pattern between the word "human" and the word "dog". (Flag: this is oversimplified compared to actual interp.) What does that pattern tell me about physical humans and physical dogs, the things I actually care about? How does that pattern even relate to physical humans and physical dogs? Well shit, that's a whole very complicated question in its own right.
On the other hand, if I'm doing interp work on an image generator... I'm forced to start lower-level, so by the time I'm working with things like humans and dogs I've already understood a whole lot of stuff about the lower-level patterns which constitute humans and dogs (which is itself probably useful, that's exactly the sort of thing I want to learn about). Then I find some relationship between the human parts of the image and the dog parts of the image, and insofar as the generator was trained on real-world images, that much more directly tells me about physical humans and dogs and how they relate statistically (like e.g. where they're likely to be located relative to each other).
But why do we care more about statistical relationships between physical humans and dogs than statistical relationships between the word "human" and the word "dog" as characters on your screen? For most of what interp is currently trying to do, it seems to me that the underlying domain the network learns to model doesn't matter that much. I wouldn't want to make up some completely fake domain from scratch, because the data distribution of my fake domain might qualitatively differ from the sorts of domains a serious AI would need to model. And then maybe the network we get works genuinely differently than networks that model real domains, so our research results don't transfer. But image generation and internet token prediction both seem very entangled with the structure of the universe, so I'd expect them both to have the right sort of high level structure and yield results that mostly transfer.
On the other hand, if I'm doing interp work on an image generator... I'm forced to start lower-level, so by the time I'm working with things like humans and dogs I've already understood a whole lot of stuff about the lower-level patterns which constitute humans and dogs
For this specific purpose, I agree with you that language models seem less suitable. And if this is what you're trying to tackle directly at the moment I can see why you would want to use domains like image generation and fluid simulation for that, rather than internet text prediction. But I think there's good attack angles on important problems in interp that don't route through investigating this sort of question as one of the first steps.
But why do we care more about statistical relationships between physical humans and dogs than statistical relationships between the word "human" and the word "dog" as characters on your screen?
An overly-cute but not completely wrong answer: because I care about whether AI kills all the physical humans, not whether something somewhere writes the string "kill all the humans". My terminal-ish values are mostly over the physical stuff.
I think the point you're trying to make is roughly "well, it's all pretty entangled with the physical stuff anyway, so why favor one medium or another? Instrumentally, either suffices.". And the point I'm trying to make in response is "it matters a lot how complicated the relationship is between the medium and the physical stuff, because terminally it's the physical we care about, so instrumentally stuff that's more simply related to the physical stuff is a lot more useful to understand.".
An overly-cute but not completely wrong answer: because I care about whether AI kills all the physical humans, not whether something somewhere writes the string "kill all the humans". My terminal-ish values are mostly over the physical stuff.
I don't understand this argument. Interpretability is not currently trying to look at AIs to determine whether they will kill us. That's way too advanced for where we're at. We're more at the stage of asking questions like "Is the learning coefficient of a network composed of independent superposed circuits equal to the learning coefficients of the individual circuits summed, or greater?"
The laws of why and when neural networks learn to be modular, why and when they learn to do activation error-correction, what the locality of updating algorithms prevents them from learning, how they do inductive inference in-context, how their low-level algorithms correspond to something we would recognise as cognition, etc. are presumably fairly general and look more or less the same whether the network is trained on a domain that very directly relates to physical stuff or not.
Interpretability is not currently trying to look at AIs to determine whether they will kill us. That's way too advanced for where we're at.
Right, and that's a problem. There's this big qualitative gap between the kinds of questions interp is even trying to address today, and the kinds of questions it needs to address. It's the gap between talking about stuff inside the net, and talking about stuff in the environment (which the stuff inside the net represents).
And I think the focus on LLMs is largely to blame for that gap seeming "way too advanced for where we're at". I expect it's much easier to cross if we focus on image models instead.
(And to be clear, even after crossing the internals/environment gap, there will still be a long ways to go before we're ready to ask about e.g. whether an AI will kill us. But the internals/environment gap is the main qualitative barrier I know of; after that it should be more a matter of iteration and ordinary science.)
Strong +1, that argument didn't make sense to me. Images are a fucking mess - they're a grid of RGB pixels, of a 3D environment (interpreted through the lens of a camera) from a specific angle. Text is so clean and pretty in comparison, and has much richer meaning, and has a much more natural mapping to concepts we understand
That sounds less messy than the path from 3D physical world to tokens (and less (edit: I meant more here!) messy than the path from human concepts to tokens)
1Sure, but I think that human cognition tends to operate at a level of abstract above the configuration of atoms in a 3D environment. Like "that is a chair" is a useful way to reason about an environment. Whilethat "that is a configuration of pixels that corresponds to a chair when projected at a certain angle in certain lighting conditions" must first be converted to "that is a chair" before anything useful can be done. Text just has a lot of useful preprocessing applied already and is far more compressed
The preprocessing itself is one of the main important things we need to understand (I would even argue it's the main important thing), if our interpretability methods are ever going to tell us about how the stuff-inside-the-net relates to the stuff-in-the-environment (which is what we actually care about).
I'm not sure I understand what you're driving at, but as far as I do, here's a response: I have lots of concepts and abstractions over the physical world (like chair). I don't have many concepts or abstractions over strings of language, apart from as factored through the physical world. (I have some, like register or language, but they don't actually feel that "final" as concepts).
As far as factoring my predictions of language through the physical world, a lot of the simplest and most robust concepts I have are just nouns, so they're already represented by tokenisation machinery, and I can't do interesting interp to pick them out.
And yes, I do think that interp work today should mostly focus on image nets for the same reasons we focus on image nets. The field’s current focus on LLMs is a mistake
A note that word on the street in mech-interp land is that often you get more signal & a greater number of techniques work on bigger & smarter language models over smaller & dumber possibly-not-language-models. Presumably due to smarter & complex models having more structured representations.
1
1Fwiw, this is not at all obvious to me, and I would weakly bet that larger models are harder to interpret (even beyond there just being more capabilities to study)
Hmm. I think there's something about this that rings true and yet...
Ok, so what if there were a set of cliff faces that had the property that climbing the bigger ones was more important and also that climbing tools worked better on them. Yet, despite the tools working better on the large cliffs, the smaller cliffs were easier to climb (both because the routes were shorter, and because the routes were less technical). Seems like if your goal is to design climbing equipment that will be helpful on large cliff faces, you should test the climbing equipment on large cliff faces, even if that means you won't have the satisfaction of completing any of your testing climbs.
What if you tried to figure out a way to understand the "canonical cliffness" and design a new line of equipment that could be tailored to fit any "slope"... Which cliff would you test first? 🤔
So you would expect Claude Opus 3 to be harder to interpret than Claude Sonnet 3.5 ?
My intuition is that larger models of the same capability would exhibit less super-position and thus be easier to interpret?
Yeah, I think this is a relevant point. Maybe for John and David's project the relevant point would be to try their ideas on absurdly oversized image models. Sometimes scale just makes things less muddled.
Might run into funding limitations. I wish there was more sources of large scale compute available to research like this.
I think interp 'works best' within a capability range, with both an upper and lower bound. (Note; this is a personal take that does not necessarily reflect the consensus in the field)
Below a certain capability threshold, it's difficult to interpret models, because those models are so primitive as to not really be able to think like humans. Therefore your usual intuitions about how models work break down, and also it's not clear if the insight you get from interpreting the model will generalise to larger models. Rough vibe; this means anything less capable than GPT2
With high capabilities, things also get more difficult. Both for mundane reasons (it takes more time and compute to get results, you need better infra to run larger models, SAEs need to get proportionately larger etc) as well as fundamental ones (e.g. the number of almost-orthogonal directions in N-dimensional space is exponential in N. So wider models can learn exponentially more features, and these features may be increasingly complex / fine-grained.)
In your MATS training program from two years ago, you talked about farming bits of information from real world examples before doing anything else as a fast way to get feedback. You then extended this to say that this is quicker than doing it with something like running experiments.
My question is then why you haven't engaged your natural latentes or what in my head I think of as a "boundary formulation through a functional" with fields such as artificial life or computational biology where these are core questions to answer?
Trying to solve image generation or trying to solve something like fluid mechanics simulations seem a bit like doing the experiment before trying to integrate it with the theory in that field? Wouldn't it make more sense to try to engage in a deeper way with the existing agent foundations theory in the real world like Michael Levin's Morphogenesis stuff? Or something like an overview of Artificial Life?
Yes as you say real world feedback loops and working on real world problems, I fully agree but are you sure that you're done with the problem space exploration? Like these fields already have a bunch of bits on crossing the theory practice gap. You're trying to cross it by applying the theory in practice yet if that's the hardest part wouldn't it make sense to sample from a place that already has done that?
If I'm wrong here, I should probably change my approach so I appreciate any insight you might have.
Great question.
First, I do think of systems biology as one of the main fields where the tools we're developing should apply.
But I would not say that fields like artificial life or computational biology have done much to cross the theory-practice gap, mainly because they have little theory at all. Artificial life, at least what I've seen of it, is mostly just running various cool simulations without any generalizable theory at all. When there is "theory", it's usually e.g. someone vibing about the free energy principle, but it's just vibing without any gears behind it. Levin's work is very cool, but doesn't seem to involve any theory beyond kinda vibing about communication and coordination. (Though of course all this could just be my own ignorance talking, please do let me know if there's some substantive theory I haven't heard of.)
Uri Alon's book is probably the best example I know of actual theory in biology, but most of the specifics are too narrow for our purposes. Some of it is a useful example to keep in mind.
Let me drop some examples of "theory" or at least useful bits of information that I find interesting beyond the morphogenesis and free energy principle vibing. I agree with you that basic form of FEP is just another formalization of bayesian network passing formalised through KL-divergence and whilst interesting it doesn't say that much about foundations. For Artificial Life, it is more a vibe check from having talked to people in the space, it seems to me they've got a bunch of thoughts about it but it seems like they've got some academic capture so it might be useful to at least talk to the researchers there about your work?
Like a randomly insultingly simple suggestion: Do a quick literature review through elicit in ActInf and Computational Biology for your open questions and see if there's links, if there are send those people a quick message. I think a bunch of the theory is in people's heads and if you nerdsnipe them they're usually happy to give you the time of day.
Here's some stuff that I think is theoretically cool as a quick sampler:
For Levin's work:
- In the link I posted above he talks about morphogenesis, the thing I find the most interesting there from an agent foundations and information processing perspective is the anti-fragility of systems with respect to information loss (similar to some of the stuff in Uri's work if I've understood that correctly.) There are lots of variations of underlying genetics yet similar structures can be decoded through similar algorithms and it just shows a huge resillience there. It seems you probably know this from Uri's work already
Active Inference stuff:
- Physics as information processing: https://youtu.be/RpOrRw4EhTo
- The reason why I find this very interesting is that it seems to me to be saying something fundamental about information processing systems from a limited observer perspective.
- I haven't gotten through the entire series yet but it is like a derivation of hierarchical agency or at least why a controller is needed from first principles.
- I think this ACS post explains it better than I do below but here's my attempt at it:
- I'm trying to find the stuff I've seen on <<Boundaries>> within Active Inference yet it is spread out and not really centered. There's this very interesting perspective of there only being model and modelled and that talking about agent foundations is a bit like taking the modeller as the foundational perspective whilst that is a model in itself. Some kind of computational intractability claims together with the above video series gets you to this place where we have a system of hierarchical agents and controllers in a system with each other. I have a hard time explaining it but it is like it points towards a fundamental symmetry perspective between an agent and it's environment.
Other videos from Levin's channel:
- Agency at the very bottom - some category theory mathy stuff on agents and their fundamental properties: https://youtu.be/1tT0pFAE36c
- The Collective Intelligence of Morphogenesis - if I remember correctly it goes through some theories around cognition of cells, there's stuff about memory, cognitive lightcones etc. I at least found it interesting: https://youtu.be/JAQFO4g7UY8
(I've got that book from URI on my reading list btw, reminded me of this book on Categorical systems theory, might be interesting: http://davidjaz.com/Papers/DynamicalBook.pdf)
Focus On Image Generators
How about audio? Is the speech-to-text domain as "close to the metal" as possible to deserve focus too or did people hit roadblocks that made image generators more attractive? If the latter, where can I read about the lessons learned, please?
I know almost nothing about audio ML, but I would expect one big inconvenience when doing audio-NN-interp to be that a lot of complexity in sound is difficult to represent visually. Images and text (/token strings) don't have this problem.
Curated and popular this week