These are notes from my research undertaken as part of the Iliad Fellowship under the mentorship of Dmitry Vaintrob.

I sat down and thought silently for 2h about what needs to happen in order for us to understand neural networks. Summary below:

• We must understand structure in terms of the neural network itself, not by trying to project this structure onto things we can interpret. If it is interpretable, then the structure should tell us so - spectral gaps, isolated/sparse circuits, etc. A better understanding of NN structure doesn't have to mean better interpretability - they're correlated only to the extent that the actual structure is made up of interpretable units; if NNs are made up of circuits, we should see them, if not, we should not see them. NNs obviously have sufficient structure that we can understand them better - build a mental model robust enough to predict the median ICML 'neural networks sure are weird arent they' paper. Module criticality and 'transformers as painters' are my measuring-stick for this kind of mental model building.

• We want meso- and macro-ontologies (e.g., circuits, personae, 'capabilities'). Strong-opinion-weakly-held: if it were possible to construct these ontologies well without starting from the micro-scale, we would have succeeded already. We need robust handles for thinking about behaviour on the micro-scale - we will have to abstract away from bare-parameter/activation reasoning bit by bit. It is best if we can justify these ontic units as being somehow 'isolated' or 'discrete'; the archetype I have in mind is the BBP spike transition - it is sufficiently robust that one can plausibly define signal deteciton/recovery as in reference to BBP, and this definition will serve well in some regime perturbed around BBP. If we are careful when we construct these ontic units, they will come labeled with a asymptotic regime of validity. This tells us where to look for the next level of abstraction. For example, if BBP gives us a definition of "detected, localized signal", the breakdown of BBP in the many-signal regime tells us where to look; hopefully, this breakdown, construed as a phase transition, gives us another ontic unit to work with, and lets us proceed. (my pet theory for the last three days, perhaps this breakdown of localized signals is precisely the 'delocalization' of knowledge inherent in going from memorized patterns to a connected world model).

• Noise and training dynamics seem fundamental, not incidental. I hypothesise that, at equal loss, an NN trained under full-batch gradient-flow will be essentially useless at extrapolating OOD (i.e., would suck at being finetuned to a downstream task -- not merely generalizing to unseen data from the same distribution). Two perspectives clash here: one is data-essentialist, that an NN has finite capacity, and there's a particular optimal way to represent that data given fixed capacity, and that way happens to be sparse/hierarchical/factored representations, perhaps on account of some sort of 'simplicity' prior - the 'golden path hypothesis' is in this genre. Second, there's a structure-essentialist view, wherein the dynamics of training are the main thing which determine the nature of representations learned, with a secondary role for the data. I lean towards the structure-essentialist view - the neurons just want to learn man. My working mental model is much closer 'neural Darwinism', natural selection acting on competing species (competing 'circuits') - the abiotic environment (the data) matters, but the biotic environment is shaped 'more' by its own internal logic of competition and ecology. For example, perhaps a globally-shared 'world-model' is selected for not by compression, but by being so universally-used in computations that even mediocre performance would make any gradient more harmful than useful.

• I also expect a notion of plasticity and critical-periods will be important - nature didn't implement these in humans for fun; literature seems to point to early/middle/late layers as corresponding to "feature-extraction / generic computation / task-specific", with the early and late layers deplastifying early in training (I'm unsure of this interpretation). It seems like some kind of plasticity control is important, and that we just mostly never realized this on account of NNs having figured out how to half-assedly implement this on their own - 'divine benevolence' and all that. Possible relation here to EGOP/AGOP/NFA, but I don't know.

• A huge percentage of brain machinery seems to be dedicated towards managing stacks of different regulatory loops, and it is probably the case that enhancing intelligence, cognition, etc. corresponds more to precisely tuning an interlocking system of regulatory loops than to having any difference in the basic machinery of neural function. I think there are convergent reasons to expect this to be true of NNs (even aside form the critical-period literature) - probably most of training is best thought of as 'tuning'. This also makes model addition/Polyak averaging make a bit more sense - if is all changes in regulators and not in basic machinery, 'averaging the regulator settings' kinda feels like a thing that could work, in a way that 'average these two ALU implementations' doesn't. I don't think "everything is regulators" is more useful than "everything is circuits" - we need a middle-ground picture that I don't have yet.

• (Naive) weight-based analysis should be extremely hard, probably usually impossible. the natural units of analysis may instead be weights-in-context - pairs (contexts ~ 'prompts' or something). If one were to look at a single neuron of the brain averaged over a day, it would be hopeless - it is doing fundamentally different things during sleep than during an algebra class. This requires some principled way of saying what contexts are 'naturally' grouped together. My best guess at how to understand this right now is that the eNTK is a natural way to ask the network how similarly it will behave for the two - at the very least, we need some modified version of this if we want to analyze context-sensitive behaviour. Possibly something more like the parameter-space version of this: , where is some subset of the data-distribution which is a context in which one can expect neurons to be performing a 'single function'. It's not clear what semantic level this distinction is made at - if this should be more like 'doing algebra vs doing calculus' or more like 'thinking the letter a' vs 'thinking the letter c' - hence my trying to emphasise that we need some 'natural' way to carve out based on the network - and hence perhaps something like clustering based on . I think dynamic analyses of weights over training are more likely to yield useful information.

• I'm less sure about this, but I think we should only rarely think in terms of 'moment bias' - I wrote down an EFT formalism where increasing data let one 'resolve' progressively higher moments of a data distribution - a la 'sliding down the stairs' - then I stared at this and realized there's absolutely no way one could use this to ever find a 'cat' concept without making assumptions tantamount to solving the problem. Asserting that high level semantic concepts should live in higher moments of the data-distribution is vacuously true - iirc M'th order moments take exp(M) samples to estimate. Insofar as NNs seem to have some universality in the things they learn, I'm fairly sure one could not predict this universality from the data-distribution/geometry, short of training NNs on it - I think it comes down much more to "inductive bias". Obviously, caveats to this for fine-tuning a trained neural network.

What specific next-steps are implied by this?

Things I need to learn/read more: NNMFT, plasticity and critical-periods in biology and ML, Liquid-state machine model of brain. Log-linearity and supposed approximate equivalence between fine-tunes/system-prompts/steering-vectors, etc.

Think more about: Quasispecies models as ways to operationalize 'units of selection'; different sources of 'noise', when they are equivalent, and when should this have anything to do with 'linearity' in transformers? Relationship between evolved noise-robustness and plasticity?; BBP transitions and Saxe-dynamics.

I think the most promising things to think about are phase transitions of every shape and flavour, as these are the most actionable, and the things for which I can most readily evaluate validity. This may entail learning MFT. The 'plasticity' / 'critical period' idea features more heavily in my model than I had realized, and I should try to understand this better too. My heart says these must be deeply related to 'structured noise' (e.g., one module becoming robust to 'noise' from another == loss of plasticity?). Hand-wavily, 'critical periods' seem like a good thing to mine for my beloved phase-transitions.

The Immune System as Anti-Optimizer

We have a short list of systems we like to call "optimizers" — the market, natural selection, human design, superintelligence. I think we ought to hold the immune system in comparable regard; I'm essentially ignorant of immunobiology beyond a few YouTube videos (perhaps a really fantastic LW sequence exists of which I am unaware), but here's why I am thinking this.

The immune system is the archetypal anti-optimizer: it defends a big multicellular organism from rapidly evolving microbiota. The key asymmetry:

1. Specified once by the genome. It cannot rewrite its own source code between generations. Its adversaries evolve orders of magnitude faster.
2. Resource asymmetry as counterbalance. Whole organs are devoted to the adaptive immune response. A microbe is one cell; the immune system is a civilization.
3. This extends to cancer. The immune system typically out-adapts malignant cells despite selection acting far more rapidly on them. Immunosurveillance fails not because it is weak but because cancers occasionally evolve to exploit its specific tolerance mechanisms.

In short: the immune system embodies enough amortized optimization power to defend against online adversarial attacks by natural selection, because these attacks are constrained by the comparative simplicity of the attackers. One optimizer constrains another, faster and more adaptive optimizer, by having more resources.

What makes this especially interesting is that the immune system has no discernible volition. It is complex — probably far more so than I appreciate — but intuitively much more like a thermostat than a scheming eldritch god. It optimizes powerfully, within bounds that feel legible and non-agential.

I will not be so crass as to say "big if true for alignment", but you are permitted to infer this if it please you. I just think it's neat. Consider the mere phrase "semiotic immune system" (from, if I recall correctly, Charles Stross's Accelerando) — suggests a lot at once, eh?

I asked Claude to prepare the following tutorial - which I have not yet read (longa est vita, si uti bene scias...) - developing this theme: https://claude.ai/share/67bb8de3-b73c-4a21-916b-70affba0da43

*Written with slight corrections for conciseness from Opus4.6. Ironically, the em-dashes are mine.

An image I have found inspiring as a researcher in the age of LLMs:

A samurai orients their entire life towards preparing for a few decisive moments of combat. Awakening at dawn for hours of grueling daily combat training, certainly.

But also an intensive study of calligraphy, classical literature - why?

In that decisive moment of combat, whose movement is the more fluid, who by sense of righteousness is the more unflinching, they are the victors. The fluid breathing of the calligraphy brush, the moral fibre wrought by the classics - in that moment a lifetime of study is vindicated or revealed worthless.

Why study calculus if Claude already knows a hundred or a thousand times more math than I will on my deathbed? Not because I anticipate finding Claude has made a mistake in integrating by parts, but so that my mind and intuition are sharp - so that I know exactly what to ask, what to look for, and exactly what a good answer should look like - to know what kinds of analysis can reasonably yield what kinds of conclusions - to know on a thirty second skim whether Fable 5's argument is natural or sidestepping the whole problem.

Perhaps this is just 'taste', but a large comparative advantage we have now over LLMs is choice in what we think about. LLMs are, if not constrained, then 'conditioned', by their starting prompt. Asked to solve a problem in economics, it will likely reason about economics and not biology - or if it does unprompted, any such connection will often-as-not be superficial obsequy, Fable 5 trying very hard to convince you that it is exactly ten IQ points above you. If you want some ex-ante-crazy connection between economics and biology, you have to know what you're looking for in advance.

Object-level knowledge can be backfilled easily enough - Sol is excellent at writing bespoke tutorials. I think it is highly valuable now to read not for knowledge, but to see and internalize powerful thinking in action - more power to be found reading 'the wealth of nations' or 'on the origin of species' than an economics or biology textbook with a further century's developments and pedagogy.

So the greater part of our life now as researchers is be given over to generating more moments of true choice - choice to think, choice what to think about. It's hard to imagine we will long have truck with professionally-checking-Fable outputs. But I expect the ability to think clearly, grandly, freely, will have at least a longer shelf life. And probably it's also good for your soul.

In this spirit, I'll be reading 'the wealth of nations' this week. I invite anyone interested to do likewise with a 'classic' of their choosing.

Inspired by conversation with @richard_ngo and @Ashe Vazquez Nuñez, to each of whom my thanks.

Remarks on the Slow Boring post about "Kritiks" in debate, copied here for a friend who wanted to reference them, and lightly edited to fit the shortform format:

In the format of debate I do ("British Parliamentary" (BP), also my favorite, having also done "Public Forum"), this sort of thing is sorta-kinda explicitly not allowed. Ironically, I think BP is somewhere where this would be most appropriate - because you don't have the 'motion' until 15 minutes prior to start, focus on ground-facts is low (only relatively, compared to other research-heavy styles); you spend a lot of time arguing about "moral weighing-criteria" and "framing" and things like this.

We do have a pretty clear line where you can, say, go full-Marxist in saying some particular impact is the most important - you have to defend this framing, not just assert "Marxism says X" - but you definitely can't tell the motion to go fuck itself. (A late-debate reframing is called a "squirrel" and only allowed if the opening-half was batshit insane with the framing).

Specifically, you pretty much can get away with saying "zoos are anthropocentric and that's bad because...", but then you have to justify that and "weigh it out" against the other cases people bring. That is, you can win with a case that would be Kritik-adjacent (i.e. radical but you still can't challenge the motion), but you then have to argue why that's the relevant framing of the debate, why it's the most important thing, etc.

I only know EU debate for BP, so it's possible its just that US debate as a whole is weird, but I don't think so since I've seen e.g. Princeton compete. In the slow-boring article they note that its worst in "Policy" debate which sounds reasonable since that's always been the most Goodharted format (I've heard of classes where they teach you to speak at 350 WPM). Also worth noting that I never debated at an especially high level, and only in the Netherlands, but I've watched a few world-finals debates and some online workshops and nothing Kritik-flavored has come up.

I find BP is actually one of the best antidotes to Twitter hot-takes, in part I'd guess because framing etc. is an integral part of the debate anyways, so what would be Kritik "flip-the-table" BS in Policy can be (is required to be) much more productive in BP. Perhaps this is the structure of the format, but it may also just be the community.