SLT is a thermodynamic theory of Bayesian learning, but not the thermodynamic theory of Bayesian learning

SLT provides a rigorous mathematical framework for Bayesian learning in a certain regime, but I argue its practical applicability to real neural networks (even in a Bayesian learning/ high-level modeling context) is limited by finite-size effects and high-dimensionality. The valuable empirical work in this space is better understood as 'thermodynamic interpretations of ML' rather than validations of SLT proper

I've been having lots of conversations with people about SLT. I like SLT as a model for Bayesian learning a lot. At the same time I think that the assumptions of SLT are a model of the reality of learning (including Bayesian learning), in the same way that variants of the harmonic oscillator are a model of a physical system. There are some results that show that every physical system under some assumptions is a harmonic oscillator in a limit, but this limit doesn't always hold.

I think the place where I am bothered by SLT rhetoric is where interesting experiments get done and interesting thermodynamic parameters get found, but instead of viewing this as results in "thermodynamic interpretation of ML", there is an incorrect assumption that the observed phenomena are explained (fully or up to a controllable error) by an expansion around a singularity of a singular learning system.

I'm planning on writing more about this, but I'll try to write out the key arguments to let people look at them and to see if I'm getting something wrong.

Essentially, I think it would be good to coordinate on a language for talking about Bayesian learning that doesn't overindex on the singular learning limit, and instead uses physics terms (free energy, susceptibility, heat capacity) for the actual observed invariants. I think that in many ways SLT is moving in this direction, and I'm excited about what they have done.

First the things I agree with:

1. We are doing Bayesian learning (or a mild variant of "Boltzmann learning", which allows rescaling the "size" of Bayesian updates by a scalar factor)
2. If we fix a neural network as a statistical system (its data distribution, architecture, loss) and take the number of data samples, n, to infinity, there is a limit where the singular learning prediction is true.

So there is always a regime where n is sufficiently large that SLT gives an exact answer to Bayesian learning. Why am I objecting to a (mathematically true) fact?

Essentially, the two key issues are that

• Neural networks are high-dimensional systems, and this makes errors in SLT approximation potentially very large (even exponentially large in something like number of parameters) at finite data.
• In order to get the exact Watanabe approximation to hold, we need to assume that for infinite data the singularity is exact. Thus if we have some predicted singular loss $L_{sing}\left(\theta \right)$ depending on a weight choice $\theta$, then we can (at least wrt the guarantees given by SLT theory) use the singularities of $L_{sing}$ to get a prediction for the asymptotic only if $L_{sing}\left(\theta \right)=L\left(\theta \right)$ exactly in the infinite-data limit. If there is any small approximation issue that relates to the architecture / data rather than to the number of samples n (e.g. the difference between the discrete Fourier Transform and the continuous FT on the circle for modular addition, approximations of a polynomial by sigmoids in modular addition and other contexts, even bit complexity issues, etc.), then we can't rely on the SLT prediction - at least theoretically. Here one might a priori expect that the singular information from an approximate model $L_{sing}$ of the loss would be predictive of the singularity of the true loss $L$, but in fact this is false: the property of being nontrivially singular at all is a measure-zero property of a (loss) function, so if there are any modeling assumptions or possible sources of error or noise, we expect the "true infinite limit" Watanabe prediction to be the same one as for a "nonsingular" loss (i.e., positive-definite Hessian at the limit). Sometimes there are small corrections (often called "gauge") from the architecture, but these are small compared to the empirical singularity-like effects one observes from thermodynamic measurements.

I want to especially harp on the fact that it takes at worst-case exponential number of samples, and therefore exponential loss precision of the difference $L_{sing}-L$ (more precisely, exp of some power of the number of parameters) to mathematically guarantee that the SLT prediction is correct. Thus an argument shaped like "SLT is mathematically correct" is, for realistic models, true but boring.

However my pair of counterarguments isn't strong by itself. There are lots of cases in the context of mathematical modeling of complex systems where theory says that we might (in worst-case situations) need to wait exponentially long/ get exponential errors/ etc., but in practice finite time suffices. And I think that there are interesting systems where the modeling assumptions above are correct. It's just that you shouldn't assume this by default.

Thus it is meaningful to ask the following question:

• in what learning problems, what regimes, and at what scales can Bayesian learning be described by an SLT prediction? In other words, when can we find a function $L_{sing}\left(\theta \right)$ that is a "sufficiently good" approximation of the true infinite-sample loss $L\left(\theta \right)$ and whose singularities "meaningfully describe the thermodynamic behavior" of Bayesian learning.

I think this is an interesting question. One might object to it by saying that this is hard to measure, since Bayesian learning is extremely hard to study in "realistic" cases. However I'd argue that we have enough examples to start studying this. And again, my sense here is that SLT prediction fail already to first order (i.e., for predicting e.g. the correct asymptotic for $\hat{\lambda }$ up to O(1) rescaling), but they fail in an interesting way.

SLT failures in Bayesian learning.

I'll write more about this later. But let me give two examples where I think this is the case.

Grokking modular addition (with MSE loss)

In this paper, grokking in (MSE) modular addition is analyzed using a technique called "mean field theory" (this is an extension of more classical work on things like the neural tangent kernel, which drops the highly restrictive assumption that the system is in a "lazy learning" regime: i.e., that the solution is a small perturbation away from a "trivial vacuum"). This is a paper about Bayesian learning and it gives exact predictions that are confirmed by experiment (one can empirically "approximate" Bayesian learning by something called Langeving SGD). The prediction in particular implies an expansion for the SLT term $\hat{\lambda }$ (the "heat capacity") and for the free energy (roughly, the stat-phys version of what is called "basin volume" in SLT). It turns out that the free energy in the relevant regime is actually explicitly not controlled by any basin around a singularity, but is rather given to first order by a high-dimensionality phenomenon (similar to critical phenomena in thermodynamics). Specifically, the first-order approximation of the free energy is compatible with the system having some fixed number of degrees of freedom per neuron (here: "row of the input weight matrix"). More precisely this paper predicts (and experiment verifies) that to first order, the NN learns to randomly sample each "weight" row from some fixed distribution. (The resulting NN has approximately correct outputs by the central limit theorem; but if one is interested in a more realistic context with a small number of neurons compared to p, higher terms in the mean-field expansion let you predict regimes with higher and higher accuracy; the leading free-energy term will still be the same). This terms is, importantly, not even a rational number (as predicted by SLT), but some numerical integral associated with the neuron distribution (not dominated by any one value or any small set of moments, except in certain limiting regimes). Note that (similar to SLT) the mean-field prediction holds, in a certain idealization, for any "input number" n so long as n is less than some exponential in the number of neurons.

LoRA for empirical "approximately singular" matrices from ML

For our second example, we look at a two-layer "deep(ish) linear network" that tries to model $W_{in}{W}_{out}=M$, (here we replace the ReLU with a trivial nonlinearity, and take W_{in}, W_{out} to be the trainable parameters and M to be the fixed "target"), there is an exact formula for the Bayesian learning prediction of this network in terms of the singular values of M ("singular" is a good term here :). The Bayesian learning dynamics are entirely controlled by the "entropy" or "volume" function $Vol(s_1,s_2,..,s_d),$ which is roughly the volume of the "space of pairs of matrices $W_{in}$, $W_{out}$ of bounded size whose product is M" (the "bounded size" is a secret parameter here, that is related to the "number of training points" parameter n in SLT).

If the (min of) input/output width is larger than the "hidden layer" width, the function $Vol(s_1,...,s_d)$ has singularities and SLT in this case makes a nontrivial prediction: namely, that if some fraction of the singular values $s_k=s_{k+1}=...=s_d=0$ is zero (note we can assume WLOG that this is the last $d-k$ values, since singular values are un-ordered), then the function Vol has a singularity. SLT now implies that if the "target" matrix M has lower-than-full rank (i.e. some of the s_i are exactly 0) then in the high-data limit $n\to \infty$ we have an exact asymptotic on the free energy function F(n) (as a function of dataset size).

Here in order for the SLT asymptotic to be true, we want to assume that the first k singular vector of M are exactly zero and the data size parameter n is much larger than the inverse (square) of the smallest nonzero singular values (this is called the "spectral gap"). If we want some kind of exact or asymptotic formula for the free energy that makes valid predictions for n without the singular gap assumptions, or without the assumption that the singular values are exactly 0, we can simply plug the (easily computable) "true" singular values $s_1,\dots ,s_d$ into the (known) exact formula -- or a known expansion whose validity (/error bound) we can mathematically verify -- and see what we get.

Ultimately this depends on what deep linear models one would want to model "in practice" in the context of ML. This might at first seem like a silly problem: ML is explicity nonlinear and any deep linear model is just a toy.

However, this is not entirely true: in some cases for a realistic neural net, one is interested in doing "LoRA" decomposition. LoRA means different things in different contexts, but one standard context is decomposing a weight matrix $W=W^{\ell }$ that appears in some layer $\ell$ of the model into a product of two matrices $W=W_L{W}_R,$ with $W_L$ and $W_R$ having lower rank. In general in such a context, one would then train all the weights of the model together (and again, typically LORA is used in a slightly modified context where a low-rank product $W_L{W}_R$ somehow supplements rather than replacing an intermediate layer). Nevertheless, we can alway model "some part" of LoRA learning as learning an approximate factorization of a matrix $W$ into a product $W_L{W}_R$ (this corresponds to "freezing" all weight layers except those that defined $W$, and modeling the loss as an approximation loss on $W$ -- sketchy, but I'd guess a reasonable "directional" guess for asymptotics in a suitable regime).

Once we have reduced to this problem, we can now write down the exact Bayesian prediction and the SLT prediction for various values of "dataset size" n. Both predictions depend only on the singular values $s_1,...,s_d$ of the matrix $W$ (this can be seen using symmetry). Now empirically, it turns out that one can approximate the "bulk" of these singular values relatively well up to a constant by a power law $s_i=i^{-\alpha }$ for some exponent $\alpha$ on the order of .3-.5 (see here for example). Here the "bulk" captures most of the singular values and it's in some sense "pretty singular": for the majority i, the value $i^{-\alpha }$ is pretty small, i.e. "close to a singularity".

Thus it makes sense to extrapolate this regime and ask whether, for a singular matrix with singularities following a power law $s_i\sim i^{-\alpha },$ there is some regime of values of n where an SLT prediction (assuming that all $s_i$ below some cutoff are zero and keeping only the singular terms from these) actually describes the true value of the free energy to first order. In other words, we're replacing the "real" model that approximately solves $W_L{W}_R=W$ by an idealized "singular" model that approximately solves $W_L{W}_R=W_{singa}$ with the singular of $W_{sing}$ given by the formula $$s_i^{\prime }=\{\begin{array}{cc}0,& i>i_0\\ s_i,& i\le i_0\end{array}.$$

Unsurprisingly, the answer is a strong "no": both the prediction that the "approximately singular" $s_i$ for $i>i_0$ are "essentially zero" and the prediction that the large $s_i$ for $i\le i_0$ are large enough to impose a "reasonable gap" completely fail, and even the power law in the asymptotic from the SLT prediction fully fails to capture the power law of the true Bayesian learning prediction in this case. Here again, note that the key issue is the high-dimensionality of the system. (Also note here that I'm blackboxing a lot of math: happy to discuss it more in comments etc., and I'm also planning to write out a more careful version of this later.)

Discussion

One can try to salvage an SLT prediction in the LoRA example (which I think is particularly damning) in a few ways. Maybe:

• The important contribution comes from the "non-power law" part of the tail
• The assumption that LoRA rank reduction is a good model of Bayesian learning more generally is flawed
• etc.

However I think that together with the known results about modular addition, these failure modes show that in a meaningful way, realistic models do not have a good "singular model" with strong predictive power, even if we are only trying to make predictions about Bayesian learning. The regime where the SLT prediction succeeds certainly exists (this is a rigorous mathematical statement after all), but it requires too large a sample number n and imposes too strong of a regularity assumption on the "true" infinite-data loss landscape $L\left(\theta \right)$ (essentially, an assumption that "small" and "large" phenomena are cleanly separated by a large gap) for it to even approximately give valid predictions in regimes we care about. Again, the "dominant" brunt of my intuition here is that this failure is related to the fact that the predictions of SLT assume that the number of parameters is fixed (i.e. O(1)) as the number of samples goes to infinity -- but in reality, the number of parameters is quite high, and the regime where SLT predictions hold exactly is in some sense exponential (or very large) compared to parameter count, and thus never attained (and essentially uninteresting).

Having said this, I want to point out that SLT has a number of really good results that I am deeply excited about. I think they both have good theoretical results for toy models or strongly-asymptotic (but still interesting) regimes where the SLT approximation is exact, which may directionally give good intuition about realistic models (in the same way that the harmonic oscillator gives very deep intuition about quantum and statistical mechanics, despite not all systems being reducible to it -- not that in particular, SLT can be understood as a "vastly generalized harmonic oscillator" with the essential property being the existence of a strongly position-localized -- though singular -- semiclassical approximation; ignore if these words are meaningless to you).

But also people who label their work as "singular learning theory" have really good empirical results where the measurements being made are strictly thermodynamic, and which continue to work in "explicitly non-SLT" contexts such as mean field theory, which capture the high-dimensionality and have totally different asymptotics than what is predicted in the SLT regime. (Note that mean-field is, of course, itself an approximation/ toy model!)

For example: timaeus (the organization that "does SLT") has an elegant result by Garrett Baker et al. that measures the effects of changes in training distribution on susceptibilities; Nina Panickssery and I have a paper on the "lambda-hat estimator" (heat capacity) tracking the description length of the algorithm learned by modular addition in generalizing vs. memorizing models; there is work on saddles; Timaeus has produced work on the relationship of attention heads and a certain thermodynamic quantity. An upcoming paper by Timaeus that I'm very excited about looks at a susceptibility-like ("conjugate variable") metric on inputs that generalizes influence functions.

All of these results are (I think) very good results directionally linking statistical mechanics invariants with interesting learning behaviors. Also, none of them are "properly SLT results": they simply assume standard links between information theory, thermodynamics, and Bayesian learning (some of them developed in the learning context by Sumio Watanabe, who first discovered them in the context of his singular models). None of them assume that one is in a regime where the SLT "approximation from singularities" even approximately holds -- and I suspect that if one were to dig even a little in most of these cases (e.g. look at larger-scale dependence on temperature, relationships between single-neuron statistics, etc.), then one would see that we are in a regime where in fact, in a strong sense, any "singular" toy model would fail to predict the behvior of interest in any regime or with any number of terms of the "singular" asymptotic expansion. (Essentially this belief is due to the fact that the lambda-hat estimator does not even approximately asymptote in the way that SLT predicts in the regimes of interest, and because in an intuition I have that I'm hoping to write down later, the SLT approach really fails to track inherently high-dimensional phenomena like grokking.)

I would be happy to be wrong about the failure of the SLT regime, and I retain hope that with sufficient processing/ renormalization, there is some sense in which the thermodynamic behavior of learning is extensively related to singularities in some suitably re-interpreted and lower-dimensional set of "relevant" variables which are not the weights.

Also I am excited about the work that Timaeus is doing and think that the "thermodynamics-aware" approach to learning that they try to follow is significantly underexplored. I just want to point out that there are very deep directions here that decouple from the assumptions of SLT and instead lean into high-dimensionality of the loss landscape, and which produce useful work that can be missed if one's model of Bayesian learning theory is "everything is singularities".

On the friendship fallacy and Owen Barfield

I just finished reading the book "The Fellowship: The Literary Lives of the Inklings", by Philip and Carol Zaleski. It's a book about an intellectually appealing and socially cohesive group of writers in Oxford who met weekly and critiqued each other's work, which included JRR Tolkien and CS Lewis. The book is very centered on Christianity (the writers also write Christian apologetics), but this works well, as understanding either Lewis or Tolkien or the Inklings in general without the lens of their deeply held thoughtful Christianity is about as silly as trying to analyze the Lion King without reading Hamlet.

But there is a core character in the book who is treated sympathetically and who I really hate: Owen Barfield, the "founding" Inkling. From his youth, he is a follower of Rudolf Steiner and a devoted Anthroposophist (a particularly benign group of Christian Occultists). Barfield was Lewis's friend, existing always in his shadow (Lewis was very famous in his lifetime as a philosopher and Christian apologist, a kind of Jordan Peterson of his time if you imagine Jordan Peterson had brains and real literary/academic credentials). He worked in a law firm and consistently saw himself as a thwarted philosopher/writer/poet, and he found recognition late in life after he wrote a Lewis biography and after his woo-adjacent ideas became more popular in the 60s.

Throughout his life, Barfield created a personal philosophy of "all the things I like/ think are interesting are kind of the same thing", and he was very sad when people he liked disapproved of, or failed to identify as "sort of the same thing" the different things that he mixed into his philosophy. While he generally is a bit of an "intellectual klutz", his fundamental failure is the "Friendship Fallacy": the idea of treating ideas as friends, as something deserving of loyalty. When he encounters different ideas he likes, he "wants them to get along," and when ideas fail to convince skeptics or produce results or interface with reality (or indeed, with faith), he simply fails to impose any kind of falsifiability requirement and treats this as a loyalty test he must pass. He totally lacks the kind of internal courage needed to kill one's darlings (whether philosophical or literary) and to treat his own ideas with skepticism and view towards falsification -- perhaps the core trait of a good thinker (Feynman's "You must not fool yourself—and you are the easiest person to fool").

Interestingly, I don't extend this antipathy to the Christianity of the group's other famous members. Unlike Barfield, Tolkien and Chesterton largely succeed (imo) in separating the domains of the literary, the psychological, and the religious. They don't pretend to be scientific authorities or predict things "in the world". Tolkien in particular is very anti-progress and a bit of a luddite, but in my understanding his work as a linguist is very good for his time. In fact, it's funny that his deeply Christian mentality created one of the most "atheist nerd"-like behaviors of creating thoroughly crafted fictional languages of fantasy cultures. I've been surprised to learn from reading a couple of his biographies that his linguistic worldbuilding in fact preceded his fantasy work: he designed Elvish before writing any work in his canon, and wrote the work to flesh out the mythology behind expressions and poems. He famously said about his work "The making of language and mythology are related functions". In fact, he viewed the work of producing plausible cultures and languages -- in my view an admirable (though non-academic) kind of secular scholarship analogous to studying alternative physical systems, etc. -- as an explicitly Christian task of "subcreation", a sort of worship-by-imitation of God.

It's a bit hard to exactly formulate a razor between the kind of "lazy scientism" of Barfield and various other forms of "pseudoscientific woo" and the serious and purely mystical/ inspirational deep religiosity of people like Tolkien (and to a lesser extent Lewis -- another interesting thing I learned was that he started out as a devoted atheist in a world where this was actually socially fraught, and was converted through a philosophical struggle involving Barfield and Tolkien in particular). But maybe the idea of a "philosophy without struggle": a tendency towards confirmation and a total lack of earnest self-questioning goes a part of the way towards explaining this distinction. Another part is the difference between a purely metaphysical personal religion and a more woo idea of a religion that "makes predictions about the world". I think the thing that really took me aback a bit was the level of academic embrace of Barfield late in his life, not just as a Lewis biographer but as a respected academic philosopher with honorary professorships and the works - a confirmation (if ever more are needed) that lazy pseudointellectualism and confirmation bias are very much not incompatible with academic success. Another theme that I think is interesting is the fact that Lewis and Tolkien were at times genuinely interested and even somewhat inspired by his ideas (though they had no time for occultism or 60-esque woo). The extent to which this happened is hard to gauge (he outlived them and wrote a lot about how he influenced them in his biographies/reminiscences, and this was then picked up by scholars). But unquestioningly, this did occur to some extent. And whether or not you class Tolkien/Lewis as "valuable thinkers", the history of science and philosophy does seem to abound with examples of clear and robust thinkers whose good ideas were to some extent inspired by charismatic charlatans and woo.

Below are my personal notes on Barfield that I wrote after reading the book.

I despise Barfield. Not in the visceral sense that the first syllable of his name may (Anthroposophically) evoke. Indeed I identify with the underdog/late-bloomer shape of his biography, with his striving towards a higher calling. I readily adopt the book's sympathy towards him as a literary character with fortunes tied to an idea deeply espoused, a thwarted writer with some modicum of undiscovered talent. My antipathy isn't even in the specifics of what he espouses: a mild but virulently wrong view of science and philosophy adjacent to all the stupid of my parents' generation of `anthroposophy' (Atlantis, Consciousness and Quantum Mechanics, anti-Evolutionism, Vibes). But I despise him as one of a Fundamental Mistake. That of confusing science and personality. Being loyal to a scientific or philosophical discipline isn't like being loyal to a person: if it's consistently fucking up and you need to make excuses for its behavior to all your reasonable smart friends, you're not being a good friend but rather a bad scientist. Barfield is almost an archetype of Bad Science if you project out the crazy/dogmatic/ political/ evil-Nazi component. He really is a nice man. But within his mild-mannered Christian friendliness which I respect, he is inflexible and unscientific. He doesn't update. He glows when people endorse his preferred view (Anthroposophy and Steiner) and sadly laments when they disagree with him -- because he can't help but feel like ``there's something there''. He wants to seamlessly draw parallels between all the nice things he and other nice people believe. He draws lines of identification back and forth between all the things he likes (Coleridge <> Himself <> Quantum Mechanics <> Anthroposophy <> Steiner <> Religion <> Consciousness <> Complementary dualism/"polarity"). He has "nothing but symbols" in his brain, and the symbols in his brain aren't strong enough to notice that they fail to signify. A person without significance, with a philosophy without significance, possessed of a brain without the capacity to grasp the concept of what it means to signify. The first of these is a tragedy (people should matter) and his late-found fame mediated through famous friends is a sweet story, maybe one he even deserves as the first-mover of the Inklings, the reason for the Lewis-Tolkien friendship, etc. The second is a neutral: theories that fail to achieve significance "in their lifetime" may be bunk but may have value: Greek Atomism, various prescient ideas about physics and computers (Babbage/ Lovelace), etc. But the third is a profound personal failing, and it's only through luck and through (mostly well-placed) trust in much smarter and more rigorous friends that he avoided attaching this vapid form of mentation to something truly vile: Nazism (which he very briefly flirted with, charmed by its interest in magic and the occult), various fundamentalisms (including an anti-evolutionary fundamentalism: his friends believed in evolution but he didn't really buy it "on vibes"; he was never a fundamentalist), Communism, etc.

Why I'm in AI sequence: 2020 Journal entry about gpt3

I moved from math academia to full-time AI safety a year ago -- in this I'm in the same boat as Adam Shai, whose reflection post on the topic I recommend you read instead of this.

In making the decision, I went through a lot of thinking and (attempts at) learning about AI before that. A lot of my thinking had been about whether a pure math academic can make a positive difference in AI, and examples that I thought counterindicated this -- I finally decided this might be a good idea after talking to my sister Lizka extensively and doing MATS in Summer of 2023. I'm thinking of doing a more detailed post about my decision and thinking later, in case there are other academics thinking about making this transition (and feel free to reach out in pm's in this case!).

But one thing I have started to forget is how scary and visceral AI risk felt when I was making the decision. I'm both glad and a little sad that the urgency is less visceral and more theoretical now. AI is "a part of the world", not an alien feature: part of the "setting" in the Venkat Rao post that was part of my internal lexicon at the time.

For now, in order to fill a gap in my constantly flagging daily writing schedule, I'll share a meandering entry from 2020 about how I thought about positive AI futures. I don't endorse a lot of it; much is simplistic and low-context, or alternatively commonplace in these circles, though some of it holds up. It's interesting reading back that the thing I thought was most interesting as a first attempt at orienting my thinking was fleshing out "positive futures" and what they might entail. Two big directional updates I've had since are thinking harder about "human alignment" and "human takeover", and trying to temper the predictions that assume singularitarian "first-past-the-post" AGI for a messier "AI-is-kinda-AGI" world that we will likely end up in.

journal entry

7/19/2020 [...] I'm also being paranoid about GPT-3.

Let's think. Will the world end, and if so, when? No one knows, obviously. GPT-3 is a good text generation bot. It can figure out a lot about semantics, mood, style, even a little about humor. It's probably not going to take over the world yet. But how far away are we from AGI?

GPT-3 makes me think, "less than a decade". There's a possibility it will be soon (within the year). I'd assign that probability 10%. It felt like 20% when I first saw its text, but seeing Sam Altman's remark and thinking a little harder, I don't think it's quite realistic for it to go AGI without a significant extra step or two. I think that I'd give it order of 50% within the decade. So it's a little like living with a potentially fatal disease, with a prognosis of 10 years. Now we have no idea what AGI will be like. It will most likely either be very weird and deadly or revolutionary and good, though disappointing in some ways. I think there's not much we can do about the weird and deadly scenarios. Humans have lived in sociopathic times (see Venkat's notes on his 14th Century Europe book). It would probably be shorter and deadlier than the plague; various "human zoo" scenarios may be pleasant to experience (after all zoo animals are happier in general than in the wild, at least from the point of view of basic needs), but harrowing to imagine. In any case, it's not worth speculating on this.

What would a good outcome look like? Obviously, no one knows. It's very hard to predict our interaction with a super-human intelligence. But here are some pretty standard "decent" scenarios: (1) After a brief period of a pro-social AI piloted by a team of decent people, we end up with a world much like ours but with AI capabilities curbed for a long period of time [...]. If it were up to me I would design this world with certain "guard rail"-like changes: to me this would be a "Foundation"-style society somewhere in New Zealand (or on the bottom of the ocean perhaps? the moon?) consisting of people screened for decency, intelligence, etc. (but with serious diversity and variance built in), and with control of the world's nukes, with the responsibility of imposing very basic non-interference and freedom of immigration criteria on the world's societies (i.e., making the "archipelago" dream a reality, basically). So enforcing no torture, disincentivizing violent conflict, imposing various controls to make sure people can move from country to country and are exposed to the basic existence of a variety of experiences in the world, but allowing for culturally alien or disgusting practices in any given country: such as Russian homophobia, strict Islamic law, unpleasant-seeming (for Western Europeans) traditions in certain tribal cultures, etc. This combined with some sort of non-interventionist altruistic push. In this sci-fi scenario the Foundation-like culture would have de facto monopoly of the digital world (but use it sparingly) and also a system of safe nuclear power plants sufficient to provide the world's power (but turned on carefully and slowly, to prevent economic jolts), but to carefully and "incontrovertibly" turn most of the proceeds into a universal basic income for the entire world population. Obviously this would have to be carefully thought out first by a community of intelligent and altruistic people with clear rules of debate/decision. --- The above was written extremely sleepy. [...]

(2) (Unlikely) AI becomes integrated with (at first, decent and intelligent later, all interested) humans via some kind of mind-machine interface, or alternatively a faithful human modeling in silica. Via a very careful and considered transition (in some sense "adiabatic", i.e. designed so as not to lose any of our human ethos and meaning that can possibly be recovered safely) we become machines, with a good and meaningful (not wireheaded, other than by considered choice) world left for the hold-outs who chose to remain purely human.

(3) The "Her" scenario: AI takes off on its own, because of human carelessness or desperation. It develops in a way that cherishes and almost venerates humans, and puts effort into making a good, meaningful existence for humans (meaningful and good in sort of the above adiabatic sense, i.e. meaningful via a set of clearly desirable stages of progress from step to step, without hidden agendas, and carefully and thoughtfully avoiding creating or simulating, in an appropriate sense, anything that would be considered a moral horror by locally reasonable intelligences at any point in the journey). AI continues its own existence, either self-organized to facilitate this meaningful existence of humans or doing its own thing, in a clearly separated and "transcendent" world, genuinely giving humans a meaningful amount of self-determination, while also setting up guardrails to prevent horrors and also perhaps eliminating or mitigating some of the more mundane woes of existence (something like cancer, etc.) without turning us into wireheads.

(4) [A little less than ideal by my book, but probably more likely than the others]: The "garden of plenty" scenario. AI takes care of all human needs and jobs, and leaves all humans free to live a nevertheless potentially fulfilling existence, like aristocrats or Victorians but less classist, socializing learning reading, etc., with the realization that all they are doing is a hobby: perhaps "human-generated knowledge" would be a sort of sport, or analog of organic produce (homeopathically better, but via a game that makes everyone who plays it genuinely better in certain subtle ways). Perhaps AI will make certain "safe" types of art, craft and knowledge (maybe math! Here I'm obviously being very biased about my work's meaning not becoming fully automated) purely the domain of humans, to give us a sense of self-determination. Perhaps humans are guided through a sort of accelerated development over a few generations to get to the no.2 scenario.

(5) There is something between numbers 3 and 4 above, less ideal than all of the above but likely, where AI quickly becomes an equal player to humans in the domain of meaning-generation, and sort of fills up space with itself while leaving a vaguely better (maybe number 4-like) Earth to humans. Perhaps imposes a time limit on humans (enforced via a fertility cap, hopefully with the understanding that humans can raise AI babies with genuine sense of filial consciousness and complete with bizzarre scences of trying to explain the crazy world of AI to their parents), after which the human project becomes the AI project, probably essentially incomprehensible to us.

There's a sense that I have that while I'm partial to scenarios 1 and 2: I want humans to retain the monopoly on meaning-generation and to be able to feel empowered and important, it will be seen to be old-fashioned and almost dangerous by certain of my peers because of the lack of emphasis on harm-prevention, stable future, etc. I think this is part of the very serious debate, so far abstract and fun, but, as AI gets better, perhaps turning heated and loud, between whether comfort or meaning are more important goals of the human project (and both sides will get weird). I am firmly on the side of meaning, with a strict underpinning of retaining bodily and psychological integrity in all the object-level and meta-level senses (except I guess I'm ok with moving to the cloud eventually? Adiabatic is the word for me). Perhaps my point of view is on the side I think it is just in the weird group of futurists and rationalists that I mostly read when reading about AI: probably the generic human who thinks about AI is horrified by all of the above scenarios and just desperately hoping it will go away on its own, or has some really idiosyncratic mix of the above or other ideas which seem obviously preferable to them.

Why you should try degrading NN behavior in experiments.

I got some feedback on the post I wrote yesterday that seems right. The post is trying to do too many things, and not properly explaining what it is doing, why this is reasonable, and how the different parts are related.

I want to try to fix this, since I think the main piece of advice in this post is important, but gets lost in all the mess.

This main point is:

experimentalists should in many cases run an experiment on multiple neural nets with a variable complexity dial that allows some "natural" degradations of the NN's performance, and certain dials are better than others depending on context.

I am eventually planning splitting out the post into a few parts, one of which explains this more carefully. When I do this I will replace the current version of the post with just a discussion of the "koan" itself: i.e., nitpicks about work that isn't careful about thinking about the scale at which it is performing interpretability.

For now I want to give a quick reductive take on what I hope to be the main takeaway of this discussion. Namely, why I think "interpretability on degraded networks" is important for better interpretability.

Basically: when ML experiments modify a neural net to identify or induce a particular behavior, this always degrades performance. Now there are two hypotheses for what is going on:

1. You are messily pulling your NN in the direction of a particular behavior, and confusing this spurious messy phenomenon with finding a "genuine" phenomenon from the program's point of view.

2. You are messily pulling your NN in the direction of a particular behavior, but also singling out a few "real" internal circuits of the NN that are carrying out this behavior.

Because of how many parameters you have to play with and the polysemanticity of everything in a NN, it's genuinely hard to tell these two behaviors apart. You might find stuff that "looks" like a core circuit, but actually is just bits of other circuits combined together, and your circuit-fitting experiment makes look like a coherent behavior, and any nice properties of the resulting behavior that make it seem like an "authentic" circuit are just artefacts of the way you set up the experiment.

Now the idea behind running this experiment at "natural" degradations of network performance is to try to separate out these two possibilities more cleanly. Namely, an ideal outcome is that in running your experiment on some class of natural degradation of your neural net, you find a regime such that

• the intervention you are running no longer significantly affects the (naturally degraded) performance
• the observed effect still takes place.

Then what you've done is effectively "cleaned up" your experiment such that you are still probably finding interpretable behaviors in the original neural net (since a good degradation is likely to contain a subset of circuits/behaviors of your original net and not many "new behaviors), in a way that sufficiently reduced the complexity that the behavior you're seeking is no longer "entangled" with a bunch of other behaviors; this should significantly update you that the behavior is indeed "natural" and not spurious.

This is of course a very small, idealized sketch. But the basic idea behind looking at neural nets with degraded performance is to "squeeze" the complexity in a controlled way to suitably match the complexity of the circuit (and how it's embedded in the rest of the network/how it interacts with other circuits). If you then have a circuit of "the correct complexity" that explains a behavior, there is in some sense no "complexity room" for other sneaky phenomena to confound it.

In the post, the natural degradation I suggested is the physics-inspired "SLGD sampling" process which in some sense tries to add a maximal amount of noise to your NN while only having a limited impact on performance (measured by loss); this has a bias of keeping "generally useful" circuits and interactions and noising more inessential/ memorize-y circuits. Other interventions that have different properties are "just adding random noise" (either to weights or activations) to suitable reduce performance, or looking at earlier training checkpoints. I suspect that different degradations (or combinations thereof) are appropriate to isolate the relevant complexity of different experiments.